CN114983302B

CN114983302B - Gesture determining method and device, cleaning equipment, storage medium and electronic device

Info

Publication number: CN114983302B
Application number: CN202210742705.5A
Authority: CN
Inventors: 韩松杉; 盛腾飞; 王睿麟
Original assignee: Dreame Innovation Technology Suzhou Co Ltd
Current assignee: Dreame Innovation Technology Suzhou Co Ltd
Priority date: 2022-06-28
Filing date: 2022-06-28
Publication date: 2023-08-08
Anticipated expiration: 2042-06-28
Also published as: CN114983302A

Abstract

The embodiment of the invention provides a method and a device for determining a gesture, cleaning equipment, a storage medium and an electronic device, wherein the method comprises the following steps: detecting a first image shot by a first camera arranged on the cleaning equipment, and determining a first ROI (region of interest) of a first plane included in the first image; determining first 3D point cloud information corresponding to a first ROI area; correcting the angular speed of the cleaning device based on the first 3D point cloud information; the attitude of the cleaning device is determined based on the corrected angular velocity of the cleaning device. According to the invention, the problem that the estimation error is large because the posture estimation is needed to be carried out by relying on the accelerometer in the related technology is effectively solved.

Description

Gesture determining method and device, cleaning equipment, storage medium and electronic device

[ field of technology ]

The present invention relates to the field of communications, and in particular, to a method and apparatus for determining a gesture, a cleaning device, a storage medium, and an electronic apparatus.

[ background Art ]

With the rapid development of artificial intelligence technology, more and more intelligent cleaning devices (such as a sweeper, a floor cleaner, a dust collector and the like) enter the life of people, so that the life of people is more and more convenient.

In the related art, the body posture estimation is a necessary premise for the autonomous navigation floor sweeping machine (taking the floor sweeping machine as an example here, of course, other cleaning devices, such as a floor washing machine, etc.) to perform mapping, planning and obstacle avoidance. The body gesture in the existing autonomous navigation floor sweeping machine is mainly yaw angle, pitch angle and roll angle based on a global coordinate system. The yaw angle is mainly estimated by using single-line laser radar, camera, gyroscope information and the like, the pitch angle and the rolling angle are mainly estimated by relying on acceleration, but when the accelerometer is used, the acceleration reading of the accelerometer can be influenced by factory calibration, offset, noise, instability, aging, self-movement acceleration and the like of the accelerometer, so that larger errors exist in the estimation of the pitch angle and the rolling angle of the global coordinate system, drift can occur after long-time use, and the errors of the estimation of the pitch angle and the rolling angle of the global coordinate system are larger.

Aiming at the problem that the attitude estimation is carried out by relying on an accelerometer in the prior art, so that the estimation error is large, no effective solution is proposed at present.

[ invention ]

The embodiment of the invention provides a method and a device for determining a gesture, cleaning equipment, a storage medium and an electronic device, which are used for at least solving the problem that the gesture estimation is needed to be carried out depending on an accelerometer in the related technology, so that the estimation error is larger.

According to an aspect of the present invention, there is provided a method of determining a posture, including: detecting a first image shot by a first camera arranged on a cleaning device, and determining a first ROI (region of interest) of a first plane included in the first image; determining first 3D point cloud information corresponding to the first ROI area; correcting the angular velocity of the cleaning device based on the first 3D point cloud information; and determining the posture of the cleaning device based on the corrected angular velocity of the cleaning device.

In an exemplary embodiment, determining the first 3D point cloud information corresponding to the first ROI area includes: under the condition that the cleaning equipment comprises a target sensor, determining 3D point cloud information of a target area acquired by the target sensor, wherein the first image is an image obtained by shooting the target area, and the difference between the shooting time of the first image and the time of the target sensor for acquiring the 3D point cloud information of the target area is smaller than a preset threshold value; registering all pixel points included in the first image with the 3D point cloud information of the target area; and determining the first 3D point cloud information corresponding to the first ROI based on a registration result.

In an exemplary embodiment, determining the first 3D point cloud information corresponding to the first ROI area includes: detecting a second image shot by the first image pickup device under the condition that the cleaning device does not comprise the target sensor, and determining a second ROI (region of interest) of a second plane included in the second image, wherein the second image is a previous frame image of the first image; extracting a first 2D feature point of the first ROI area, and extracting a second 2D feature point of the second ROI area; matching the first 2D feature points with the second 2D feature points based on a feature point matching mode; and determining the first 3D point cloud information corresponding to the first ROI based on a matching result.

In an exemplary embodiment, determining the first 3D point cloud information corresponding to the first ROI area based on the matching result includes: determining plane constraint of the first camera equipment, and determining a target homography matrix based on the matching result and the plane constraint; determining a rotation matrix and a translation vector without scale between the first image and the second image based on the target homography matrix; determining an absolute position delta between the first image and the second image based on a first sensor and/or a first algorithm of the cleaning device; determining a scaled translation vector based on the non-scaled translation vector and the absolute position increment; and triangulating the translation vector with the scale and the matching result to determine first 3D point cloud information corresponding to the first ROI area.

In one exemplary embodiment, detecting a first image captured by a first image capturing apparatus provided on a cleaning apparatus, determining a first ROI area of a first plane included in the first image includes: and detecting the first image by using a neural network model, and determining the first ROI of the first plane included in the first image, wherein the neural network model is a model obtained by training an initial neural network model by using training data, the training data comprises a plurality of groups of data, and each group of data comprises a training image and the ROI of the plane included in the training image.

In an exemplary embodiment, correcting the angular velocity of the cleaning device based on the first 3D point cloud information includes: screening a target 3D point cloud from the first 3D point cloud according to a target screening mode, and determining a first plane equation based on the target 3D point cloud; determining a target maximum 3D point cloud from the target 3D point clouds, and generating a target plane equation through least square optimization based on the target maximum 3D point cloud and a Huber robust kernel function, wherein the target maximum 3D point cloud is a 3D point cloud which is smaller than a preset distance threshold from the first plane equation; the angular velocity is corrected based on the target plane equation.

In one exemplary embodiment, correcting the angular velocity based on the target plane equation includes: determining a quaternion of a previous moment of the cleaning device, and determining a second plane equation of the previous moment of the cleaning device based on the quaternion of the previous moment of the cleaning device; determining a plane equation difference between the target plane equation and the second plane equation, and obtaining a gyroscope angular velocity measurement of the cleaning device; and determining an angular velocity correction amount based on the plane equation difference value, and correcting the angular velocity measurement value of the gyroscope based on the angular velocity correction amount to obtain corrected angular velocity.

In an exemplary embodiment, after determining an angular velocity correction amount based on the plane equation difference value and correcting the gyroscope angular velocity measurement value based on the angular velocity correction amount, the method further includes: updating the quaternion of the previous moment posture based on the corrected angular speed to obtain the quaternion of the updated current moment posture; normalizing the quaternion of the gesture at the current moment, and converting the quaternion after normalization into an Euler angle; an updated pose of the cleaning device is determined based on the euler angles.

In an exemplary embodiment, before detecting a first image captured by a first image capturing apparatus provided on the cleaning apparatus, and determining a first ROI area of a first plane included in the first image, the method further includes: calibrating a parameter of a sensor included in the cleaning apparatus, the parameter being at least one of: an internal reference matrix and a distortion model of the first image pickup apparatus; an internal reference of a gyroscope of the cleaning device; an external parameter between the first camera device coordinate system and the gyroscope coordinate system; external parameters between the gyroscope coordinate system and the machine body coordinate system of the cleaning equipment; a point cloud reference model of a target sensor included in the cleaning apparatus; and the external parameters between the target sensor coordinate system and the gyroscope coordinate system.

According to another aspect of the present invention, there is also provided a gesture determining apparatus including: a first determining module, configured to detect a first image captured by a first image capturing device provided on a cleaning device, and determine a first ROI area of a first plane included in the first image; a second determining module, configured to determine first 3D point cloud information corresponding to the first ROI area; the correction module is used for correcting the angular speed of the cleaning equipment based on the first 3D point cloud information; and a third determining module for determining the posture of the cleaning device based on the corrected angular velocity of the cleaning device.

According to another aspect of the invention, there is also provided a cleaning apparatus comprising the attitude determination means described in the above apparatus embodiments.

According to another embodiment of the present invention, there is also provided a computer-readable storage medium including a stored program, wherein the program when run performs the method described in any one of the embodiments above.

According to another embodiment of the invention, there is also provided an electronic device comprising a memory, in which a computer program is stored, and a processor, which is arranged to execute the method according to any of the embodiments described above by means of the computer program.

According to the invention, the first image shot by the first camera equipment arranged on the cleaning equipment can be detected to determine the first ROI area of the first plane included in the first image, further determine the first 3D point cloud information corresponding to the first ROI area, correct the angular velocity of the cleaning equipment based on the first 3D point cloud information, and then determine the gesture of the cleaning equipment based on the corrected angular velocity of the cleaning equipment.

[ description of the drawings ]

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:

FIG. 1 is a block diagram of a hardware configuration of a method for determining a gesture according to an embodiment of the present invention;

FIG. 2 is a flow chart of a method of determining a pose of a map according to an embodiment of the present invention;

fig. 3 is a block diagram of the configuration of the attitude determination apparatus according to the embodiment of the present invention.

[ detailed description ] of the invention

The invention will be described in detail hereinafter with reference to the drawings in conjunction with embodiments. It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

The method embodiments provided by the embodiments of the present application may be performed in a mobile device, or similar computing device. Taking the mobile device as an example, fig. 1 is a block diagram of a hardware structure of a gesture determining method according to an embodiment of the present invention. As shown in fig. 1, the mobile apparatus may include one or more processors 102 (only one is shown in fig. 1) (the processor 102 may include, but is not limited to, a microprocessor MCU, a programmable logic device FPGA, or the like) and a memory 104 for storing data, and in one exemplary embodiment, may also include a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the configuration shown in fig. 1 is merely illustrative and not limiting of the configuration of the mobile device described above. For example, the mobile device may also include more or fewer components than shown in fig. 1, or have a different configuration than the equivalent functions shown in fig. 1 or more than the functions shown in fig. 1.

The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a method for determining an attitude in an embodiment of the present invention, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, that is, to implement the above-described method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission means 106 is arranged to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communications provider of the mobile device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.

The invention is illustrated below with reference to examples:

in this embodiment, a method for determining a gesture is provided, as shown in fig. 2, and the method includes the following steps:

s202, detecting a first image shot by a first image pickup device arranged on a cleaning device, and determining a first ROI (region of interest) of a first plane included in the first image;

s204, determining first 3D point cloud information corresponding to the first ROI area;

s206, correcting the angular speed of the cleaning equipment based on the first 3D point cloud information;

s208, determining the posture of the cleaning device based on the corrected angular speed of the cleaning device.

The above operations may be performed by a controller, or a decision module, or a device with shooting and updating capabilities (for example, an intelligent sweeper, an intelligent scrubber, etc.), or may be performed by another processing device or a processing unit with similar processing capabilities, where the controller or other execution entity may be separately present or integrated into the cleaning device. The following describes the operations performed by the controller (only one exemplary description, and other devices or modules may be used to perform the operations in actual operations):

In the above embodiment, the first image capturing device includes, but is not limited to, an RGB color camera, a gray-scale camera, and the like, and the first plane and the operation plane of the cleaning device may be in a parallel relationship, and the first plane may have multiple types of planes, for example, a floor, a carpet, a table top, a cabinet table top, and the like, so that the first ROI areas of multiple types of first planes included in the first image may be determined simultaneously, for example, when the first plane in the first image includes three planes, respectively, a floor, a table top, and a cabinet table top, the floor, the table top, and the table top in the first image may be detected simultaneously, and the first ROI areas corresponding to the floor, the table top, and the cabinet table top respectively may be determined simultaneously, and priorities of different types of first planes may be set preferentially, for example, when the first plane in the first image includes three planes, respectively, a floor, a table top, and a cabinet table top, the priority of the first plane may be set to be ground > desktop > cabinet table top, and then the three first planes included in the first image may be detected according to the ground > desktop > cabinet table top setting, and the corresponding first ROI areas may be sequentially determined, in addition, the first images may be plural, so that the plural first images may be detected simultaneously, and the first ROI areas of one or more first planes included in the plural first images may be determined simultaneously, and the first image photographed first may be detected first according to the order in which the plural first images are photographed by the first photographing apparatus, so as to determine the first ROI area of one or more first planes included in the first image, to be described, the description of the type of the first plane and the priority of the first image is only one exemplary embodiment, the type of the first plane and the priority of the first image are not limited to the above examples.

In the above embodiment, there may be multiple first ROI areas, so that first 3D point cloud information corresponding to the multiple first ROI areas may be determined simultaneously, for example, when a first plane included in the first image is a ground, and the first image is detected, and when multiple first ROI areas of the ground are determined, first 3D point cloud information corresponding to the multiple first ROI areas may be determined simultaneously, and corresponding first 3D point cloud information may be determined according to the priority of determining the first ROI areas, for example, when the first plane in the first image includes three planes, which are respectively a ground, a desktop, and a cabinet mesa, the priority of the first plane may be set to be a ground > desktop, and then the three first planes included in the first image may be detected according to the ground > desktop setting, and then the corresponding first ROI areas may be determined sequentially, and the first 3D points corresponding to the first determined first ROI areas may be determined preferentially according to the order of the determined first ROI areas, and the first ROI areas may be described as the first ROI areas, and the priority may be implemented as the first ROI areas.

In the above embodiment, the first image captured by the first image capturing device provided on the cleaning device may be detected to determine the first ROI area of the first plane included in the first image, further determine the first 3D point cloud information corresponding to the first ROI area, and correct the angular velocity of the cleaning device based on the first 3D point cloud information, and then determine the posture of the cleaning device based on the corrected angular velocity of the cleaning device.

In an exemplary embodiment, determining the first 3D point cloud information corresponding to the first ROI area includes: under the condition that the cleaning equipment comprises a target sensor, determining 3D point cloud information of a target area acquired by the target sensor, wherein the first image is an image obtained by shooting the target area, and the difference between the shooting time of the first image and the time of the target sensor for acquiring the 3D point cloud information of the target area is smaller than a preset threshold value; registering all pixel points included in the first image with the 3D point cloud information of the target area; and determining the first 3D point cloud information corresponding to the first ROI based on a registration result. In this embodiment, the predetermined threshold may be a preset value, may be set to 3 ms, 5 ms, 10 ms, 15 ms, etc., for example, when the predetermined threshold is 10 ms, the difference between the capturing time of the first image and the time of the target sensor for capturing the 3D point cloud information of the target area needs to be less than 10 ms, and in addition, the target sensor includes, but is not limited to, a 3D point cloud sensor, a TOF sensor, an RGBD sensor, etc., it should be noted that the setting of the above predetermined threshold is only an exemplary embodiment, other predetermined thresholds may be used in practical application, and the setting of the predetermined threshold may be adjusted according to practical application, and in addition, the method for registering the first 3D point cloud and the first image may be used but not limited to a method for performing conversion alignment based on the coordinate system of the 3D point cloud sensor and the first image capturing device.

In an exemplary embodiment, determining the first 3D point cloud information corresponding to the first ROI area includes: detecting a second image shot by the first image pickup device under the condition that the cleaning device does not comprise the target sensor, and determining a second ROI (region of interest) of a second plane included in the second image, wherein the second image is a previous frame image of the first image; extracting a first 2D feature point of the first ROI area, and extracting a second 2D feature point of the second ROI area; matching the first 2D feature points with the second 2D feature points based on a feature point matching mode; and determining the first 3D point cloud information corresponding to the first ROI based on a matching result. In the present embodiment, the feature point matching method includes an optical flow method, a descriptor matching method, and the like, and of course, other methods having a 2D feature matching function may be used.

In an exemplary embodiment, determining the first 3D point cloud information corresponding to the first ROI area based on the matching result includes: determining plane constraint of the first camera equipment, and determining a target homography matrix based on the matching result and the plane constraint; determining a rotation matrix and a translation vector without scale between the first image and the second image based on the target homography matrix; determining an absolute position delta between the first image and the second image based on a first sensor and/or a first algorithm of the cleaning device; determining a scaled translation vector based on the non-scaled translation vector and the absolute position increment; and triangulating the translation vector with the scale and the matching result to determine first 3D point cloud information corresponding to the first ROI area. In this embodiment, the edge and/or wrong matched 2D feature points may be screened based on the matching result and the ranac method, then more and more accurate matched 2D feature points may be obtained from the unmasked matched 2D feature points by plane constraint, further, the target homography matrix may be determined based on these more accurate matched 2D feature points, so as to improve the accuracy and robustness of the matched 2D feature points, in addition, the first sensor includes but is not limited to a code disc sensor, a wheel speed meter sensor, etc., the first algorithm includes but is not limited to two adjacent frame algorithms of a 2D laser sensor, for example, an absolute position increment between a first image and a second image may be obtained by calculating a code disc of a left and right wheel of a sweeper, or an absolute position increment between two frames may be calculated by adopting two adjacent frame ICP algorithms of the 2D laser sensor, if there is no sensor such as a code disc, a wheel speed meter, or no other algorithm (i.e. the first algorithm mentioned above), and in the case that an accurate translation scale may be calculated, an approximate value may be adopted, and a motion vector may be obtained by adopting a motion vector may not be obtained by multiplying a motion vector by an approximate vector, and a motion vector may not be obtained by an approximate vector, and a motion vector may be obtained by a motion vector, or a motion vector may be a motion method may be obtained by a motion vector, or a motion vector may be a motion method by a motion method, for example, a scrubber, etc.), assuming that the average depth of the 3D point cloud of the current frame is N meters (e.g., 2.5 meters, 3 meters, 3.5 meters, etc.), the 3D point clouds of all the interior points obtained above are normalized, and the normalization operation makes the 3D point clouds of all the interior points have a depth of N meters.

It should be noted that the illustration of the matching 2D feature points and the absolute position increment is only one exemplary embodiment, and the matching 2D feature points and the absolute position increment are not limited to the illustration.

In one exemplary embodiment, detecting a first image captured by a first image capturing apparatus provided on a cleaning apparatus, determining a first ROI area of a first plane included in the first image includes: and detecting the first image by using a neural network model, and determining the first ROI of the first plane included in the first image, wherein the neural network model is a model obtained by training an initial neural network model by using training data, the training data comprises a plurality of groups of data, and each group of data comprises a training image and the ROI of the plane included in the training image. In this embodiment, the initial neural network model may be trained in advance by using training data, where the training data used in training may be from a database, for example, images including the planar ROI area that have been detected are collected in advance and formed into a database, and then the first image is identified based on the trained neural network model, and by using the neural network training method, the planar ROI area included in the first image may be more accurately identified.

In an exemplary embodiment, correcting the angular velocity of the cleaning device based on the first 3D point cloud information includes: screening a target 3D point cloud from the first 3D point cloud according to a target screening mode, and determining a first plane equation based on the target 3D point cloud; determining a target maximum 3D point cloud from the target 3D point clouds, and generating a target plane equation through least square optimization based on the target maximum 3D point cloud and a Huber robust kernel function, wherein the target maximum 3D point cloud is a 3D point cloud which is smaller than a preset distance threshold from the first plane equation; the angular velocity is corrected based on the target plane equation. In this embodiment, the target screening method includes a ranac method, or the like, and of course, other screening methods may be used, and in addition, the first plane equation including the target maximum 3D point cloud may be used as an initial value, and a Huber robust kernel function is added to generate the target plane equation through least square optimization, where the Huber robust kernel function is applied, the weight of the outer points in the 3D point cloud may be further determined, and the optimal target plane equation may be determined from the target maximum 3D point cloud, so as to effectively reduce the influence of the outer point data in the 3D point cloud or reduce the influence of the observation noise of the 3D point cloud.

In the above embodiment, the predetermined distance threshold may be a preset value, and may be set to 2 cm, 5 cm, 10 cm, or the like, for example, when the predetermined distance threshold is 5 cm, a 3D point cloud that is less than 5 cm from the first plane equation is determined from the target 3D point clouds, and thus, these 3D point clouds may be determined as the target maximum 3D point clouds.

In one exemplary embodiment, correcting the angular velocity based on the target plane equation includes: determining a quaternion of a previous moment of the cleaning device, and determining a second plane equation of the previous moment of the cleaning device based on the quaternion of the previous moment of the cleaning device; determining a plane equation difference between the target plane equation and the second plane equation, and obtaining a gyroscope angular velocity measurement of the cleaning device; and determining an angular velocity correction amount based on the plane equation difference value, and correcting the angular velocity measurement value of the gyroscope based on the angular velocity correction amount to obtain corrected angular velocity. In this embodiment, the second plane equation of the previous moment posture of the cleaning device may be determined based on the quaternion of the previous moment posture of the cleaning device, so as to calculate an optimal plane equation difference between the target plane equation and the second plane equation, and correct the measured value of the angular velocity of the gyroscope of the previous moment posture of the cleaning device based on the plane equation difference, so as to obtain the corrected angular velocity.

In an exemplary embodiment, after determining an angular velocity correction amount based on the plane equation difference value and correcting the gyroscope angular velocity measurement value based on the angular velocity correction amount, the method further includes: updating the quaternion at the previous moment based on the corrected angular velocity to obtain the quaternion of the updated current moment posture; normalizing the quaternion of the updated current moment gesture, and converting the quaternion after normalization into an Euler angle; an updated pose of the cleaning device is determined based on the euler angles. In this embodiment, due to calculation errors and other factors, the quaternion gradually loses the normalization feature, and therefore, it is necessary to perform normalization processing (i.e., the normalization processing) on the updated quaternion, and convert the quaternion after normalization processing into a yaw angle, a pitch angle, a roll angle (i.e., the euler angle), and the like.

In an exemplary embodiment, before detecting a first image captured by a first image capturing apparatus provided on the cleaning apparatus, and determining a first ROI area of a first plane included in the first image, the method further includes: calibrating a parameter of a sensor included in the cleaning apparatus, the parameter being at least one of: an internal reference matrix and a distortion model of the first image pickup apparatus; an internal reference of a gyroscope of the cleaning device; an external parameter between the first camera device coordinate system and the gyroscope coordinate system; external parameters between the gyroscope coordinate system and the machine body coordinate system of the cleaning equipment; a point cloud reference model of a target sensor included in the cleaning apparatus; and the external parameters between the target sensor coordinate system and the gyroscope coordinate system. In this embodiment, the internal parameters of the gyroscope of the cleaning device include, but are not limited to, offset, scale factor of 3-axis or single-axis, the external parameters between the first camera device coordinate system and the gyroscope coordinate system include, but are not limited to, rotation matrix and translation, the external parameters between the gyroscope coordinate system and the body coordinate system of the cleaning device include, but are not limited to, rotation matrix and translation, the point cloud internal model of the target sensor included in the cleaning device, and the external parameters between the target sensor coordinate system and the gyroscope coordinate system include, but are not limited to, rotation matrix and translation.

In the above embodiment, the calibration of the parameters of the sensor included in the cleaning device may be performed when the cleaning device is not shipped, or may be performed by a manager or a maintainer after the cleaning device is applied, and in addition, may be adjusted according to the actual application situation of the cleaning device, or the like.

It will be apparent that the embodiments described above are merely some, but not all, embodiments of the invention.

The following describes the present invention with reference to specific embodiments by taking the cleaning apparatus as an example of a sweeping robot:

step one: and calibrating the internal reference and the external reference offline. The external parameters and the internal parameters of each sensor included in the floor sweeping machine to be used can be calibrated in advance, and the following parameters are mainly calibrated:

1. an intra-camera (corresponding to the first image pickup apparatus described above) references a matrix and a distortion model. According to the difference of the selected mathematical models of the cameras, the corresponding internal reference matrixes and distortion models of the cameras are also different, and taking a pinhole model as an example, the internal reference matrix K comprises: focal length f _x And f _y Principal point offset coefficient x ₀ And y ₀ An axis inclination coefficient s, etc.;

the distortion model may be of the formula wherein [ x ] _c ；y _c ] ^T Is the coordinates of the undistorted point, [ x ] _corrected ；y _corrected ] ^T Is the coordinates of the point after de-distortion, r is the center distance of the point, parameter [ k ] ₁ ,k ₂ ,k ₃ ]Is a polynomial parameter of radial distortion, parameter [ p ] ₁ ,p ₂ ]Is the tangential distortion coefficient.

2. The gyroscope internal parameters include: offset of 3-axis or uniaxial, scale factor, etc.;

3. the external parameters of the camera coordinate system and the gyroscope coordinate system include: rotation matrix and translation;

4. the external parameters of the gyroscope coordinate system and the machine body coordinate system comprise: rotation matrix and translation;

5. a point cloud reference model of a TOF sensor (e.g., a self-contained TOF sensor of a sweeper);

6. the parameters of the TOF sensor coordinate system and the gyroscope coordinate system include: rotation matrix and translation.

Step two: plane detection

1. Detecting an object

The detection target is a plane in the image. Such planes should be in parallel relation to the plane of operation of the cleaning device and may be of various types, such as floors, carpets, desktops, cabinet countertops, etc.

2. Detection algorithm

(1) Detection of 2D images by camera to obtain ROI areas

The detection and segmentation algorithm based on the camera image adopts a deep learning method, and compared with the traditional method, the deep learning method has obvious advantages in the image fields of recognition rate, accuracy, semantics and the like, the invention does not limit the specific implementation method of the detection and segmentation algorithm of the plane of the camera image, and the detection result of the algorithm is the region or the range of the ROI on the image plane.

(2) 3D point cloud for calculating ROI (region of interest)

Depending on whether or not there is a point cloud sensor (corresponding to the target sensor described above), two cases are classified:

a. if the camera does not have a 3D point cloud sensor. A first step of extracting 2D feature points (corresponding to the second 2D feature points) from an ROI area (corresponding to the second ROI area) of a previous frame image (corresponding to the second image); a second step of extracting 2D feature points (corresponding to the first 2D feature points) in an ROI area (corresponding to the first ROI area) of a current frame image (corresponding to the first image); thirdly, matching the characteristic points of the previous frame image and the current frame image, wherein the method is not limited to an optical flow method, a descriptive sub-matching method and the like; fourthly, based on the 2D feature point matching result, solving a homography matrix (corresponding to the target homography matrix) by utilizing polar constraint (corresponding to the plane constraint) of a camera, wherein an outer point can be restrained by adopting a Ranac method, and an inner point can be screened; fifthly, decomposing the homography matrix into a rotation matrix between two frames of images and a translation vector without scale; step six, obtaining translation vectors with scales according to other sensors or other algorithms of the sweeper, for example, obtaining absolute position increment between two frames of images by adopting code plates of left and right wheels of the sweeper, or calculating the absolute position increment between two frames by adopting an ICP algorithm of two adjacent frames of 2D laser sensors, if no sensor (corresponding to the first sensor) such as a code plate, a wheel speed meter and the like or no other algorithm (corresponding to the first algorithm) can accurately calculate the accurate translation scales, adopting an approximate scale to replace the absolute scale, for example, obtaining the instruction speed of the sweeper according to the running state of the sweeper, and based on a uniform speed assumption, multiplying the interval time of two frames, then approximately calculating the absolute position increment between the two frames; and seventhly, triangulating according to the relative pose of two adjacent frames and the 2D coordinates of the matched inner points to obtain the 3D coordinates of all the inner points, wherein the relative pose of the two adjacent frames refers to a rotation matrix and a translation vector with a scale between the two frames. (ransac, random sample consensus, random sample agreement, which means that parameters of a mathematical model can be estimated in an iterative manner from a set of data sets containing "outliers").

In the sixth step, if an accurate translation vector between two frames cannot be obtained, or an approximate translation vector between two frames cannot be obtained, alternative methods such as point cloud depth normalization may be adopted, for example, first randomly designating a scale of one translation vector, then performing point cloud triangularization based on the translation vector with the random scale to obtain 3D coordinates of all internal points, and assuming that an average depth of 3D point clouds of the current frame is N meters (for example, 2.5 meters, 3 meters, 3.5 meters, etc.), normalizing the 3D point clouds of all the obtained internal points, where the normalization operation makes the 3D point clouds of all the internal points have a depth of N meters.

b. If the camera is provided with a 3D point cloud sensor, for example an RGBD camera or a TOF sensor. The method comprises the steps that firstly, 3D point clouds based on a point cloud sensor coordinate system are obtained through the point cloud sensor; secondly, according to the registration of the camera and the point cloud sensor, obtaining the corresponding relation between each pixel in the camera image and each point cloud of the point cloud sensor; and thirdly, according to the pixel range of the ROI area in the camera image, the 3D point cloud of the corresponding area is taken out.

If k ROI areas exist in the current frame image, a 3D point cloud of the area corresponding to the k (k is larger than or equal to 1) group is obtained.

3. The RANSAC algorithm is detailed

The data is composed of "interior points" which are data constituting model parameters, and "exterior points" which are data unsuitable for the model. At the same time RANSAC assumes: given a set of data containing a small fraction of "inliers", there is a procedure that can estimate a model that fits the "inliers". Basic ideas and flow: the model is estimated by iteratively selecting the dataset until a model of demand is estimated. The specific implementation steps are as follows:

a. selecting a minimum data set from which a model can be estimated;

b. computing a data model using the data set;

c. all data are brought into the model, and the number of 'inner points' is calculated; (accumulating data within a certain error range that fits the current iteration push-out model);

d. comparing the number of 'interior points' of the current model and the best model which is pushed out before, and recording model parameters of the maximum 'interior points' number and the 'interior points' number;

e. the first four steps are repeated until the iteration is finished or the current model is the required model (the number of inner points is larger than a certain number).

4. Solving plane equation of point cloud

Firstly, performing outlier rejection and outlier screening on the point cloud extracted in the previous step through a Ranac process, and obtaining plane equation parameters corresponding to the maximum outlier number;

Step two, using the screened interior points, taking the plane equation (corresponding to the first plane equation) corresponding to the maximum interior point number as an initial value, adding Huber robust kernel function, and optimizing the optimal plane equation P by least square _{_opt} (corresponding to the objective plane equation above). The plane equation is represented by the following equation, where A, B, C is the corresponding plane equation parameter:

A*x+B*y+C*z-1＝0

if the 3D point cloud of k groups of corresponding areas exists in the current frame image, k plane equations are obtained:

A1*x+B1*y+C1*z-1＝0

A2*x+B2*y+C2*z-1＝0

…

Ai*x+Bi*y+Ci*z-1＝0

Ak*x+Bk*y+Ck*z-1＝0

wherein the value range of i is [1, 2. ], k ], ai, bi and Ci are parameters corresponding to the ith plane equation.

Step three: pose estimation method based on plane equation

First, the gesture q is at the last moment _{_last} Expressed in terms of quaternions, i.e. q _{_last} ＝q _{_0} +q _{_1} *i+q _{_2} *j+q _{_3} * k, where q _{_0} Is a real part parameter, [ q ] _{_1} ，q _{_2} ，q _{_3} ]Corresponding parameters for three imaginary parts.

Where j, k and m are three imaginary vectors of quaternions. The three imaginary vectors satisfy the following relationship:

second, the gesture q at the last moment is utilized _{_last} Obtain the plane equation (corresponding to the second plane equation) P of the last moment _{_last} : dx+ey+fz-1=0, wherein D, E and F are coefficients of three dimensions of the plane equation, and the corresponding relationship between the coefficients and the quaternion parameters is:

D＝q ₁ q ₃ -q ₀ q ₂

E＝q _o q ₁ +q ₂ q ₃

F＝q ₀ q ₀ +q ₃ q ₃ -0.5

Third, calculate the optimal plane equation P _{_opt} And the plane equation P of the last moment _{_last} Is defined as e= [ e ] _{_x} ，e _{_y} ，e _{_z} ]Wherein [ e ] _{_x} ，e _{_y} ，e _{_z} ]The components of the plane difference (corresponding to the plane equation difference values described above) in the three dimensions x, y and z are:

e _x ＝B*F+C*E

e _y ＝C*D+A*F

e _z ＝A*E+B*D

if there are k plane equations of the 3D point cloud of k sets of corresponding regions in the current frame image, the difference is defined as e=r _{_1} *e _{_1} +r _{_2} *e _{_2} +...+r _{_i} *e _{_i} +r _{_k} *e _{_k} Wherein i has a value in the range of [1,2 ], k]，e _{_i} ＝[e _{_xi} ，e _{_yi} ，e _{_zi} ]Is the i-th set of plane equations and the plane equation P at the previous moment _{_last} Is the difference between (1):

e _xi ＝Bi*F+Ci*E

e _yi ＝Ci*D+Ai*F

e _zi ＝Ai*E+Bi*D

r _{_k} is the uncertainty of the ith plane equation, representing the credibility of the ith plane, which is proportional to the number of points of the 3D point cloud of the ith plane, for example.

Fourth, correcting the angular velocity of the gyroscope. Obtaining an angular velocity signal w from a gyro sensor _{_sensor，} E of the plane equation obtained in the last step is added into the angular velocity component through proportional and integral operation to obtain the corrected angular velocity w _correct ＝[w _x ，w _y ，w _z ]：

w _{_correct} ＝w _{_sensor} +k _{_prop} *e+k _{_integ} *e*Δt

Wherein [ w ] _x ，w _y ，w _z ]An x component, a y component and a z component, k, respectively, of the corrected angular velocity _{_prop} Is a proportionality coefficient, k _{_integ} Is the integral coefficient and Δt is the gyroscope sampling period.

And fifthly, updating the gesture. By means of the corrected angular velocity w _{_correct} For the posture q at the last moment _{_last} Updating by using the corrected angular velocity, and updating the quaternion q _{_correct} The calculation formula of (2) is as follows:

q _{_correct} ＝q _{_last} +Δq

wherein Δq= [ Δq ] ₀ ，Δq ₁ ，Δq ₂ ，Δq ₃ ]To update delta, Δq ₀ Is the real part parameter of the update delta, [ delta ] q ₁ ，Δq ₂ ，Δq ₃ ]Three imaginary parameters of the update delta:

Δq ₀ ＝-(q ₁ *w _x +q ₂ *w _y +q ₃ *w _z )*Δt/2

Δq ₁ ＝(q ₀ *w _x +q ₂ *w _z -q ₃ *w _y )*Δt/2

Δq ₂ ＝(q ₀ *w _y -q ₁ *w _z +q ₃ *w _x )*Δt/2

Δq ₃ ＝(q ₀ *w _z +q ₁ *w _y -q ₂ *w _x )*Δt/2

sixth, calculating a normalized quaternion q corresponding to the quaternion at the current moment _{_norm} ＝[q _{_n0} ，q _{_n1} ，q _{_n2} ，q _{_n3} ]. Wherein q is _{_n0} Is a real part parameter, [ q ] _{_n1} ，q _{_n2} ，q _{_n3} ]Corresponding parameters for three imaginary parts. The calculation formula is as follows:

where j=0, 1,2,3.

Seventh, normalizing the quaternion q at the current moment _{_norm} Converting into Euler angles at the current moment, wherein the conversion formula is as follows:

the euler angle can be obtained based on the quaternion by the following relation,is yaw angle, θ is pitch angle, +.>The roll angle is:

wherein the range of values of arctan and arcsin isThis does not cover all the directions of the euler angles (for pitch angle, it +.>Has been satisfied), therefore, with respect to yaw and roll angles, it is necessary to replace arctan with atan 2: />

Wherein the value range of atan2 is

From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.

The embodiment also provides a device for determining the gesture, which is used for implementing the above embodiment and the preferred implementation, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

Fig. 3 is a block diagram of a configuration of an attitude determination apparatus according to an embodiment of the present invention, as shown in fig. 3, including:

a first determining module 32, configured to detect a first image captured by a first image capturing device provided on a cleaning device, and determine a first ROI area of a first plane included in the first image;

a second determining module 34, configured to determine first 3D point cloud information corresponding to the first ROI area;

a correction module 36 for correcting an angular velocity of the cleaning device based on the first 3D point cloud information;

a third determination module 38 for determining a pose of the cleaning device based on the corrected angular velocity of the cleaning device.

In an exemplary embodiment, the second determining module 34 includes: a first determining unit, configured to determine, when it is determined that the cleaning device includes a target sensor, 3D point cloud information of a target area acquired by the target sensor, where the first image is an image obtained by capturing the target area, and a difference between a capturing time of the first image and a time of the target sensor acquiring the 3D point cloud information of the target area is less than a predetermined threshold; the first matching unit is used for registering all pixel points included in the first image with the 3D point cloud information of the target area; and the second determining unit is used for determining the first 3D point cloud information corresponding to the first ROI area based on the registration result.

In an exemplary embodiment, the second determining module 34 includes: a third determining unit configured to detect a second image captured by the first image capturing apparatus, and determine a second ROI area of a second plane included in the second image, where the second image is a previous frame image of the first image, in a case where it is determined that the target sensor is not included in the cleaning apparatus; an extraction unit for extracting a first 2D feature point of the first ROI region and extracting a second 2D feature point of the second ROI region; the second matching unit is used for matching the first 2D characteristic points with the second 2D characteristic points based on a characteristic point matching mode; and a fourth determining unit, configured to determine the first 3D point cloud information corresponding to the first ROI area based on a matching result.

In one exemplary embodiment, the fourth determining unit includes: a first determination subunit configured to determine a plane constraint of the first image capturing apparatus, and determine a target homography matrix based on the matching result and the plane constraint; a second determination subunit configured to determine a rotation matrix and a translation vector without scale between the first image and the second image based on the target homography matrix; a third determination subunit for determining an absolute position increment between the first image and the second image based on a first sensor and/or a first algorithm of the cleaning device; a fourth determination subunit configured to determine a scaled translation vector based on the non-scaled translation vector and the absolute position increment; and a fifth determining subunit, configured to triangulate the translation vector with the size and the matching result, so as to determine first 3D point cloud information corresponding to the first ROI area.

In one exemplary embodiment, the first determining module 32 includes: and a fifth determining unit, configured to detect the first image by using a neural network model, and determine the first ROI area of the first plane included in the first image, where the neural network model is a model obtained by training an initial neural network model by using training data, where the training data includes multiple sets of data, and each set of data includes a training image and the ROI area of the plane included in the training image.

In one exemplary embodiment, the correction module 36 includes: a sixth determining unit, configured to screen a target 3D point cloud from the first 3D point clouds according to a target screening manner, and determine a first plane equation based on the target 3D point cloud; the generating unit is used for determining a target maximum 3D point cloud from the target 3D point clouds and generating a target plane equation through least square optimization based on the target maximum 3D point cloud and a Huber robust kernel function, wherein the target maximum 3D point cloud is the 3D point cloud which is smaller than a preset distance threshold from the first plane equation; and the correction unit is used for correcting the angular speed based on the target plane equation.

In an exemplary embodiment, the correction unit includes: a sixth determining subunit configured to determine a quaternion of a previous time pose of the cleaning apparatus, and determine a second plane equation of the previous time pose based on the quaternion of the previous time pose; an acquisition unit configured to determine a plane equation difference value between the target plane equation and the second plane equation, and acquire a gyroscope angular velocity measurement value of the cleaning apparatus; and the correction subunit is used for determining an angular velocity correction amount based on the plane equation difference value, and correcting the angular velocity measurement value of the gyroscope based on the angular velocity correction amount to obtain corrected angular velocity.

In an exemplary embodiment, the above apparatus further includes: the updating module is used for updating the quaternion of the previous moment posture based on the corrected angular speed after the angular speed correction is obtained by determining the angular speed correction based on the plane equation difference value and correcting the angular speed measurement value of the gyroscope based on the angular speed correction so as to obtain the quaternion of the updated current moment posture; the conversion module is used for carrying out normalization processing on the quaternion of the gesture at the current moment and converting the quaternion after normalization processing into an Euler angle; and a fourth determining module for determining an updated pose of the cleaning device based on the euler angle.

In an exemplary embodiment, the above apparatus further includes: the calibration module is used for calibrating parameters of a sensor included in the cleaning equipment before detecting a first image shot by a first camera equipment arranged on the cleaning equipment and determining a first ROI (region of interest) of a first plane included in the first image, wherein the parameters are at least one of the following: an internal reference matrix and a distortion model of the first image pickup apparatus; an internal reference of a gyroscope of the cleaning device; an external parameter between the first camera device coordinate system and the gyroscope coordinate system; external parameters between the gyroscope coordinate system and the machine body coordinate system of the cleaning equipment; a point cloud reference model of a target sensor included in the cleaning apparatus; and the external parameters between the target sensor coordinate system and the gyroscope coordinate system.

There is also provided in this embodiment a cleaning apparatus which may include the attitude determination means as described in any one of the above.

It should be noted that each of the above modules may be implemented by software or hardware, and for the latter, it may be implemented by, but not limited to: the modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.

Embodiments of the present invention also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

In the present embodiment, the above-described computer-readable storage medium may be configured to store a computer program for performing the steps of:

s1, detecting a first image shot by a first image shooting device arranged on a cleaning device, and determining a first ROI (region of interest) of a first plane included in the first image;

s2, determining first 3D point cloud information corresponding to the first ROI area;

s3, correcting the angular speed of the cleaning equipment based on the first 3D point cloud information;

s4, determining the posture of the cleaning device based on the corrected angular speed of the cleaning device.

In one exemplary embodiment, the computer readable storage medium may include, but is not limited to: a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.

An embodiment of the invention also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

In an exemplary embodiment, the electronic apparatus may further include a transmission device connected to the processor, and an input/output device connected to the processor.

In an exemplary embodiment, the above-mentioned processor may be arranged to perform the following steps by means of a computer program:

According to the method for determining the attitude of the gyroscope, which is high in accuracy, high in robustness, free of drift over time and independent of the accelerometer, plane detection of a horizontal plane is carried out through camera and/or TOF sensor data of a sweeper (taking the sweeper as an example, other cleaning equipment can be used, such as a floor washing machine and the like), plane constraint is constructed, and then the attitude estimation of the sweeper is carried out by fusing the angular velocity data of a 3-axis or single-axis of the gyroscope.

It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of determining a pose, comprising:

Detecting a first image shot by a first camera arranged on a cleaning device, and determining a first ROI (region of interest) of a first plane included in the first image;

determining first 3D point cloud information corresponding to the first ROI area;

correcting the angular velocity of the cleaning device based on the first 3D point cloud information;

and determining the posture of the cleaning device based on the corrected angular velocity of the cleaning device.

2. The method of determining a pose according to claim 1, wherein determining first 3D point cloud information corresponding to the first ROI area comprises:

under the condition that the cleaning equipment comprises a target sensor, determining 3D point cloud information of a target area acquired by the target sensor, wherein the first image is an image obtained by shooting the target area, and the difference between the shooting time of the first image and the time of the target sensor for acquiring the 3D point cloud information of the target area is smaller than a preset threshold value;

registering all pixel points included in the first image with the 3D point cloud information of the target area;

and determining the first 3D point cloud information corresponding to the first ROI based on a registration result.

3. The method of determining a pose according to claim 1, wherein determining first 3D point cloud information corresponding to the first ROI area comprises:

detecting a second image shot by the first image pickup device under the condition that the cleaning device does not comprise the target sensor, and determining a second ROI (region of interest) of a second plane included in the second image, wherein the second image is a previous frame image of the first image;

extracting a first 2D feature point of the first ROI area, and extracting a second 2D feature point of the second ROI area;

matching the first 2D feature points with the second 2D feature points based on a feature point matching mode;

and determining the first 3D point cloud information corresponding to the first ROI based on a matching result.

4. The method of determining a pose according to claim 3, wherein determining first 3D point cloud information corresponding to the first ROI area based on a matching result comprises:

determining plane constraint of the first camera equipment, and determining a target homography matrix based on the matching result and the plane constraint;

determining a rotation matrix and a translation vector without scale between the first image and the second image based on the target homography matrix;

Determining an absolute position delta between the first image and the second image based on a first sensor and/or a first algorithm of the cleaning device;

determining a scaled translation vector based on the non-scaled translation vector and the absolute position increment;

and triangulating the translation vector with the scale and the matching result to determine first 3D point cloud information corresponding to the first ROI area.

5. The method according to claim 1, wherein detecting a first image captured by a first image capturing apparatus provided on a cleaning apparatus, determining a first ROI area of a first plane included in the first image, includes:

and detecting the first image by using a neural network model, and determining the first ROI of the first plane included in the first image, wherein the neural network model is a model obtained by training an initial neural network model by using training data, the training data comprises a plurality of groups of data, and each group of data comprises a training image and the ROI of the plane included in the training image.

6. The method of determining a pose according to claim 1, wherein correcting an angular velocity of the cleaning device based on the first 3D point cloud information comprises:

Screening a target 3D point cloud from the first 3D point cloud according to a target screening mode, and determining a first plane equation based on the target 3D point cloud;

determining a target maximum 3D point cloud from the target 3D point clouds, and generating a target plane equation through least square optimization based on the target maximum 3D point cloud and a Huber robust kernel function, wherein the target maximum 3D point cloud is a 3D point cloud which is smaller than a preset distance threshold from the first plane equation;

the angular velocity is corrected based on the target plane equation.

7. The method of determining a pose according to claim 6, wherein correcting said angular velocity based on said target plane equation comprises:

determining a quaternion of a previous moment of the cleaning device, and determining a second plane equation of the previous moment of the cleaning device based on the quaternion of the previous moment of the cleaning device;

determining a plane equation difference between the target plane equation and the second plane equation, and obtaining a gyroscope angular velocity measurement of the cleaning device;

and determining an angular velocity correction amount based on the plane equation difference value, and correcting the angular velocity measurement value of the gyroscope based on the angular velocity correction amount to obtain corrected angular velocity.

8. The attitude determination method according to claim 7, wherein after determining an angular velocity correction amount based on the plane equation difference value and correcting the gyroscope angular velocity measurement value based on the angular velocity correction amount, the method further comprises:

updating the quaternion of the previous moment posture based on the corrected angular speed to obtain the quaternion of the updated current moment posture;

normalizing the quaternion of the gesture at the current moment, and converting the quaternion after normalization into an Euler angle;

an updated pose of the cleaning device is determined based on the euler angles.

9. The method according to claim 1, wherein before detecting a first image captured by a first image capturing apparatus provided on a cleaning apparatus, determining a first ROI area of a first plane included in the first image, the method further comprises:

calibrating a parameter of a sensor included in the cleaning apparatus, the parameter being at least one of:

an internal reference matrix and a distortion model of the first image pickup apparatus;

An internal reference of a gyroscope of the cleaning device;

an external parameter between the first camera device coordinate system and the gyroscope coordinate system;

external parameters between the gyroscope coordinate system and the machine body coordinate system of the cleaning equipment;

a point cloud reference model of a target sensor included in the cleaning apparatus;

and the external parameters between the target sensor coordinate system and the gyroscope coordinate system.

10. A posture determining device, characterized by comprising:

a first determining module, configured to detect a first image captured by a first image capturing device provided on a cleaning device, and determine a first ROI area of a first plane included in the first image;

a second determining module, configured to determine first 3D point cloud information corresponding to the first ROI area;

the correction module is used for correcting the angular speed of the cleaning equipment based on the first 3D point cloud information;

and a third determining module for determining the posture of the cleaning device based on the corrected angular velocity of the cleaning device.

11. A cleaning apparatus comprising the attitude determination means of claim 10.

12. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored program, wherein the program when run performs the method of any of the preceding claims 1 to 9.

13. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method according to any of the claims 1 to 9 by means of the computer program.