WO2022193559A1

WO2022193559A1 - Projection correction method and apparatus, storage medium, and electronic device

Info

Publication number: WO2022193559A1
Application number: PCT/CN2021/115160
Authority: WO
Inventors: 孙世攀; 张聪; 胡震宇
Original assignee: 深圳市火乐科技发展有限公司
Priority date: 2021-03-19
Filing date: 2021-08-27
Publication date: 2022-09-22
Also published as: CN112689135B; CN112689135A

Abstract

The present disclosure relates to the technical field of projection, and relates to a projection correction method and apparatus, a storage medium, and an electronic device. The method comprises: acquiring, by means of a camera, a preset image projected by a projector to obtain a photographed image; determining target feature points in the photographed image, and determining, for each target feature point, depth information of the target feature point in a photographing space of the camera to obtain three-dimensional coordinates of the target feature point; and determining, according to the three-dimensional coordinates of all target feature points, a normal vector of a fitting plane constructed by all the target feature points, and correcting an original image of the projector according to the normal vector and current pose information of the projector. The beneficial effects of the present disclosure are: keystone correction of the projection can be realized by means of one camera, so that not only the number of arranged devices is decreased, but also the depth information of the target feature points can be rapidly calculated; and the complexity of calculating the three-dimensional coordinates of the target feature points is reduced.

Description

Projection correction method, device, storage medium and electronic device

This disclosure claims the priority of the Chinese patent application with the application number 202110297235.1 and the invention titled "Projection Correction Method, Apparatus, Storage Medium and Electronic Device" filed with the China Patent Office on March 19, 2021, the entire contents of which are incorporated by reference in this disclosure.

technical field

The present disclosure relates to the field of projection technology, and in particular, to a projection correction method, an apparatus, a storage medium, and an electronic device.

Background technique

A projector is a device that projects an image onto a wall or projection screen by optical projection. In traditional projectors, the projector needs to be facing the wall to ensure that the projected picture is a normal rectangle. Once the projector is placed improperly, the projected picture will be distorted.

At present, the keystone correction technology of projectors is mainly based on binocular correction. However, binocular correction requires the use of two cameras or a distance sensor to match the camera, and additional cameras or distance sensors are used for projectors. It is said that the hardware cost will be increased, and the calibration parameters will be more.

SUMMARY OF THE INVENTION

In order to overcome the problems existing in the related art, the present disclosure provides a projection correction method, device, storage medium and electronic device.

In order to achieve the above objects, in a first aspect, the present disclosure provides a projection correction method, the method comprising:

In response to the received correction instruction, controlling the projector to project a preset image to the projection plane;

Capture the preset image projected by the projector by using the camera of the projector to obtain a captured image;

In the captured image, identify the target feature point of the preset image;

For each target feature point, determine the depth of the target feature point in the shooting space of the camera according to the mapping relationship pre-calibrated for the target feature point and the camera coordinates of the target feature point on the captured image information to obtain the three-dimensional coordinates of the target feature point in the projection space of the projector, wherein the mapping relationship is between the depth information of the target feature point calibrated at different depths and the offset of the camera coordinates connection relation;

Determine the normal vector of the projection plane relative to the projector according to the three-dimensional coordinates of each of the target feature points;

Correcting the two-dimensional vertex coordinates of the original image of the projector according to the normal vector of the projection plane and the current pose information of the projector, to obtain the two-dimensional vertex coordinates of the corrected original image;

The projector is controlled to project according to the two-dimensional vertex coordinates of the corrected original image.

In a second aspect, the present disclosure provides a projection correction device, the device comprising:

a response module, configured to control the projector to project a preset image to the projection plane in response to the received correction instruction;

a photographing module, configured to photograph the preset image projected by the projector through the camera of the projector to obtain a photographed image;

a target feature point determination module, configured to identify the target feature point of the preset image in the captured image;

The three-dimensional coordinate determination module is configured to, for each of the target feature points, determine that the target feature point is in the The depth information in the shooting space of the camera to obtain the three-dimensional coordinates of the target feature point in the projection space of the projector, wherein the mapping relationship is the depth information and camera coordinates of the target feature point calibrated at different depths The relationship between the offsets;

a normal vector module, configured to determine the normal vector of the projection plane relative to the projector according to the three-dimensional coordinates of each of the target feature points;

a correction module, configured to correct the two-dimensional vertex coordinates of the original image of the projector according to the normal vector of the projection plane and the current pose information of the projector to obtain the two-dimensional vertex of the corrected original image coordinate;

The projection module controls the projector to project according to the two-dimensional vertex coordinates of the corrected original image.

In a third aspect, the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processing device, implements the steps of the method provided in the first aspect.

In a fourth aspect, the present disclosure provides an electronic device, comprising:

a memory on which a computer program is stored;

A processor, configured to execute the computer program in the memory, to implement the steps of the method provided in the first aspect above.

Through the above technical solution, in response to the received correction instruction, a preset image projected by the projector is collected through a camera to obtain a shot image; target feature points are determined in the shot image, and for each target feature point, according to the target feature point The pre-calibrated mapping relationship of the feature points and the camera coordinates of the target feature point on the captured image determine the depth information of the target feature point in the shooting space of the camera to obtain the three-dimensional coordinates of the target feature point; According to the three-dimensional coordinates of all target feature points, the normal vector of the projection plane is determined, and the two-dimensional vertex coordinates of the original image of the projector are corrected according to the normal vector and the current pose information of the projector. In this way, projection trapezoidal correction can be achieved through one camera, which not only reduces the number of device settings, but also can quickly calculate the depth information of the target feature points through the pre-calibrated mapping relationship, reducing the calculation of the three-dimensional coordinates of the target feature points. The complexity, further, improves the correction efficiency.

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

Description of drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the attached image:

1 is a flowchart of a projection correction method according to an exemplary embodiment;

2 is a schematic flowchart of identifying target feature points according to an exemplary embodiment;

3 is a schematic diagram of mathematical modeling of a mapping relationship shown according to an exemplary embodiment;

4 is another flowchart of a projection correction method according to an exemplary embodiment;

FIG. 5 is a schematic diagram showing the principle of calculating three-dimensional imaging vertex coordinates of a standard image according to an exemplary embodiment;

6 is a schematic diagram of the principle of vector decomposition according to an exemplary embodiment;

7 is a block diagram of a projection correction apparatus according to an exemplary embodiment;

Fig. 8 is a block diagram of an electronic device according to an exemplary embodiment.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for the purpose of A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

Fig. 1 is a flowchart of a projection correction method according to an exemplary embodiment. The projection correction method can be applied to electronic equipment such as projectors. As shown in FIG. 1 , the projection correction method includes the following steps:

S101, in response to the received correction instruction, control the projector to project a preset image to the projection plane.

Here, the preset image refers to an image projected onto the projection plane, and generally, the preset image in this embodiment may adopt a checkerboard image. Wherein, the projection plane refers to an area, such as a wall or a curtain, that is used to display the output image of the projector. It should be understood that when the projector is perpendicular to the wall or the curtain, the preset image is a standard rectangular image, and the checkerboard image is not distorted at this time, and when the projector is not perpendicular to the wall or the curtain, the projected image is not Rectangular image, the checkerboard image is distorted at this time.

In some embodiments, the correction instruction may or may not be automatically triggered. For example, if it is automatically triggered, when the projector self-detects that the projected image is a non-rectangular image, the projector can automatically trigger a correction command; if it is not automatically triggered, the user can press the controller that communicates with the projector. The button is used to trigger the controller to send a correction command to the projector. The button can be a virtual button or a physical button. This embodiment does not limit this.

S102, the preset image projected by the projector is captured by the camera of the projector to obtain a captured image.

Here, after the projector projects the preset image to the projection plane, the preset image is photographed by the camera to obtain the photographed image, so as to model the wall or curtain projected by the projector according to the photographed image to obtain the wall or the curtain three-dimensional information.

S103, in the captured image, identify the target feature point of the preset image.

Here, the target feature point is the feature point set on the preset image and used for modeling the wall or the curtain, and the feature point can be set in form or quantity according to the actual situation. For example, when the preset image is a checkerboard image, the target feature points in the preset image refer to intersections between black and white grids in the preset image.

S104, for each target feature point, according to the pre-calibrated mapping relationship for the target feature point and the camera coordinates of the target feature point on the captured image, determine that the target feature point is in the shooting space of the camera to obtain the three-dimensional coordinates of the target feature point in the projection space of the projector, wherein the mapping relationship is the difference between the depth information of the target feature point calibrated at different depths and the offset of the camera coordinates relationship between.

Here, due to the correlation between the depth information of the target feature points calibrated at different depths in advance and the offset of the camera coordinates, after the camera coordinates of the target feature points are determined in the captured image, the camera coordinates can be based on the camera coordinates. And the mapping relationship calculates the depth of the target feature point. The depth information refers to the depth of the target feature point of the preset image projected on the projection plane relative to the camera. For example, the preset images projected by the projector are taken at depths of 1.2m and 1.9m, respectively, and the camera coordinates of the target feature points at 1.2m and 1.9m are determined, so as to calculate the depth information of the same target feature point and the camera. The relationship between coordinates.

It should be noted that, after the camera coordinates and depth information of the target feature coordinate point are calculated, the depth information of the target feature point in space can be determined according to the camera coordinates and the depth information. The depth information is the Z-axis coordinate in the three-dimensional coordinates of the target feature point. The X-axis coordinates and the Y-axis coordinates of the three-dimensional coordinates of the target feature point on the captured image are obtained by scaling according to the camera coordinates of the target feature point on the preset image, combined with the depth information of the target feature point. The specific principle is as follows: the preset image is projected on the projection plane through the principle of pinhole imaging, and the X-axis coordinates and Y-axis coordinates of the target feature points displayed on the projection plane are in the camera coordinates of the target feature on the preset image. The X-axis coordinates and Y-axis coordinates of , are obtained by converting the depth information.

S105: Determine the normal vector of the projection plane relative to the projector according to the three-dimensional coordinates of each of the target feature points.

Here, the obtained three-dimensional coordinates of all target feature points can be fitted, so as to model the projection plane according to the three-dimensional coordinates of the target feature points to obtain a fitted plane, which reflects the three-dimensional information of the projection plane, and then Get the normal vector of the projection plane relative to the projector. Wherein, the fitting may be fitting using the least squares method.

S106, correct the two-dimensional vertex coordinates of the original image of the projector according to the normal vector of the projection plane and the current pose information of the projector to obtain the two-dimensional vertex coordinates of the corrected original image.

Here, the normal vector of the projection plane refers to a line segment perpendicular to the projection plane, and the current pose information of the projector refers to the current placement position of the projector, and the current pose information can be obtained through an attitude sensor (IMU). When the projector is projecting, if the placement position is skewed, the projected light of the projector will be offset relative to the wall or curtain, which will cause the projector to display keystone distortion on the projection plane of the wall or curtain. According to the normal vector of the projection plane and the current pose information of the projector, the position deviation of the projector relative to the projection plane (wall or curtain) can be calculated, and then the two-dimensional vertex coordinates of the original image of the projector can be calculated according to the position deviation. make adjustments.

The original image refers to the original output image of the projector. Generally speaking, the original image is a rectangular image, such as an image with a width of w and a height of h. When the projector is tilted relative to the projection plane, the original rectangular image projected on the projection plane will appear as an irregular quadrilateral, such as a convex quadrilateral. In order to make the image projected by the projector on the projection plane appear as a rectangle, it is necessary to Correct the two-dimensional vertex coordinates of the original image, so that the corrected original image is projected as a rectangle on the projection plane.

It should be understood that the two-dimensional vertex coordinates refer to the four vertex coordinates of the original image. Correcting the two-dimensional vertex coordinates of the original image may be in the form of digital adjustment, which only changes the vertex coordinates of the original image output by the projector without changing the lens structure of the projector.

S107, controlling the projector to project according to the two-dimensional vertex coordinates of the corrected original image.

Here, the projector is controlled to project according to the two-dimensional vertex coordinates of the corrected original image, so that the original image of the projector appears as a rectangle on the projection plane.

With the above technical solution, projection trapezoidal correction can be realized through one camera, which not only reduces the number of device settings, but also can quickly calculate the depth information of the target feature points through the pre-calibrated mapping relationship, reducing the time required for calculating the target feature points. The complexity of the three-dimensional coordinates further improves the correction efficiency.

In the present disclosure, when the preset image is a checkerboard image, it is considered that the camera of the camera will be affected by environmental factors, such as exposure, during actual shooting, resulting in a black and white image with a checkerboard. There are differences in the regions, and the feature points depend on the image quality. Under this difference, the feature points determined by the conventional corner detection algorithm are inaccurate, which in turn leads to inaccurate parameters obtained from the feature points later. the accuracy of the correction. Therefore, the initial feature points detected by the corner detection algorithm are optimized, and the following describes in detail the target feature points identified in the preset image in S103.

FIG. 2 is a schematic flowchart of identifying target feature points according to an exemplary embodiment. As shown in FIG. 2, in an achievable embodiment, in S103, identifying the target feature points of the preset image, including:

S1031, using a corner detection algorithm to determine the initial feature points in the checkerboard image;

S1032, according to the vertical direction and the horizontal direction, perform straight line fitting on the initial feature points within the same preset area range;

S1033 , taking an intersection between any straight line in the vertical direction and any straight line in the horizontal direction obtained by fitting as the target feature point.

Here, first, the corner detection algorithm is used to determine the initial feature points in the preset image, and then the initial feature points in the same preset area are linearly fitted according to the vertical and horizontal directions respectively; The intersection between any obtained straight line in the vertical direction and any straight line in the horizontal direction is taken as the target feature point.

In the present disclosure, since the offset of the initial feature point is also a small-range offset, the same preset area range is a range that can be determined in advance. Among them, the straight line fitting can be realized by the least square method.

It should be understood that, in the case where the projector is perpendicular to the wall, the black and white intersection of the projected image is formed by the intersection of two straight lines. In this ideal case, the number of horizontal and vertical lines is known. Correspondingly, the number of straight lines in the horizontal direction and the straight lines in the vertical direction obtained by fitting is the same as the number in the ideal situation, and the number of feature points is also the same.

The corner detection algorithm may be, for example, the findChessboardCorners algorithm of OpenCV.

By adopting the above technical solution, the initial feature points detected by the corner point detection algorithm are optimized, the accuracy of the target feature points of the preset image is ensured, and the correction accuracy is improved.

In an achievable embodiment, in S104, for each target feature point, the target is determined according to the pre-calibrated mapping relationship for the target feature point and the camera coordinates of the target feature point on the captured image The depth information of the feature points in the shooting space of the camera, including:

For each of the target feature points, according to the pre-calibrated mapping relationship for the target feature point and the camera coordinates of the target feature point on the captured image, calculate the distance of the target feature point in the shooting space of the camera. depth information, wherein the mapping relationship is:

Wherein, h is the depth information of the target feature point, p ₁ is the first preset calibration parameter of the target feature point, p ₂ is the second preset calibration parameter of the target feature point, and X is the camera coordinate of the target feature point , wherein the first preset calibration parameter and the second preset calibration parameter are both constants.

Here, after calculating the camera coordinates of the target feature point, the camera coordinates are substituted into the above calculation formula, and then the depth information of the target feature point can be calculated. Among them, in the above calculation formula, p ₁ and p ₂ are a constant.

It is worth noting that X can be the abscissa or ordinate of the camera coordinates, and the abscissa and the ordinate are symmetrical in the operation process. Therefore, the above calculation formula is applicable to the abscissa or the ordinate of the camera coordinates.

Therefore, in the present disclosure, through the pre-calibrated mapping relationship, the depth information of the target feature point can be quickly calculated, thereby calculating the three-dimensional coordinates of the target feature point in space.

Next, the calibration process of the first preset calibration parameter and the second preset calibration parameter will be described.

For each target feature point, the first preset calibration parameter and the second preset calibration parameter of the target feature point are obtained in the following manner:

When the projector is at a first distance from the projection plane, and the projection light of the projector is perpendicular to the projection plane, a preset image is projected to the projection plane, and the image is captured by the camera of the projector. the preset image to obtain the first image;

determining, based on the first image and the first distance, first camera coordinates and first depth information of the target feature point on the first image;

When the projector is a second distance away from the projection plane, and the projection light of the projector is perpendicular to the projection plane, a preset image is projected to the projection plane, and passes through the camera of the projector photographing the preset image to obtain a second image;

determining, according to the second image and the second distance, second camera coordinates and second depth information of the target feature point on the preset image;

According to the first camera coordinates and the first depth information, as well as the second camera coordinates and the second depth information, combined with the pre-established mapping relationship between the depth information of the target feature point and the camera coordinates, the first camera coordinates are obtained. The preset calibration parameter and the second preset calibration parameter, wherein the mapping relationship is:

Wherein, h is the depth coordinate of the target feature point, X is the camera coordinate, p ₁ is the first preset calibration parameter, and p ₂ is the second preset calibration parameter.

Here, the process of calibrating the mapping relationship is actually projecting preset images at two different depths, and using the camera to capture the preset images at the different depths, for example, setting the projection light of the projector to be perpendicular to the Projection plane (wall or curtain), and then the projector projects preset images to the projection planes at distances of 1.2m and 1.9m respectively, and uses the camera to shoot to obtain the first image and the second image. Then, the camera coordinates and depth information of the target feature point in the first image are calculated, wherein the first distance is equivalent to the depth information of the target feature point in the first image. The calculation process of the depth information of the target feature points of the second image and the camera coordinates is the same as that of the first image, and details are not repeated here. It is worth noting that the camera coordinate is a two-dimensional coordinate.

After calculating the camera coordinates and depth information of the target feature points at two depths, substitute them into the following formula:

The values of the first preset calibration parameter and the second preset calibration parameter can be obtained by calculation.

The above embodiment will be described below with reference to FIG. 3 .

FIG. 3 is a schematic diagram of mathematical modeling of a mapping relationship according to an exemplary embodiment. In FIG. 3, point H is the position of a certain target feature point at a first distance, and point I is the target feature point at a second distance. The position of , K point is the point where the target feature point at the second distance position is projected to the camera normalization plane, J point is the point where the target feature point at the first distance position is projected to the camera normalization plane, and L point is Auxiliary point, used to describe the depth of point H from the normalized plane of the camera, point Q is the point where the target feature point is projected on the normalized plane of the camera, point D is the origin of the camera, point A is the origin of the projector, and point M is The auxiliary point is used to describe the depth of the H point from the camera origin plane, and the N point is the position of the target feature point in the original image.

According to the mathematical geometric relationship, it can be obtained: JQ/DN=HL/HM;

The abscissa of point Q is marked as x _q , the abscissa of point H (target feature point) to the projection point J of the camera is marked as X, the length of LM is marked as f, the length of HM is marked as h, and the length of DN is marked as b , then transform JQ/DN=HL/HM to get:

(x _q -X)/b=(hf)/h;

Further, simplify the above formula to:

h=b*f /( ( _bxq )+X);

Further, the above formula can be seen as:

h=p ₁ /(p ₂ +X);

The relationship between h and X can be obtained from the above formula, where h is the depth information of the target feature point, p ₁ is the first preset calibration parameter of the target feature point, and p ₂ is the second preset calibration parameter of the target feature point parameter, X is the abscissa of the target feature point. Therefore, the values of the first preset calibration parameter and the second preset calibration parameter can be obtained by substituting the first camera coordinates and the first depth information, and the second camera coordinates and the second depth information into the above calculation formula.

FIG. 4 is another flowchart of a projection correction method according to an exemplary embodiment. As shown in FIG. 4, in an achievable embodiment, in S106, according to the normal vector of the projection plane and the current pose information of the projector, the two-dimensional vertex coordinates of the original image of the projector are Perform correction to obtain the two-dimensional vertex coordinates of the corrected original image, including:

S1061, according to the normal vector of the projection plane and the current pose information of the projector, calculate and obtain the offset information of the normal vector of the projector, wherein the offset information includes yaw angle, pitch angle and roll angle corner;

S1062, based on the offset information, calculate the two-dimensional vertex coordinates of the projected image projected by the projector to the projection plane;

S1063, based on the two-dimensional vertex coordinates of the projected image and the two-dimensional vertex coordinates of the original image of the projector, establish a homography matrix relationship between the projected image and the original image;

S1064, select a target rectangle from the projected image of the projector, and determine the two-dimensional vertex coordinates of the target rectangle;

S1065 , based on the two-dimensional vertex coordinates of the target rectangle and in combination with the homography matrix relationship, obtain the two-dimensional vertex coordinates of the corrected original image.

Here, in step S1061, the offset information of the projector is calculated according to the normal vector of the projection plane and the current pose information of the projector. Among them, the roll angle can be obtained by an inertial sensor (Inertial Measurement Unit, IMU) set on the projector, that is, the IMU obtains the current pose information of the projector, and then calculates the roll angle according to the current pose information. The calculation method is as follows: The prior art is not described in detail here. The yaw angle and pitch angle can be calculated according to the normal vector of the projection plane. The specific method is to project the normal vector of the projection plane on the plane where the yaw angle is located, and then calculate the included angle of the projection, which is the yaw angle. For example, if the XOY plane is the plane where the yaw angle is located, the angle between the projection and the X axis is calculated. Project the normal vector of the projection plane on the plane where the pitch angle is located, and then calculate the included angle of the projection, which is the pitch angle. For example, if the XOZ plane is the plane where the yaw angle is located, the angle between the projection and the X axis is calculated.

In step S1062, the vertex coordinates of the projected image are calculated based on the offset information, which is to calculate the vertex coordinates of the original image of the projector projected onto the wall or curtain, and the vertex coordinates are two-dimensional coordinates.

Among them, the two-dimensional vertex coordinates of the projected image can be calculated by the following steps:

First, according to the yaw angle and the pitch angle, the first preset calculation formula is used to obtain the measurement normal vector of the projected image relative to the projector, wherein the first preset calculation formula is:

Wherein, X ₁ is the X-axis coordinate of the measurement normal vector, Y ₁ is the Y-axis coordinate of the measurement normal vector, Z ₁ is the Z-axis coordinate of the measurement normal vector, H is the yaw angle, V is the pitch angle.

Then, based on the measurement normal vector and the coordinate information of the preset target point, the position information of the plane where the projection image is located is determined, wherein the target point is a preset center point where the projection image is rotated.

Here, since the target point is the preset center point where the preset projection image is rotated, such as yaw, pitch, and roll, the coordinate information of the target point is unchanged. After the measurement normal vector and the target point are determined, the position information of the plane where the projected image is located can be determined.

Next, based on the position information and in combination with a pre-established ray vector, the three-dimensional vertex coordinates of the projected image are obtained, wherein the ray vector is the vertex of the projected image projected by the projector and the optical center of the projector Unit vector of lines between .

Here, the ray vector refers to the unit vector of the line between the vertex of the projected image projected by the projector and the optical center of the projector, that is, the projector projects the projected image outward, and the four vertices of the projected image projected by the projector are connected to the optical center. The connection between them can be determined. After determining the position information of the plane where the projection image is located, the intersection of the ray vector and the plane where the projection image is located can be determined through the ray vector, and the intersection point is the coordinates of the four vertices of the projection image projected by the original image on the projection plane.

Among them, the ray vector can be calculated according to the roll angle, and the specific calculation method is as follows:

obtaining the optical-mechanical parameters of the projector, wherein the optical-mechanical parameters include a rising angle, a projection ratio, and an aspect ratio of the projected light;

According to the optical-mechanical parameters of the projector, the three-dimensional imaging vertex coordinates of the standard image projected by the projector on the projection plane under preset conditions are obtained, wherein the preset conditions are that the projector is placed horizontally, the The projection light of the projector is perpendicular to the projection plane, and the projector is separated from the projection plane by a preset distance threshold;

According to the three-dimensional imaging vertex coordinates of the standard image, the unit vector of the connecting line between the vertex of the standard image and the optical center of the projector is calculated, and the unit vector is used as the ray vector.

Here, the projector will change the similarity of the projected images due to the depth. For example, the projected image projected onto the projection plane is a rectangle. Regardless of the depth, the projected image is always a rectangle. Therefore, if the projector projects to the projection plane under the preset conditions, the three-dimensional imaging vertex coordinates of the standard image projected under the preset conditions can be calculated according to the optical-mechanical parameters of the projector. The rising angle refers to the rising angle of the projected light of the projector, and in general, the rising angle is related to the model of the projector.

The specific process of calculating the three-dimensional imaging vertex coordinates of the standard image is as follows:

Fig. 5 is a schematic diagram showing the principle of calculating the three-dimensional imaging vertex coordinates of a standard image according to an exemplary embodiment. As shown in Figure 5, the standard image has four vertices, namely the first vertex 0, the second vertex 1, the third vertex 2, and the fourth vertex 3, wherein the first vertex is the vertex located in the upper right corner of the projected image, The second vertex is the vertex located at the upper left corner of the projected image, the third vertex is the vertex located at the lower right corner of the projected image, and the fourth vertex is the vertex located at the lower left corner of the projected image.

According to the optomechanical parameters, the preset distance threshold is defined as f, the throw ratio is throwRatio, w is the width of the projected image, and h is the height of the projected image. According to the triangular relationship, there is throwRatio=f/w. but

Since throwRatio=f/w, aspectRatio=w/h, then h=f/throwRatio, therefore,

Then the three-dimensional imaging vertex coordinates of the first vertex 0 are:

rcCoodinate[0][0]=(-1)*f*tan(θ)

srcCoodinate[0][1]=f*tan(γ)+f*tan(dOffsetAngle)

rcCoodinate[0][2]=f

The three-dimensional imaging vertex coordinates of the second vertex 1 are:

srcCoodinate[1][0]=(-1)*srcCoodinate[0][0]

srcCoodinate[1][1]=srcCoodinate[0][1]

srcCoodinate[1][2]=f

The three-dimensional imaging vertex coordinates of the third vertex 2 are:

srcCoodinate[2][0]=srcCoodinate[0][0]

srcCoodinate[2][1]=f*tan(dOffsetAngle)

srcCoodinate[2][2]=f

The three-dimensional imaging vertex coordinates of the fourth vertex 3 are:

srcCoodinate[3][0]=(-1)*srcCoodinate[0][0]

srcCoodinate[3][1]=f*tan(dOffsetAngle)

srcCoodinate[3][2]=f

Wherein, srcCoodinate[0][0] is the X-axis coordinate of the first vertex, f is the preset distance threshold, _doffsetAngle is the rising angle, and srcCoodinate[0][1] is the first The Y-axis coordinate of the vertex, srcCoodinate[1][0] is the X-axis coordinate of the second vertex, srcCoodinate[1][1] is the Y-axis coordinate of the second vertex, and srcCoodinate[0][2] is the Y-axis coordinate of the second vertex The Z-axis coordinate of the first vertex, srcCoodinate[1][2] is the Z-axis coordinate of the second vertex, srcCoodinate[2][0] is the X-axis coordinate of the third vertex, and srcCoodinate[2] [1] is the Y-axis coordinate of the third vertex, srcCoodinate[2][2] is the Z-axis coordinate of the third vertex, srcCoodinate[3][0] is the X-axis coordinate of the fourth vertex, srcCoodinate[3][1] is the Y-axis coordinate of the fourth vertex, and srcCoodinate[3][2] is the Z-axis coordinate of the fourth vertex.

After the three-dimensional imaging vertex coordinates of the standard image are calculated, vectors can be used to calculate the optical center of the projector and the four ray vectors of the four vertices. The unit vector is the ray vector of the vertex divided by the modulo of the ray vector.

It should be understood that the ray vector is related to the opto-mechanical parameters of the projector, and the ray vector is unchanged under the condition that the opto-mechanical parameters of the projector do not change.

In step 1544, vector decomposition is performed on the three-dimensional imaging vertex coordinates of the projected image to obtain the two-dimensional imaging vertex coordinates of the projected image.

Here, after calculating the three-dimensional imaging vertex coordinates of the projected image, it is necessary to convert the three-dimensional imaging vertex coordinates of the four vertices into two-dimensional two-dimensional imaging vertex coordinates based on vector decomposition. The specific method is to decompose the vector into basis vectors on the horizontal plane, for example,

is a pair of basis vectors,

Find the basis vector as the X-axis of the coordinate system for the intersection of the projected image and the horizontal plane,

and

vertical. in,

It can be calculated by the following formula:

cosslineU=horizonPlanN×rotatePlanN

Among them, horizonPlanN is the normal vector of the horizontal plane, × is the cross product of the vector, rotatePlanN is the normal vector of the projected image, and norm(coSSlineU) is the modulus of the vector cosslineU.

Fig. 6 is a schematic diagram showing the principle of vector decomposition according to an exemplary embodiment. As shown in FIG. 6 , the projection image has four vertices in total, G, I, J, and H. After the three-dimensional imaging vertex coordinates of the projected image are obtained, a coordinate system is established with any point G, I, J and H as the coordinate origin to convert the three-dimensional imaging vertex coordinates into two-dimensional imaging vertex coordinates. In the present disclosure, a coordinate system is established with point H as the coordinate origin, and the process of vector decomposition to calculate the coordinates of two-dimensional imaging vertexes is described in detail. Then, the following formula can be used to convert the three-dimensional imaging vertex coordinates of points G, I, and J into two-dimensional imaging vertex coordinates.

vectorP=point3D-pointOrigin

Among them, x is the X-axis coordinate of the two-dimensional imaging vertex coordinate, vectorP(0) is the X-axis coordinate of the vector vectorP,

for

The Y-axis coordinate of ,

for

The X-axis coordinate of vectorP(1) is the Y-axis coordinate of the vector vectorP,

for

the X-axis coordinate,

for

The Y-axis coordinate of , y is the Y-axis coordinate of the two-dimensional imaging vertex coordinate, point3D is the three-dimensional imaging vertex coordinate of the vertex to be solved, such as any vertex in G, I, J, pointOrigin is the coordinate of the H point, and vectorP is the HG One of vector, HJ vector and HI vector. For example, when solving the two-dimensional imaging vertex coordinates of point G, point3D is the three-dimensional imaging vertex coordinates of point G, and vectorP is the HG vector.

Therefore, by the above calculation formula, the three-dimensional imaging vertex coordinates of the projected image can be converted into the two-dimensional imaging vertex coordinates of the projected image.

In some achievable embodiments, after obtaining the three-dimensional imaging vertex coordinates of the standard image projected by the projector on the projection plane under preset conditions according to the optical-mechanical parameters of the projector, the method further includes:

obtain the current roll angle of the projector;

When the current roll angle does not meet the preset threshold, according to the current roll angle, combined with the second preset calculation formula, the X-axis coordinate and the Y-axis coordinate in the three-dimensional imaging vertex coordinates of the standard image are corrected, Wherein, the second preset calculation formula is:

ansP _[i][x] =(anyP _[i][x] -rotateP.x)*cos(-r)-(anyP _[i][y] -rotateP.y)*sin(-r)+rotateP. x

ansP _[i][y] =(anyP _[i][x] -rotateP.x)*sin(-r)-(anyP _[i][y] -rotateP.y)*cos(-r)+rotateP. y

Wherein, ansP _[i][x] is the corrected X-axis coordinate of the ith vertex of the standard image, ansP _[i][y] is the corrected Y-axis of the ith vertex of the standard image Coordinates, anyP _[i][x] is the X-axis coordinate of the ith vertex of the standard image before correction, anyP _[i][y] is the Y-axis of the ith vertex of the standard image before correction coordinates, rotateP.x is the X-axis coordinate of the rotation center of the projector rolling, rotateP.y is the Y-axis coordinate of the rotation center, and r is the current roll angle;

The corrected X-axis coordinates and Y-axis coordinates are used as new X-axis coordinates and Y-axis coordinates of the vertices of the standard image.

Here, the current roll angle of the projector can be obtained through an inertial sensor (Inertial Measurement Unit, IMU) set on the projector. When the current roll angle does not meet the preset threshold, it means that the projector has rolled. For example, if the current roll angle is not 0, it means that the projector has a roll rotation. When the projector rolls, its standard image will roll with the optical center ray as the rotation axis, and the X-axis and Y-axis coordinates of the three-dimensional imaging vertex coordinates of the standard image will change. The design formula calculates the X-axis and Y-axis coordinates of the three-dimensional vertex coordinates of the rolling standard image, and obtains the corrected X-axis and Y-axis coordinates of each vertex, thereby obtaining the new three-dimensional imaging vertex coordinates of the standard image. Then, the ray vector is recalculated based on the new three-dimensional imaging vertex coordinates, and the three-dimensional imaging vertex coordinates of the projected image are solved.

It should be understood that the coordinates of the rotation center rotateP can be (0, 0), the rotation center rotateP refers to the rotation center of the projector to roll, and the above-mentioned preset center point is an imaginary projector when yaw and pitch occur. The offset of the projected image after rotation.

As a result, the roll angle can take into account the change of the projector's rotated projection image after sending the roll, so as to achieve accurate keystone correction.

In step S1063, a homography is a concept in projective geometry, also called a projective transformation. It is to map a point (three-dimensional homogeneous vector) on a projective plane to another projective plane. Assuming that the homography matrix relationship between the two images is known, the image of one plane can be transformed to the other plane. The transformation through the plane is to perform projection correction on the same plane. Therefore, after knowing the two-dimensional vertex coordinates of the original image of the projector and the two-dimensional vertex coordinates of the projected image, the corresponding homography matrix relationship can be constructed. The homography matrix relationship refers to the mapping of the original image of the projector on the wall. The relationship between projected images of a surface or curtain.

In step S1064, the target rectangle refers to a rectangle selected in the area of the projected image, and the rectangle may be the rectangle with the largest area in the area in the projected image of the projector. By setting the target rectangle to the rectangle with the largest area, it is to maximize the projected area and improve the user experience.

It should be understood that, in the above-mentioned embodiments, an embodiment of calculating the two-dimensional vertex coordinates of the projected image is proposed. Use other methods to compute the 2D vertex coordinates of the projected image. For example, based on the offset information and the vertex coordinates of the original image, the vertex coordinates of the rotated original image are calculated. Among them, the vertex coordinates of the rotated original image refer to the vertex coordinates of the original image after the yaw angle, pitch angle and roll angle are rotated, and then based on the calculated projection depth of the projector, calculate the rotated The vertex coordinates of the original image are mapped to the 2D vertex coordinates of the projected image of the projected plane. The projection depth refers to the distance between the projector and the projection plane.

Wherein, in an achievable implementation manner, in step S1063, selecting a target rectangle from the projected image may include:

A point is arbitrarily selected from any side of the projected image, and the point is taken as the vertex of the rectangle to be constructed, and the aspect ratio of the original image is taken as the aspect ratio of the rectangle to be constructed. Generate a rectangle within the area of the image;

The rectangle with the largest area is selected from the generated rectangles as the target rectangle.

Here, the specific method of selecting the target rectangle may be to arbitrarily select a point on either side of the projected image, and use the point as the vertex of the rectangle to be constructed, and the aspect ratio of the original image as the aspect ratio of the rectangle to be constructed. A rectangle is generated in the area of the projected image, and the rectangle with the largest area is selected as the target rectangle from the generated rectangles.

For example, traverse the longest side of the projected image and the side adjacent to the longest side, select any point as the vertex of the rectangle to be constructed, and generate a rectangle with an aspect ratio consistent with the original image around the projected image. , at the end of the traversal, find the rectangle with the largest area from all the generated rectangles as the target rectangle.

Therefore, by selecting the rectangle with the largest area as the target rectangle, it can be ensured that the projected image area viewed by the user is the largest, thereby improving the viewing experience of the user.

In step S1065, after obtaining the two-dimensional vertex coordinates of the target rectangle, the two-dimensional vertex coordinates of the target rectangle can be used as the input value of the homography matrix relationship, and the two-dimensional vertex coordinates of the corrected original image can be obtained by calculating, such that The projector projects according to the two-dimensional vertex coordinates of the corrected original image, so that the projected image presented in the user's field of view is a rectangle. That is, before the correction, the projected image in the user's field of view appears as a trapezoid, while the corrected projected image appears as a rectangle.

It should be noted that the vertex coordinates mentioned in the above embodiments refer to the coordinates of the four corner points of the projection plane.

Fig. 7 is a block diagram of a projection correction apparatus according to an exemplary embodiment. As shown in FIG. 7, the apparatus 400 includes:

The response module 401 is configured to control the projector to project a preset image to the projection plane in response to the received correction instruction;

A photographing module 402, configured to photograph the preset image projected by the projector through the camera of the projector to obtain a photographed image;

The target feature point determination module 403 is configured to identify the target feature point of the preset image in the captured image;

The three-dimensional coordinate determination module 404 is configured to, for each of the target feature points, determine that the target feature point is located in the target feature point according to the pre-calibrated mapping relationship for the target feature point and the camera coordinates of the target feature point on the captured image. The depth information in the shooting space of the camera is obtained to obtain the three-dimensional coordinates of the target feature point in the projection space of the projector, wherein the mapping relationship is the depth information of the target feature point calibrated at different depths and the camera. The relationship between the offsets of the coordinates;

The normal vector module 405 is configured to determine the normal vector of the projection plane relative to the projector according to the three-dimensional coordinates of each of the target feature points;

The correction module 406 is configured to correct the two-dimensional vertex coordinates of the original image of the projector according to the normal vector of the projection plane and the current pose information of the projector, so as to obtain the two-dimensional coordinates of the corrected original image. vertex coordinates;

The projection module 407 controls the projector to project according to the two-dimensional vertex coordinates of the corrected original image.

Optionally, the preset image is a checkerboard image, and the response module 401 includes:

an instruction response submodule, configured to use a corner detection algorithm to determine the initial feature points in the checkerboard image;

The straight line fitting sub-module is configured to perform straight line fitting on the initial feature points within the same preset area according to the vertical direction and the horizontal direction respectively;

The feature point determination sub-module is configured to take the intersection point between any straight line in the vertical direction and any straight line in the horizontal direction obtained by fitting as the target feature point.

Optionally, the three-dimensional coordinate determination module 404 is specifically configured as:

Optionally, the correction module 406 includes:

An offset information calculation module, configured to calculate offset information of the projector's normal vector according to the normal vector of the projection plane and the current pose information of the projector, wherein the offset information includes yaw angle, pitch angle and roll angle;

a vertex calculation module, configured to calculate, based on the offset information, the two-dimensional vertex coordinates of the projected image projected by the projector to the projection plane;

a homography matrix module configured to establish a homography matrix relationship between the projected image and the original image based on the two-dimensional vertex coordinates of the projected image and the two-dimensional vertex coordinates of the original image of the projector;

A target rectangle selection module, configured to select a target rectangle from the projected image of the projector, and determine the two-dimensional vertex coordinates of the target rectangle;

The vertex correction module is configured to obtain the two-dimensional vertex coordinates of the corrected original image based on the two-dimensional vertex coordinates of the target rectangle and the homography matrix relationship.

Optionally, the target rectangle selection module is specifically configured as:

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, the

When the program is executed by the processor, the steps of the projection correction method provided by the present disclosure are implemented.

Fig. 8 is a block diagram of an electronic device according to an exemplary embodiment. As shown in FIG. 8 , the electronic device 500 may include: a processor 501 and a memory 502 . The electronic device 500 may also include one or more of a multimedia component 503 , an input/output (I/O) interface 504 , and a communication component 505 .

Wherein, the processor 501 is used to control the overall operation of the electronic device 500 to complete all or part of the steps in the above-mentioned projection correction method.

The memory 502 is used to store various types of data to support operations on the electronic device 500, such data may include, for example, instructions for any application or method operating on the electronic device 500, and application-related data, Such as contact data, messages sent and received, pictures, audio, video, and so on. The memory 502 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (Static Random Access Memory, SRAM for short), electrically erasable programmable read-only memory ( Electrically Erasable Programmable Read-Only Memory (EEPROM for short), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (Read-Only Memory, ROM for short), magnetic memory, flash memory, magnetic disk or optical disk.

Multimedia components 503 may include screen and audio components. Wherein the screen can be, for example, a touch screen, and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in memory 502 or transmitted through communication component 505 . The audio assembly also includes at least one speaker for outputting audio signals.

The I/O interface 504 provides an interface between the processor 501 and other interface modules, and the above-mentioned other interface modules may be a keyboard, a mouse, a button, and the like. These buttons can be virtual buttons or physical buttons.

The communication component 505 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC for short), 2G, 3G or 4G, or one or a combination of them, so the corresponding communication component 505 may include: Wi-Fi module, Bluetooth module, NFC module.

In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), digital signal processors (Digital Signal Processor, DSP for short), digital signal processing devices (Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic components Implementation is used to perform the above-mentioned projection correction method.

In another exemplary embodiment, there is also provided a computer-readable storage medium comprising program instructions, the program instructions implement the steps of the above-mentioned projection correction method when executed by a processor. For example, the computer-readable storage medium can be the above-mentioned memory 502 including program instructions, and the above-mentioned program instructions can be executed by the processor 501 of the electronic device 500 to complete the above-mentioned projection correction method.

The preferred embodiments of the present disclosure have been described above in detail with reference to the accompanying drawings. However, the present disclosure is not limited to the specific details of the above-mentioned embodiments. Various simple modifications can be made to the technical solutions of the present disclosure within the scope of the technical concept of the present disclosure. These simple modifications all fall within the protection scope of the present disclosure.

In addition, it should be noted that each specific technical feature described in the above-mentioned specific implementation manner may be combined in any suitable manner under the circumstance that there is no contradiction. In order to avoid unnecessary repetition, various possible combinations are not described in the present disclosure.

In addition, the various embodiments of the present disclosure can also be arbitrarily combined, as long as they do not violate the spirit of the present disclosure, they should also be regarded as the contents disclosed in the present disclosure.

Claims

A projection correction method, characterized in that the method comprises:

In response to the received correction instruction, controlling the projector to project a preset image to the projection plane;

Capture the preset image projected by the projector by using the camera of the projector to obtain a captured image;

In the captured image, identify the target feature point of the preset image;

For each target feature point, determine the depth of the target feature point in the shooting space of the camera according to the mapping relationship pre-calibrated for the target feature point and the camera coordinates of the target feature point on the captured image information to obtain the three-dimensional coordinates of the target feature point in the projection space of the projector, wherein the mapping relationship is between the depth information of the target feature point calibrated at different depths and the offset of the camera coordinates connection relation;

Determine the normal vector of the projection plane relative to the projector according to the three-dimensional coordinates of each of the target feature points;

Correcting the two-dimensional vertex coordinates of the original image of the projector according to the normal vector of the projection plane and the current pose information of the projector, to obtain the two-dimensional vertex coordinates of the corrected original image;

The projector is controlled to project according to the two-dimensional vertex coordinates of the corrected original image.
The method according to claim 1, wherein the preset image is a checkerboard image; and the identifying the target feature points of the preset image comprises:

Use a corner detection algorithm to determine the initial feature points in the checkerboard image;

According to the vertical direction and the horizontal direction, the initial feature points in the same preset area are fitted with straight lines;

An intersection between any straight line in the vertical direction obtained by fitting and any straight line in the horizontal direction is taken as the target feature point.
The method according to claim 1, wherein, for each of the target feature points, according to a mapping relationship pre-calibrated for the target feature point and the camera coordinates of the target feature point on the captured image, Determine the depth information of the target feature point in the shooting space of the camera, including:

For each of the target feature points, according to the pre-calibrated mapping relationship for the target feature point and the camera coordinates of the target feature point on the captured image, calculate the distance of the target feature point in the shooting space of the camera. depth information, wherein the mapping relationship is:

Wherein, h is the depth information of the target feature point, p 1 is the first preset calibration parameter of the target feature point, p 2 is the second preset calibration parameter of the target feature point, and X is the camera coordinate of the target feature point , wherein the first preset calibration parameter and the second preset calibration parameter are both constants.
The method according to claim 3, wherein, for each target feature point, the first preset calibration parameter and the second preset calibration parameter of the target feature point are obtained in the following manner:

When the projector is at a first distance from the projection plane, and the projection light of the projector is perpendicular to the projection plane, a preset image is projected to the projection plane, and the image is captured by the camera of the projector. the preset image to obtain the first image;

determining, based on the first image and the first distance, first camera coordinates and first depth information of the target feature point on the first image;

When the projector is a second distance away from the projection plane, and the projection light of the projector is perpendicular to the projection plane, a preset image is projected to the projection plane, and passes through the camera of the projector photographing the preset image to obtain a second image;

determining, according to the second image and the second distance, second camera coordinates and second depth information of the target feature point on the preset image;

According to the first camera coordinates and the first depth information, as well as the second camera coordinates and the second depth information, combined with the pre-established mapping relationship between the depth information of the target feature point and the camera coordinates, the first camera coordinates are obtained. The preset calibration parameter and the second preset calibration parameter, wherein the mapping relationship is:

Wherein, h is the depth coordinate of the target feature point, X is the camera coordinate, p 1 is the first preset calibration parameter, and p 2 is the second preset calibration parameter.
The method according to claim 1, wherein the two-dimensional vertex coordinates of the original image of the projector are corrected according to the normal vector of the projection plane and the current pose information of the projector, Get the 2D vertex coordinates of the corrected original image, including:

Calculate the offset information of the normal vector of the projector according to the normal vector of the projection plane and the current pose information of the projector, wherein the offset information includes a yaw angle, a pitch angle and a roll angle;

Based on the offset information, calculating the two-dimensional vertex coordinates of the projected image projected by the projector to the projection plane;

establishing a homography matrix relationship between the projected image and the original image based on the two-dimensional vertex coordinates of the projected image and the two-dimensional vertex coordinates of the original image of the projector;

Select a target rectangle from the projected image of the projector, and determine the two-dimensional vertex coordinates of the target rectangle;

Based on the two-dimensional vertex coordinates of the target rectangle, combined with the homography matrix relationship, the two-dimensional vertex coordinates of the corrected original image are obtained.
The method according to claim 5, wherein the selecting a target rectangle from the projected image of the projector comprises:

A point is arbitrarily selected from any side of the projected image, and the point is taken as the vertex of the rectangle to be constructed, and the aspect ratio of the original image is taken as the aspect ratio of the rectangle to be constructed. Generate a rectangle within the area of the image;

The rectangle with the largest area is selected from the generated rectangles as the target rectangle.
A projection correction device, characterized in that the device comprises:

a response module, configured to control the projector to project a preset image to the projection plane in response to the received correction instruction;

a photographing module, configured to photograph the preset image projected by the projector through the camera of the projector to obtain a photographed image;

a target feature point determination module, configured to identify the target feature point of the preset image in the captured image;

The three-dimensional coordinate determination module is configured to, for each of the target feature points, determine that the target feature point is in the The depth information in the shooting space of the camera to obtain the three-dimensional coordinates of the target feature point in the projection space of the projector, wherein the mapping relationship is the depth information and camera coordinates of the target feature point calibrated at different depths The relationship between the offsets;

a normal vector module, configured to determine the normal vector of the projection plane relative to the projector according to the three-dimensional coordinates of each of the target feature points;

a correction module, configured to correct the two-dimensional vertex coordinates of the original image of the projector according to the normal vector of the projection plane and the current pose information of the projector to obtain the two-dimensional vertex of the corrected original image coordinate;

The projection module controls the projector to project according to the two-dimensional vertex coordinates of the corrected original image.
The device according to claim 7, wherein the preset image is a checkerboard image, and the response module comprises:

an instruction response submodule, configured to use a corner detection algorithm to determine the initial feature points in the checkerboard image;

The straight line fitting sub-module is configured to perform straight line fitting on the initial feature points within the same preset area according to the vertical direction and the horizontal direction respectively;

The feature point determination sub-module is configured to take the intersection point between any straight line in the vertical direction and any straight line in the horizontal direction obtained by fitting as the target feature point.
A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processing device, the steps of the method according to any one of claims 1-6 are implemented.
An electronic device, comprising:

a memory on which a computer program is stored;

A processor for executing the computer program in the memory to implement the steps of the method of any one of claims 1-6.