RU2794046C1 - Object positioning method - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: computer technology, in particular method for positioning an object. Computer technology, in particular method for positioning an obje method includes the following steps: capturing an image of an approximate area in which an object positioning support device is located, comprising at least one group of light sources, wherein in each group of light sources, the main light source emits in a colour different from the auxiliary light source, moreover the exact location of the light sources in the device to support the positioning of the object and their colours are known to the positioned object; converting the colour space of the image to HSV colour space; attempting to detect individual light sources and determining the location and/or orientation of the area in which the object positioning support device is placed relative to the positioned object based on the detection result and/or identifying the object on which or near which the object positioning support device is located based on discovery result.

EFFECT: increase of accuracy and detection range of the positioning support device under difficult environmental conditions.

1 cl, 11 dwg

Description

The field of technology to which the invention belongs

The present invention relates to the field of motion control, and more specifically, to a method for positioning an object.

State of the art

Unmanned aerial vehicles (UAVs, UAVs) are gradually becoming a part of everyday life - they perform cargo delivery, various studies, analysis of the surrounding space, object control, photo and video shooting, are used in rescue operations and find many other applications. As is the case with other modes of transport, developments are underway around the world aimed at fully automating the operation of UAVs.

One of the tasks of UAV automation is its autonomous landing. Many different methods are known from the prior art, which can be divided into 2 groups: landing on a prepared surface - for example, on a landing site, and on an unprepared surface - for example, on the ground.

Traditionally, navigation based on the global navigation satellite system (GNSS, GNSS) is used for UAV landing, however, if the task is high-precision landing at a given location and with a given orientation, then navigation directly and only on the basis of GNSS is not enough, since it provides accuracy only within 5-8 meters. One of the solutions to this problem can be the use of differential correction systems, however, in many scenarios of UAV operation, such systems may not be available.

In order to increase accuracy, many well-known solutions use additional navigation methods. Some of these methods use communication based navigation between the landing platform and the UAV, however, they require additional communication equipment both on the landing platform and on board the UAV, which complicates the system and its wide application possibilities.

There are also known methods of additional navigation based on the use of a camera installed on the UAV. The camera in a typical case is a regular on-board tool, therefore, such methods do not require the installation of additional equipment on the UAV. There are many ways in which a black-and-white marker of a square or other shape is placed at the landing site, the camera captures an image, marker recognition is performed, and thereby the exact landing site and / or the position of the camera relative to the marker is determined, which allows for a fairly accurate landing. However, such methods have a number of disadvantages: a high resolution of the camera is required, the marker must be completely captured by the camera, it is not visible from afar and it is not visible in difficult environmental conditions (night, fog, precipitation). To solve this problem, some solutions use light sources.

For example, CN 111506091 A discloses a landing control system in which a high-resolution LED screen (light-emitting diode screen) is placed on the surface of the landing platform, which shows one pattern over the entire area when the UAV is at a high altitude above the platform, and then shows 5 small 2D patterns as the UAV approached the platform. AprilTag are used as patterns. However, such a solution is very expensive and complex structurally and algorithmically, requires agreement between the platform and the UAV regarding which patterns to show at the moment in accordance with the current height of the UAV, and also requires a high screen resolution, which makes it impossible to use high-power light sources, which significantly impairs the possibility of using at long range and in difficult environmental conditions. For correct recognition of such a pattern, it must be visible in its entirety. In order to compromise between the size and brightness of individual LEDs and the overall screen resolution, the screen can be increased, but then it will take up a lot of space and clutter the platform, which makes it difficult to implement it on a small platform with non-planar geometry and difficult to place other equipment on it . All these shortcomings significantly limit the possibility of wide application of such a solution.

Also known is the solution disclosed in document KR 10-1524936 B1, which describes an UAV storage and charging device containing a plurality of landing platforms. At the corners of each platform there are n*n matrices of LED and IR lamps. LED lamps work during the day, and IR lamps work at night. The matrices together form a unique pattern. When the UAV flies close enough to the platform, it detects and recognizes this pattern, determines its location, communicates it to the platform, obtains the optimal landing path and landing clearance from it, and lands. The need to use LED and IR lamps in each cell of the matrix increases the complexity and reduces the reliability of the design, and also reduces the maximum possible brightness of the lamp. In addition, such a solution, like the previous one, requires landing coordination between the platform and the UAV. Specific types of patterns and methods for their recognition are not disclosed in this document. For correct recognition of known patterns, they must be visible in their entirety, that is, the camera must capture all corners of the platform. If only one or two matrices fall into the camera's field of view, or if not all lamps in the matrix are recognized, then uncertainty may arise. If you try to display known patterns in each matrix, similar to the one used in the previous solution, then this will lead to the need to increase the resolution of the matrices, which either reduces the maximum possible brightness of the lamps, or makes each matrix unnecessarily cumbersome. This impairs the ability to use at long range in difficult environmental conditions.

Thus, there is a need in the art for a method that would allow the positioning of an object in a wide range of environmental conditions and would work at a great distance from the area of the pattern.

The essence of the invention

In order to overcome at least some of the aforementioned shortcomings of the prior art, the present invention is directed to a method for positioning an object.

According to the present invention, there is provided a method for positioning an object, wherein the object to be positioned is a mobile or wearable device containing a camera, a processor, and a memory, the method comprising the steps of:

capturing, by means of the camera of the object being positioned, an image of the approximate area in which the object positioning support device is located, comprising at least one group of light sources, each group of light sources comprising at least one main light source and at least two auxiliary light sources , and in each group of light sources, the main light source emits in a color different from the auxiliary light source, and the exact location of the light sources in the device for supporting the positioning of the object and their colors are known to the positioned object;

converting, by the positional object processor, the color space of the image to the HSV color space;

performing, by means of the positioned object processor, an attempt to detect individual light sources located in the area in which the device for supporting the positioning of the object is located, and/or at least one resulting light spot formed by a group or multiple groups of light sources located in the area in which hosts a device to support object positioning using color segmentation in the HSV color space; And

determine, by means of the positioned object processor, the location and/or orientation of the area in which the object positioning support device is placed relative to the positioned object based on the detection result and/or identify the object on or near which the object positioning support device is located based on the result detection.

Technical result

The present invention improves the efficiency of devices, systems and methods for positioning an object. This provides:

- simplifying the design of the device to support the positioning of the object in comparison with devices containing high-resolution screens or complex arrays of multiple light sources;

- simplification of the design of the positioned object due to the absence of the need to install additional equipment on it in addition to the camera and the processing unit;

- reduced requirements for the characteristics of the camera of the positioned object, such as resolution and viewing angle, compared with solutions using devices to support object positioning with high-resolution screens or complex matrices of multiple light sources;

- simplified integration with device placements to support object positioning that have non-planar geometry;

- improving accuracy under difficult environmental conditions;

- reduction of noise in the air due to the absence of the need for communication between the UAV and the device to support the positioning of the object;

- the possibility of reducing the size of the placement of devices to support the positioning of the object;

- increased accuracy at large angles to the device to support the positioning of the object;

- increased accuracy when hitting only part of the device to support the positioning of the object in the field of view of the camera;

- sufficient accuracy with successful recognition of not all light sources in the group (with an increase in the number of recognized light sources, the accuracy increases);

- increasing the recognition range of the device to support the positioning of the object.

It should be understood that not each of the embodiments can simultaneously provide all of these advantages over all known prior art solutions. Accordingly, some embodiments may have only some of these advantages or other advantages over some known solutions.

These and other advantages of the present invention will become apparent upon reading the following detailed description with reference to the accompanying drawings.

Brief description of the drawings

On FIG. 1 shows the UAV autonomous landing system according to the present invention.

Fig. 2 shows an equivalent circuit of a typical example of the arrangement of light sources on a landing platform.

Fig. 3 illustrates a flow diagram of an autonomous landing method.

On FIG. 4 illustrates an example of the result of color segmentation of an image.

On FIG. 5 illustrates an example of grouping auxiliary light sources.

On FIG. 6 illustrates an example of correlating a main light source with a group of auxiliary light sources.

On FIG. Figures 7-9 show examples of landing platform detection in poor lighting conditions at different distances between the UAV and the landing platform.

On FIG. 10 shows a device for supporting object positioning.

On FIG. 11 shows an example of a practical implementation of a device for supporting object positioning.

It should be understood that the figures may be represented schematically and not to scale and are intended primarily to improve understanding of the present invention.

Detailed description

General overview of the landing system

Next, with reference to FIG. 1, an example of a UAV autonomous landing system according to the present invention will be described. It should be noted that this example is not limiting and is only intended to provide a general understanding of the proposed UAV landing and positioning principles. Other examples other than those shown in FIG. 1, but also within the scope of the invention.

The unmanned aerial vehicle (UAV) 110 lands on the landing platform 120 using the object positioning support device 130. The object positioning support device 130 includes 5 groups of light-emitting diodes (LED), each group of LEDs contains 4 LEDs arranged relative to each other in a square shape. 4 groups (131, 132, 133, 134) are located at the corners of the platform and form a square, and the 5th group (135) is located in the center of the square. Dotted lines in Fig. 1 from designation 130 to groups 131, 132, 133, 134, 135 indicate their exemplary lines of communication with a common control unit (not shown) of device 130 to support object positioning.

One of the LEDs in the group is the main one, the rest of the LEDs in the group are auxiliary. Each group of LEDs allows you to uniquely identify the direction to the center of the platform. To do this, the color of the main LED in each group is different from the color of the additional LEDs in that group. The color of the main LED in the central group 135 is different from the color of all other LEDs in all groups. For example, as shown in FIG. 1, the main LED in the central 5th group 135 is red, the main LEDs in the remaining 4 corner groups 131, 132, 133, 134 are green, and the auxiliary LEDs in all 5 groups 131, 132, 133, 134, 135 are Blue colour. The location of the LEDs relative to the landing platform 120 and their colors (or in other words, the pattern) are known in advance to the UAV 110.

UAV 110 uses navigational aids (eg, GPS) to approach the landing area at some altitude, where landing platform 120 should be approximately located. and an onboard autonomous landing control unit. When the camera is operating, all the LEDs of the device 130 to support object positioning are on. During processing of the images captured by the camera, the autonomous landing control unit of the UAV 110 detects at least a portion of at least one group of LEDs of the object positioning support device 130, and using the stored pattern information, determines the location and orientation of the landing platform 120 relative to the UAV 110. Than the more groups of LEDs detected by the UAV 110, the higher the positioning accuracy. The UAV 110 then approaches the landing platform 120 based on the received data, detects as many groups of LEDs of the device 130 as possible to support object positioning, ascertains the location and orientation of the landing platform 120 relative to the UAV 110, and eventually makes an autonomous landing.

In the example shown in FIG. 1, LEDs are used, but in general any other suitable light/optical (visible) source may be used.

Accordingly, the UAV camera 110 must be capable of detecting light/optical radiation. The UAV camera 110 may be a photo and/or video camera. For the purposes of the present invention, the use of a monocular camera is sufficient, but a stereo camera may also be used if desired.

In the example shown in FIG. 1, 4 groups of LEDs are used in the corners of a square and 1 group of LEDs in the center of the square, however other patterns can be used in general. light sources that are in the same group, hereinafter referred to as adjacent light sources. The group of lights located closest to the center of the landing platform and/or to the center of the pattern will be referred to hereinafter as the central group of lights, and the remaining groups will be called peripheral groups of lights (or corner if they are located at the corners of the conditional polygon) . It should be noted, however, that the presence of a central group of light sources is not necessary to implement the basic principles of the present invention. In addition, the number of light sources in a group may differ both within the same object positioning support device 130 and between different object positioning support devices.

An equivalent circuit of a typical 4x4 example (i.e., 4 groups of 4 LEDs arranged in a square) is shown in FIG. 2. Auxiliary LEDs 201, 202 and 203 (shown in blue) are located on the outer edges (corners) of the group relative to the center of the landing platform, and the main LED 204 (shown in green) is located on the inner edge (corner) of the group, that is, relative to its group it is directed towards the center of the landing platform. The group of LEDs 201, 202, 203 and 204 together form a square with side d _in . More specifically, under d _in in FIG. 2 means the distance between the centers of the LEDs that form the side of the square. The center of the landing platform is conventionally marked as a square 205 (shown in red). For simplicity, the remaining 3 groups are not shown in FIG. 2. For each group of LEDs, the relative vector R _c is known, which starts at the center of the group and ends at the center of the landing platform. Dotted lines in Fig. 2 shows the positions of another, central group of LEDs that could be used to indicate or clarify the orientation of the landing platform. In general, any configuration of lights within the groups and any configuration of the groups relative to the center of the landing platform can be used. Some specific examples with other configurations will be described later in this document.

In the example shown in FIG. 1, the device 130 for maintaining object positioning is part of the landing platform 120, however, in other embodiments, the device 130 may be a separate device that can be installed on the surface of the landing platform 120 or on any other surface on which the UAV 110 is to land. in this document, for convenience of description, it is the embodiment with landing on the landing platform that may be mentioned, but it should be borne in mind that other embodiments are possible.

UAV 110 for the purposes of the present invention can be considered, for example, a drone, a quadrocopter, a multicopter, an unmanned helicopter, a vertical takeoff and landing aircraft (VTOL), or any other aircraft on which a camera and an autonomous control unit are installed or can be installed. landing so that it is capable of autonomous landing.

A landing platform may also be referred to interchangeably as a landing pad, or simply a platform or platform.

Autonomous landing method

The processing performed in the UAV will be described next with reference to FIG. 3 which illustrates a flowchart of the autonomous landing method.

The landing area coordinates are known in advance by the UAV, and in step S310, the UAV may use conventional positioning aids (eg, global navigation satellite system (GNSS) or inertial navigation system) to advance to the landing area coordinates. The accuracy of typical positioning aids may not be sufficient to make a high-precision landing at a given point with a given orientation, so in general, the location where the UAV actually moved in step S310 is only a point above the approximate landing area. To further refine the location and orientation of the UAV uses the detection of the device proposed in the present invention to support the positioning of the object.

In step S320, the camera mounted on the UAV captures an image falling within its field of view. In order for at least one group of light sources of the object positioning support device to fall into the field of view, the camera must capture the approximate landing area where it is located. The larger the camera's field of view, the more likely it is that light sources will enter the field of view. If the UAV camera is fixed (stationary), then to increase the probability of light sources entering the field of view, the UAV can start shooting from a relatively high altitude. If the UAV camera is movable (rotary), then it can be directed to the approximate landing area.

The camera's field of view (i.e., the image) may be exposed to light or glow sources other than the light sources of the object positioning support device. In order to reduce the number of such extraneous light sources and the effect of ambient light, the light sources of the object positioning support device can be used with high intensity sources, and the camera can shoot at a fast shutter speed, such as 1/500, 1/1000. This embodiment is less dependent on time of day and weather conditions. In addition, in one of the embodiments of the invention, the shutter speed of the camera may vary depending on the time of day and weather conditions. Such a dependence can be predetermined or adaptive, for example, based on the number of light sources recognized at further stages in the frame and their quality.

In step S330, the autonomous landing control unit, having received the image from the camera, converts the image color model to HSV (Hue-Saturation-Value, Hue-Saturation-Value, or HSB (Hue-Saturation-Brightness, Hue-Saturation-Brightness)), if the original color model (color space, palette) used by the camera is different from HSV. In particular, there are many palettes in which the camera can shoot natively: RGB, BGR, YUYV, YUY, CMYK, etc.

The HSV palette is used to perform image segmentation by color, since it is more difficult to perform such an operation in other palettes with high accuracy.

In step S340, the autonomous landing control unit performs auxiliary light source detection. To do this, knowing the color of the auxiliary light sources, the autonomous landing control unit searches for the contours of the auxiliary light sources in the HSV color space using color segmentation. In particular, a search is made for areas of the image whose color falls within a predetermined spectrum interval in the HSV palette, approximately corresponding to the previously known emission spectrum of auxiliary light sources. This approach may be sufficient for many applications.

In another embodiment, the found light sources can be further filtered by the intensity of the radiation - for example, it can be determined whether the intensity of the radiation of a given light source falls within a predetermined intensity interval. This approach can be useful when the surrounding space is relatively noisy in terms of multiple extraneous light sources entering the frame, similar in color to the desired light sources. The predetermined intensity range can only be limited to the lower intensity value - in which case only the least bright sources, which are most likely not the required light sources, will be discarded. The preset intensity interval can also be limited by the upper intensity value - in this case, too bright sources will also be rejected, which with a high degree of probability are not the required light sources.

In yet another embodiment, the desired intensity interval may not be predetermined, but may be determined by the autonomous landing control unit. For example, the intensity distribution density of all found light sources can be determined. Further, the intensity interval in which the distribution density is the highest can be chosen as the required one. In another embodiment, the expected number of light sources may be taken into account, and the intensity interval in which the distribution density is closest to the expected one may be selected as required. For example, the landing platform contains 12 auxiliary light sources, 30 contours of light sources are found in the image with a similar color, but with different intensities; the intensity range is divided into 5 equal intervals, and 12 contours fall into the 4th interval, 13 contours fall into the 1st interval, and the remaining 5 contours are distributed over the remaining three intervals. Since 12 brightest light sources are expected to be found, the 4th interval is chosen as the required one, and the rest are discarded. Thus, the accuracy can be improved.

For each found contour (or in other words, for each found auxiliary light source), the autonomous landing control unit finds the center (for example, by the method of k-means - by the nearest distance).

In step S350, the autonomous landing control unit performs main light source detection. To do this, knowing the color of the main light sources, the autonomous landing control unit searches for the contours of the main light sources in the HSV color space. In particular, a search is made for areas of the image whose color falls within a predetermined spectrum interval in the HSV palette, approximately corresponding to the predetermined emission spectrum of the main light sources.

For each found contour (or in other words, for each found main light source), the autonomous landing control unit finds the center (for example, by the k-means method - by the nearest distance). In this case, if only peripheral groups of light sources are used in the device to support the positioning of an object, then the search for contours and centers is performed only for the main light sources of peripheral groups. If both peripheral and central groups of light sources are used, then the contours and center are also searched for the main source of the central group.

In one of the embodiments, if no light source is found at step S340 or at step S350, then such a frame will be skipped, not taken into account. The method proceeds to step S320 to acquire a new landing area image.

On FIG. 4 illustrates an example of visualizing the result of performing steps S340-S350. In particular, it is shown that a plurality of blue contours/dots, presumably corresponding to the auxiliary light sources (step S340), a plurality of red contours/dots, presumably corresponding to the main peripheral lights (step S350), and one green contour/dot, presumably corresponding to the main to the central light source (step S350). At the same time, translucent red dots are actually noise.

Once the auxiliary light sources and/or main light sources have been found separately, it is necessary to determine how they relate to each other.

To do this, first, at step S360, the autonomous landing control unit performs clustering of the auxiliary light sources (if any were found at step S340), that is, determines whether each auxiliary light source belongs to one or another group. In particular, the k-means method is used. In this case, as is known, for the k-means method, it is required to pre-specify the required number of clusters into which the found set of points should be partitioned. The present invention proposes to use the following approach:

where N is an integer number of groups (clusters) rounded up,

M _al is the total number of auxiliary lights found in the frame,

a is the number of auxiliary lights in each group.

For example, if 10 auxiliary light sources are found in step S340, and 3 auxiliary light sources are used in each group, then the number of groups will be N=10/3≈4 groups. In another example, if 5 auxiliary light sources were used in each group, then the number of groups would be N=10/5=2 groups.

It should be noted that the principle of determining the number N of groups according to formula (1) is actually a first approximation, a hypothesis that often turns out to be true due to the features of the proposed method. As a hypothesis test, the distance between adjacent lights in the group can be determined and compared with a predetermined interval from the minimum distance to the maximum distance between the lights in the group. For example, if initially according to formula (1) it is determined that the detected 4 light sources should be divided into N = 2 groups, and when checking it turned out that in the 2nd group the distance between adjacent light sources does not fall within a predetermined interval from 5 to 15 cm, then the method can proceed to the hypothesis of N=3 groups (obtained by an increment to the previously calculated value of N=2). In another embodiment, those light sources that do not fall within a predetermined interval may simply be ignored (discarded), and the initially determined number of groups may be retained.

Accordingly, the autonomous landing control unit performs k-means clustering of the auxiliary light sources into a certain number of groups.

On FIG. 5 illustrates an example of grouping auxiliary light sources. In particular, it is shown that using the k-means method, it was revealed that the auxiliary light sources found earlier (see Fig. 4) belong to 4 different groups.

Then, in step S370, the autonomous landing control unit determines the distances between the groups of auxiliary light sources and the found main lights (if any were found in step S350). In particular, the coordinates of the center of each found group are calculated and the matrix D of distances between the centers of all found groups and the centers of all found main light sources is calculated.

where C _i are the coordinates of the center of the i -th group (cluster),

P _j are the coordinates of the center of the j -th main light source,

N is the number of groups

K is the number of found main light sources.

Next, in step S380, the autonomous landing control unit determines correspondences between groups of auxiliary light sources and main light sources. Specifically, in the distance matrix D calculated in step S370, in each i th row, the minimum element min(D _i,j ) is determined, or in other words, in each i th group, the minimum distance d _min =min(D _{i ,j} ) to the main light source. The index j of this element D _i,j indicates that the j -th main light source from the K found main light sources corresponds to the i -th group of N groups (is in it, belongs to it).

On FIG. 6 illustrates an example of correlating a main light source with a group of auxiliary light sources. In particular, it is shown that by performing steps S370 to S380 according to the principle of minimum distance d _min , one of the main peripheral light sources (red) found in step S350 is compared with one of the auxiliary light source groups found in step S360. At the same time, the probability of an extraneous light source in the frame, falsely recognized as the main one, located even closer to the auxiliary light sources of one group than the true main light source of this group, is extremely small, since this can only happen if an extraneous light source with intensity luminosity and with a color (spectrum) that is similar to the desired main light source, is located either on the landing platform itself inside the perimeter of the group (which is practically excluded due to the fact that the configuration of the landing platform and the device to support the positioning of the object is controlled by the person using this invention), either in the space between the landing platform/object positioning support device (i.e. is a fixed overhead light source or is on an aircraft - both are unlikely). Thus, the approach with the search for the minimum distance d _min is relatively simple and at the same time highly accurate.

Thereafter, in step S390, the autonomous landing control unit determines the location and orientation of the landing area (eg, landing platform) relative to the UAV.

In particular, for each i -th group, the Perspective-n-Point (PnP) problem is solved, that is, the problem of determining the orientation and location of the camera in space from the image obtained with its help, by obtaining a perspective projection of points in the image onto the camera sensor plane. As is known, the condition for solving such a problem is n correspondences between the points of the image and the points of the object model, as well as the known relative position of the points of the object model. In this case, a well-known object model for the UAV is a device for supporting the positioning of the object and the arrangement of groups and light sources on it.

Namely, the location P _i of the center of the landing area and the orientation R _i of the center of the landing area are calculated (as mentioned above, R _i is a relative vector indicating the direction from the i th group to the center of the landing area). Specific methods for solving the PnP problem are known to those skilled in the art and are not disclosed here in more detail. As a non-limiting example, the algorithm disclosed in Y. Wu and Z. Hu, “Pnp problem revisited,” Journal of Mathematical Imaging and Vision, vol. 24, no. 1, pp. 131–141, 2006.

When the values of location P _i and orientation R _i for each i -th group are obtained, by calculating the median of these values, the refined location P and orientation R of the center of the landing area are determined:

where N is the number of groups.

Then, the autonomous landing control unit controls the UAV actuators by itself, or sends the location and orientation data of the landing area determined in step S390 to the UAV flight controller so that the UAV approaches the landing area based on the received data. Thus, the autonomous landing method further comprises step S395 (not shown in FIG. 3) in which the UAV performs movement towards the landing area.

As it approaches, when the UAV has made some next movement towards the landing area, the UAV may again repeat steps S320-S395, surveying the landing area, detecting as many light sources as possible, refining the location and orientation of the landing area relative to the UAV, and moving closer to the area. landing. Ultimately, after completing the required number of iterations, the UAV makes an autonomous landing with increased accuracy at a given point and with a given orientation, even in conditions of poor visibility.

The proposed method has been tested in practice for landing a UAV in autonomous mode on a landing platform. The landing platform used as a sample has a square shape 40x40 cm in size, the UAV beam span is comparable in size, the groups of LEDs are located at the corners of the platform at a distance of 30 cm from each other. WS2812B addressable diodes were used as light sources. The LEDs in the group are arranged so that they form a square, while the distance between them (the side of the square) is 5 cm. The UAV successfully lands at a given point with a given orientation. The positioning accuracy is approximately 1-7 cm depending on the distance between the UAV and the landing platform - the closer the UAV is to the landing platform and the more groups of light sources detected, the higher the accuracy.

Examples of detection of the landing platform in poor lighting conditions at different distances between the UAV and the landing platform are shown in Fig. 7-9. In particular, in FIG. 7 shows the case at a distance of 2 m, FIG. 8 - 5 m, in Fig. 9 - 10 m.

As seen in FIG. 7-9, the autonomous landing control unit successfully copes with the task of detecting at least some groups of LEDs, despite the night time. Meanwhile, in FIG. 7 shows that the landing platform is located at a sufficiently large angle relative to the UAV camera, but this did not prevent the detection of LED groups. Due to the features of the proposed method, even when not all groups could be found, the center of the landing platform is determined successfully.

A stationary typical camera was used on the test UAV, and even taking into account the relatively small dimensions of the UAV and the unremarkable characteristics of the camera itself, the proposed method made it possible to successfully detect the landing platform for autonomous landing already at a height of 20 m in poor visibility conditions. The use of a higher resolution camera would increase the range/altitude at which the proposed method could be effectively performed.

In fact, in the proposed method, in order to determine the location of the landing area, it is sufficient to detect at least one group of light sources. If more than one group of light sources is detected in the field of view of the UAV camera, the positioning accuracy increases. Also, the positioning accuracy increases with an increase in the number of groups of light sources on the landing platform and with an improvement in camera characteristics (resolution, aperture ratio, matrix type, optics quality, computing capabilities, data processing algorithms used, etc.).

In addition, taking into account the dimensions of the test sample mentioned above, even the size of a group of light sources of 5x5 cm turned out to be sufficient for autonomous landing of the UAV on the landing platform in poor visibility conditions, that is, the proposed method can be used even if there is little free space.

Moreover, it is possible to use a landing platform having a complex (non-planar) geometry, because it is easier to mount small separate independent groups of light sources on it than mounting a single relatively large flat pattern / LED array.

It was mentioned above that for the purposes of the present invention, if necessary, a camera with two lenses instead of one can be used. The use of a stereo camera will improve the accuracy of the proposed method due to the availability of additional data that can, for example, be averaged, compared, processed independently, selected the best, etc.

Configuration options

It has also been mentioned above that, in general, any configuration of light sources within groups and any configuration of groups relative to the center of the landing platform can be used.

In one embodiment, the configuration may be a 3x3 triangle pattern. The light sources in each group are located relative to each other to form a triangle, one of the vertices of which is directed towards the center of the landing platform and contains the main light source.

In another embodiment, the configuration may include one or more groups of light sources arranged relative to each other to form a circle or arc.

In other embodiments, the implementation of the configuration may include groups of light sources that form other geometric objects (shapes, polylines, curves, line segments).

The basic principle is that in each group there are auxiliary lights emitting in one color and a main light source emitting in another color. The relative positions and colors of the light sources known in advance to the UAV make it possible to determine the location and orientation of the landing area.

The above method for autonomous landing of the UAV provides for determining the orientation of the landing platform. It should be noted that in some applications it is sufficient to use configurations that allow only the direction of the angles of the landing platform to be determined without accurately determining the orientation of the landing platform as a whole. For example, this applies to the case where a multi-beam UAV is used, and it does not matter which of the beams will be in which corner of the landing platform when landing.

If it is necessary to land with a more precise orientation, then configurations of light sources can be used to accurately determine the orientation of the landing platform relative to the UAV. For this, for example, a central group of light sources may be used, or the color of the main light source or auxiliary light sources in one peripheral group may differ, respectively, from the color of the main light sources or auxiliary light sources in other peripheral groups, or the color of one or more auxiliary light sources may differ from the color of other auxiliary lights within the same group, or the relative position of the groups relative to each other may be asymmetrical about the center of the landing area, or the location or colors of the main lights or auxiliary lights may be asymmetrical about the center of the area landings, etc.

Range increase

It was mentioned above that in one of the embodiments, if no light source is found at step S340 or at step S350, then such a frame will be skipped, not taken into account, and the method proceeds to step S320 to obtain a new image of the landing area. In another embodiment, in such a case, the method does not immediately proceed to step S320, but first, the autonomous landing control unit attempts to detect entire groups of light sources (step S355). The peculiarity of this approach is that at a great distance from the UAV (and sometimes even at a relatively short distance in conditions of poor visibility), the light sources in one group can merge into a single light spot, which can also be considered as a light source. It is important to consider that the color of the main light in the group is different from the color of the auxiliary lights, so the resulting color of the group of lights is different from the color of the main light and the color of the auxiliary lights, and this resulting color is predicted in advance, since the colors of the lights in UAVs are known to the group in advance. In addition, this feature can be advantageously used by selecting the resulting colors of the group of LEDs in such a way that they are unique for the surrounding conditions of the light background or terrain. The range of possible variations and color combinations is extremely wide, and a person skilled in the art will be able to easily implement this invention by applying the principles set forth, so specific options will not be described here.

By detecting the group of light sources in their entirety by its preliminarily known resulting color, the UAV thus obtains information about the approximate position of the landing area and the approximate distance from the UAV to the landing area. With this information, the UAV performs some next movement towards the landing area, repeats steps S320-S395, taking pictures of the landing area and detecting light sources, until landing.

Thus, an increase in the efficiency of the autonomous landing method is provided by increasing the recognition range of light sources even in conditions of poor visibility.

Moreover, taking into account the fact that the landing area can contain many groups of light sources, this set at a relatively large distance can form a new pattern - a supergroup of resulting colors. The mutual arrangement of the groups is known in advance by the UAV, therefore the above principles related to the detection of individual LEDs in a group and determining their role and location in the group are also applicable to supergroups. Similarly, the resulting color of one group in a supergroup may differ from the resulting color of other groups in the supergroup, making it possible to separate groups into primary and secondary groups by preselecting their resulting colors. Thus, even at a long range, it is possible to determine the orientation of the landing area. In addition, this makes it possible to detect groups of light sources with different resulting colors, which increases the reliability of detecting exactly the landing area, and not random interference.

In the further development of this approach, even a supergroup can merge at a long distance into a single light spot, which can be considered as another, global light source, which generally characterizes the landing area, as if it had only one light source of this color. Similarly, the resulting color of such a global light source can be pre-matched. Accordingly, an additional increase in the detection range of the landing area is provided.

The order of detection of individual lights, groups, supergroups and the global light source may differ from that described above - for example, at a far distance from the landing area, the probability of detecting individual lights may be lower than larger light spots, so at the very beginning the autonomous landing control unit can attempt to detect a global light source and zoom as it gets closer, trying to find smaller and smaller structures. For example, if it is already possible to confidently find the supergroup, then the need to search for a global light source is eliminated, and the autonomous landing control unit can attempt to detect individual light sources, already knowing the approximate position of the groups in the supergroup.

In addition, at long range, when the GNSS readings are still relatively accurate, verification of the position of the found groups or global light source can be performed, and if they are located far from the approximate landing area indicated by GNSS, such results may be considered invalid. Thereby, the accuracy of detecting a long-range landing area and preventing erroneous wandering around it can be improved.

Autonomous landing device

Next, the autonomous landing device according to the present invention will be described.

It should be understood that this device generally corresponds to the autonomous landing control unit described above and performs its functions in the method 200, and if any information is not disclosed in relation to the device, but is disclosed in relation to the method, and vice versa, then this does not imply that in device or method, this function or step cannot be performed, but is done only in order not to clutter up the description with repetition of details.

The autonomous landing device comprises a processor and a memory coupled to the processor and containing instructions that cause the processor to perform the following operations:

send a command to the camera to capture an image when the UAV has moved to the landing area, as described in relation to steps S310-S320;

receive an image of the landing area from the camera;

convert, as described in step S330, the color space of the image to the HSV color space;

detect in the image, as described in relation to step S340, at least one auxiliary light source located in the landing area, using color segmentation in the HSV color space;

to detect in the image, as described in relation to step S350, at least one main light source located in the landing area, using color segmentation in the HSV color space;

form on the image, as described in relation to the step steps S360-S380, groups of detected light sources;

determine, as described in relation to step S390, the location and orientation of the landing area by solving the PnP task; And

control the actuators of the UAV so that the UAV approaches the landing area based on the determined location and orientation of the landing area.

The device for autonomous landing can be located on the UAV itself - in this case, the complete autonomy of the UAV is ensured due to the absence of the need to communicate with the landing platform.

In another embodiment, the autonomous landing device may be external to the UAV. In such a case, the device for autonomous landing further comprises a wireless communication unit for communicating with the UAV to receive a message from the UAV about the movement to the landing area, send an image capture command to the camera, receive images from the camera installed on the UAV, and send to UAV commands for exerting a control effect on the actuators of the UAV. In this case, the UAV is released from the computational load associated with determining the location and orientation of the landing area and the formation of control actions on the actuators. In some cases, this can reduce the power consumption of the UAV and reduce the performance requirements for UAV computing modules. The characteristics of the UAV camera and its exact location relative to the UAV are either known in advance to the device for autonomous landing, or transmitted to it from the UAV in a corresponding message.

In yet another embodiment, the autonomous landing device may itself comprise a camera. The camera, processor and memory of the autonomous landing device can be located in a single housing. This is especially useful if the UAV does not include a camera but is required to perform an autonomous landing according to the present invention. In this embodiment, the device can be completely mounted on the UAV and connected by wire or wirelessly, for example, to its flight controller in order to receive from it a message about the movement to the landing area and send a control action to the UAV actuators.

Device to support object positioning

Next, with reference to FIG. 10, a device for supporting object positioning will be described in more detail.

The object positioning support device comprises at least one light source group unit 1000.

Each light source group block 1000 includes a main light source block 1010, auxiliary light source blocks 1020, 1030, 1040, connectors 1050 and 1060, and a control block 1070.

Block 1010 of the main light source contains at least one main light source. In a particular example, this may be a single LED or several closely spaced LEDs forming a single light source (eg, an LED array or LED strip).

Each of the blocks 1020, 1030, 1040 auxiliary light sources contains at least one auxiliary light source. In a particular example, this may be a single LED or several closely spaced LEDs forming a single light source (eg, an LED array or LED strip). In general, the number of auxiliary light source blocks is two or more.

The emission color of the main light source block 1010 is different from the emission color of the auxiliary light source blocks 1020, 1030, 1040.

The emission color of each of the auxiliary light source units 1020, 1030, 1040 may be different if necessary (for example, to increase the likelihood of determining the direction to the center of the landing platform even if the contours of a small number of light sources are found), but is preferably the same, since this simplifies the design and can increase the speed of recognition.

The exact location of the lights in the landing area and their colors are known in advance to the autonomous landing device so that the PnP task for the camera located on the UAV can be successfully solved.

In various embodiments, the light source blocks 1010, 1020, 1030, 1040 in each light source group block 1000 are arranged relative to each other to form a triangle, square, circle, arc, other geometric objects (shapes, polylines, curves, line segments), or a combination thereof. .

Also, the device for supporting the positioning of the object as a whole may contain several blocks 1000 of a group of light sources - for example, 3 blocks 1000 arranged in a triangle, 4 blocks 1000 arranged in a square, 5 blocks 1000, of which 4 are located in a square, and 1 is in the center, etc.

The light source group unit 1000 may optionally have its own power supply in order to provide increased autonomy and mobility of the device to support object positioning. In another embodiment, the light source group unit 1000 is powered from an external power source. In this case, the power source external to the block 1000 of the group of light sources may still be part of the device for supporting object positioning, that is, it may contain one power source for all individual blocks 1000 included in the device. In another embodiment, the power may be external and in relation to the device to support the positioning of the object. This may slightly increase power consumption, but allows the use of more powerful light sources, resulting in improved recognition, positioning accuracy, and landing accuracy.

The control unit 1070 is optional. If each light source group unit 1000 has its own control unit, an external control unit may not be required, and the versatility and compactness of the apparatus for supporting object positioning is increased. In another embodiment, the object positioning support device may comprise one control unit for all separate units 1000 included in the device.

Connectors 1050 and 1060 are optional, they may be required to receive power from an external power supply, to receive external control, etc. The exact number and configuration of connectors depends on the requirements of the particular application.

The light source group block 1000 also contains various circuit components (not shown in FIG. 10) such as resistors, capacitors, etc., to provide the necessary operation of the light sources and other components of the block 1000.

All components of one light source block 1000 can be placed on one printed circuit board, as shown in the example in FIG. 11. By connecting different units 1000 of a group of light sources with each other using connectors, you can connect them in a series circuit, which reduces the number of wires and simplifies installation compared to connecting a star from a power source and a single control unit. At the same time, each individual block 1000 of the group of light sources still remains independent and compact, which allows the proposed device to be used to support object positioning on landing platforms with different surfaces, including non-flat ones.

In another embodiment, all blocks 1000 group of light sources can be placed on a single printed circuit board. This design is rigid and does not require measurements and calibration upon installation or high accuracy during installation, since all dimensions and relative positions of light sources are unchanged.

Object positioning method

The above described embodiments in which the UAV uses the proposed device to support the positioning of the object for the purpose of autonomous landing. However, there are much wider applications for such a device.

Thus, the object positioning support device can be placed not only on the landing platform, but in any other location where it is required to ensure the positioning of objects - for example, on a lamppost, on the outer or inner wall of a building, at a bus stop, on a road sign, in a parking lot, on another industrial or domestic facility, on a vehicle (including an unmanned vehicle), on a robot, on pieces of furniture, clothing, products, etc.

A positioned object can be not only a UAV, but also any other vehicle, robot, any wearable device with a camera - a mobile phone, tablet, glasses, helmet, etc., i.e. any movable (mobile or wearable) object (device ) containing a camera, a processing unit (such as a processor), and memory.

Accordingly, using the principles of the present invention, a positionable object with a camera can capture an image containing a device for supporting the positioning of an object, and knowing in advance the exact parameters of the internal structure of this object, can perform detection of light sources and/or the resulting light spot formed by a group or multiple groups light sources, and based on the detection result, determine its location and/or orientation relative to the object positioning support device. In addition, knowing in advance the exact location and orientation parameters of the device to support the positioning of the object in space, the positioned object can determine its own location and orientation.

Thus, the positioned object can move (fly, pass, pass, be carried) in the area in which the object positioning support device is located, and determine or refine its location and orientation, or perform other related actions even in poor visibility conditions (night, twilight , snow, rain, fog) and/or at a relatively large distance from the device to support object positioning, when standard patterns such as a QR code or a fractal marker are useless.

In another possible embodiment, the device for supporting the positioning of an object can serve as an identifier (marker, pattern) that allows you to identify the object on which or near which it is located.

For example, a car can park in a desired location, drive up to a refueling / charging point, recognize road signs (for example, a pedestrian crossing) without using neural networks and without having a pre-mapped terrain map. VR devices can refine object overlay settings to create augmented and mixed reality. The UAV, while on its way to its end point, can determine that it is moving in the right direction or correct its route. A robot moving along a pavement in snowfall conditions can detect another robot in its path, which has a device to support the positioning of an object, without communicating with it and without requiring computationally expensive computer vision methods that may not work in such conditions.

Application

Devices, systems and methods according to the present invention can be used for positioning mobile or wearable objects - for example, UAVs performing cargo delivery, various studies, analysis of the surrounding space, object control, photo and video filming, rescue operations, autonomous monitoring for agricultural land protection tasks, cartography, remote chemical and physical analysis, control of germination and ripeness of the crop, chemical processing, etc.

Additional Implementation Features

Various illustrative blocks and modules described in connection with the disclosure herein may be implemented or executed by a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device (PLD), discrete logic element or transistor logic, discrete hardware components, or any combination of the foregoing, designed to perform the functions described in this document. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (eg, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors together with a DSP core, or any other similar configuration).

Some blocks or modules, individually or collectively, may be, for example, a computer, and include a processor that is configured to call and execute computer programs from memory to perform method steps or functions of blocks or modules in accordance with embodiments of the present invention. According to embodiments, the device may further include a memory. The processor may call and execute computer programs from memory to execute the method. The memory may be a separate device independent of the processor or may be integrated with the processor. The memory may store code, instructions, commands, and/or data for execution on a set of one or more processors of the device described. Codes, instructions, commands may direct the processor to perform the steps of a method or device function.

The functions described in this document may be implemented in hardware, software running on one or more processors, firmware, or any combination of the foregoing, as applicable. The hardware and software implementing the functions may also be physically located in different locations, including such a distribution that parts of the functions are implemented in different physical locations, that is, distributed processing or distributed computing may be performed.

If the amount of data is large, multi-threaded data processing can be performed, which in a simple representation can be expressed in the fact that the entire set of data to be processed is divided into a set of subsets, and each processor core performs processing in relation to its assigned data subset.

The above memory may be volatile or non-volatile memory, or may include both volatile and non-volatile memory. One of ordinary skill in the art will also understand that when referring to memory and storage of data, programs, codes, instructions, instructions, and the like, it is understood that there is a machine-readable (or computer-readable, processor-readable) storage medium. A computer-readable storage medium can be any available medium that can be used to carry or store a desired piece of program code in the form of instructions or data structures, and that can be accessed by a computer, processor, or other general purpose or special purpose processing device. .

By way of example, and not limitation, computer-readable media may include Read Only Memory (ROM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read Only Memory (EEPROM), flash memory, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), increased speed synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct access bus random access memory (DR RAM), and the like.

The information and signals described herein may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, and elementary signals that may be exemplified in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combinations of the above.

The processor may include one or more processors. At the same time, one or more processors may be a general purpose processor such as a central processing unit (CPU), an application processor (AP) and the like, a graphics processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU) etc.

It should be understood that although terms such as "first", "second", "third" and the like may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, areas, layers and/or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Thus, a first element, component, region, layer, or section may be referred to as a second element, component, region, layer, or section without departing from the scope of the present invention. As used herein, the term "and/or" includes any and all combinations of one or more of the respective listed positions. Elements mentioned in the singular do not exclude the plurality of elements, unless otherwise specified.

The functionality of an element specified in the description or claims as a single element may be practiced by means of several components of the device, and conversely, the functionality of elements indicated in the description or claims as several separate elements may be practiced by means of a single component.

In one embodiment, the elements/blocks/modules of the proposed device are in a common housing, can be placed on the same frame/structure/printed circuit board/crystal and are structurally connected to each other through assembly (assembly) operations and functionally through communication lines. The mentioned communication lines or channels, unless otherwise indicated, are typical communication lines known to specialists, the material implementation of which does not require creative efforts. The communication link may be a wire, a set of wires, a bus, a track, a wireless link (inductive, RF, infrared, ultrasonic, etc.). Communication protocols over communication lines are known to those skilled in the art and are not disclosed separately.

The functional connection of elements should be understood as a connection that ensures the correct interaction of these elements with each other and the implementation of one or another functionality of the elements. Particular examples of functional communication may be communication with the ability to exchange information, communication with the ability to transmit electric current, communication with the ability to transmit mechanical motion, communication with the ability to transmit light, sound, electromagnetic or mechanical vibrations, etc. The specific type of functional connection is determined by the nature of the interaction of the mentioned elements, and, unless otherwise indicated, is provided by well-known means, using principles well-known in the art.

Although the exemplary embodiments have been described in detail and shown in the accompanying drawings, it should be understood that such embodiments are illustrative only and are not intended to limit the present invention, and that the present invention should not be limited to the particular arrangements and structures shown and described, since a person skilled in the art on the basis of the information set forth in the description and the knowledge of the prior art may be obvious various other modifications and embodiments of the invention, without going beyond the essence and scope of this invention.

Claims (5)

A method for positioning an object, in which the positioned object is a mobile or wearable device containing a camera, processor and memory, the method comprising the steps of: capturing, by means of the camera of the object being positioned, an image of the approximate area in which the object positioning support device is located, comprising at least one group of light sources, each group of light sources comprising at least one main light source and at least two auxiliary light sources , and in each group of light sources, the main light source emits in a color different from the auxiliary light source, and the exact location of the light sources in the device for supporting the positioning of the object and their colors are known to the positioned object; converting, by the positional object processor, the color space of the image to the HSV color space; performing, by means of the positioned object processor, an attempt to detect individual light sources located in the area in which the device for supporting the positioning of the object is located, and/or at least one resulting light spot formed by a group or multiple groups of light sources located in the area in which hosts a device to support object positioning using color segmentation in the HSV color space; And determining, by means of the positioned object processor, the location and/or orientation of the area in which the device for supporting object positioning is located relative to the positioned object based on the detection result and/or identifying the object on which or near which the device for supporting object positioning is located, based on discovery result.

Images