CN114184193A

CN114184193A - Positioning method and system

Info

Publication number: CN114184193A
Application number: CN202010963509.1A
Authority: CN
Inventors: 李佳宁; 李�杰; 毛慧; 浦世亮
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2020-09-14
Filing date: 2020-09-14
Publication date: 2022-03-15

Abstract

The application relates to a positioning method and a positioning system, and belongs to the field of image processing. The method comprises the following steps: acquiring a first positioning image and a second positioning image, wherein the first positioning image comprises M first infrared images, the second positioning image comprises M second infrared images, the M first infrared images are obtained by performing infrared exposure on M first infrared cameras in the equipment, and the M second infrared images are obtained by performing infrared exposure on M second infrared cameras in the equipment; and positioning the equipment according to an indoor map, the first positioning image and the second positioning image, wherein the indoor map comprises map information of key frames obtained by shooting an indoor environment. This application avoids natural light to produce the influence to the location.

Description

Positioning method and system

Technical Field

The present application relates to the field of image processing, and in particular, to a positioning method and system.

Background

Simultaneous localization and mapping (SLAM) can be used for indoor global consistent localization. The SLAM can be used for establishing an indoor map while positioning the equipment. SLAM can be applied to indoor applications such as mobile service robots and sweeper, and can realize global and local path planning in indoor environments.

However, since the sparse feature points used by the current SLAM positioning technology are sensitive to indoor illumination changes, such as positioning under natural light in the daytime and positioning under light conditions at night, the positioning is greatly affected by changes of visible light.

Disclosure of Invention

In order to solve the problems in the related art, embodiments of the present application provide a positioning method and system. The technical scheme is as follows:

on one hand, the method is applied to equipment, the equipment comprises M infrared camera components, the shooting direction of each infrared camera component is different, each infrared camera component comprises a first infrared camera, a second infrared camera and an infrared light supplementing plate, the shooting direction of the first infrared camera, the shooting direction of the second infrared camera and the infrared light transmitting direction of the infrared light supplementing plate are the same, the infrared light supplementing plate is used for transmitting infrared light, and M is an integer greater than 1; the method comprises the following steps:

acquiring a first positioning image and a second positioning image, wherein the first positioning image comprises M first infrared images, the second positioning image comprises M second infrared images, the M first infrared images are obtained by performing infrared exposure on M first infrared cameras in the equipment, and the M second infrared images are obtained by performing infrared exposure on M second infrared cameras in the equipment;

and positioning the equipment according to an indoor map, the first positioning image and the second positioning image, wherein the indoor map comprises map information of key frames obtained by shooting an indoor environment.

Optionally, the positioning the device according to the indoor map, the first positioning image, and the second positioning image includes:

when the M first infrared images and the M second infrared images are images shot by the equipment for the first time, obtaining map information of a target key frame from the indoor map according to the first positioning image, wherein the image content in the target key frame and the image content in the first positioning image belong to the same indoor space;

and positioning the equipment according to the map information of the target key frame and the first positioning image.

Optionally, the map information of the target key frame includes a bag-of-words vector of the target key frame, where the bag-of-words vector is used to describe the image content of the target key frame;

the acquiring of the map information of the target key frame from the indoor map according to the first positioning image includes:

acquiring a bag-of-words vector of the first positioning image;

calculating an image distance between the first positioning image and each key frame according to the bag-of-word vector of the first positioning image and the bag-of-word vector of each key frame included in the indoor map;

and selecting a key frame, as a target key frame, from the indoor map, wherein the image content of the key frame is the same as the image content of the first positioning image, according to the image distance between the first positioning image and each key frame.

Optionally, after the positioning the device according to the indoor map, the first positioning image, and the second positioning image, the method further includes:

and when the first positioning image is a key frame, obtaining the map information of the first positioning image, and storing the map information of the first positioning image into the indoor map.

Optionally, the map information of the first positioning image includes a pose obtained by positioning the device, a bag-of-word vector of the first positioning image, feature descriptor information of at least one feature point included in the first positioning image, a three-dimensional spatial position of a map point corresponding to each feature point in the at least one feature point, and a frame identifier of a keyframe having a co-view relationship with the first positioning image.

Optionally, the M first infrared images and the M second infrared images are both infrared night vision images.

Optionally, the central wavelength of the infrared light is 850nm or 940 nm.

On the other hand, the positioning system comprises equipment and an upper computer, wherein the equipment comprises M infrared camera components, the shooting direction of each infrared camera component is different, each infrared camera component comprises a first infrared camera, a second infrared camera and an infrared light supplementing plate, the shooting direction of the first infrared camera, the shooting direction of the second infrared camera and the infrared light transmitting direction of the infrared light supplementing plate are the same, the infrared light supplementing plate is used for transmitting infrared light, and M is an integer greater than 1;

the upper computer is used for acquiring a first positioning image and a second positioning image, the first positioning image comprises M first infrared images, the second positioning image comprises M second infrared images, the M first infrared images are images obtained by performing infrared exposure on M first infrared cameras in the equipment, and the M second infrared images are images obtained by performing infrared exposure on M second infrared cameras in the equipment; and positioning the equipment according to an indoor map, the first positioning image and the second positioning image, wherein the indoor map comprises map information of key frames obtained by shooting an indoor environment.

Optionally, the upper computer is used for:

the upper computer is used for:

acquiring a bag-of-words vector of the first positioning image;

Optionally, the upper computer is further configured to:

and the storage module is used for acquiring the map information of the first positioning image when the first positioning image is a key frame, and storing the map information of the first positioning image into the indoor map.

Optionally, the central wavelength of the infrared light is 850nm or 940 nm.

In another aspect, the present application provides a non-transitory computer-readable storage medium for storing a computer program, which is loaded by a processor to execute the instructions of the positioning method.

In another aspect, the present application provides an apparatus comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the above-mentioned positioning method.

In another aspect, the present application provides a computer program product comprising a computer program stored in a computer readable storage medium, and the computer program is loaded by a processor to implement the above positioning method.

The technical scheme provided by the embodiment of the application can have the following beneficial effects:

because equipment includes M infrared camera shooting components, every infrared camera shooting component's shooting direction is different, infrared camera shooting component includes first infrared camera, second infrared camera and infrared light supplement board, can acquire first location image and second location image through equipment, first location image and second location image are infrared image, do not receive the influence of visible light when carrying out the location according to first location image and second location image like this, the map information of every key frame in the indoor map also is not influenced by visible light, the map information of every key frame in the indoor map also obtains based on infrared image, thereby can use the indoor map of having established to fix a position, improve positioning accuracy. In addition, the first positioning image comprises M first infrared images with different shooting directions, and the second positioning image comprises M second infrared images with different shooting directions, so that the equipment is positioned according to the first positioning image and the second positioning image shot by the multi-view camera, and the robustness can be better.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic structural diagram of an apparatus provided in an embodiment of the present application;

fig. 2 is a schematic structural diagram of a network architecture according to an embodiment of the present application;

fig. 3 is a flowchart of a positioning method provided in an embodiment of the present application;

fig. 4 is a flowchart of another positioning method provided in the embodiments of the present application;

FIG. 5 is a schematic diagram illustrating a position relationship among feature points, map points, and projection points according to an embodiment of the present disclosure;

fig. 6 is a flowchart of another positioning method provided in the embodiments of the present application;

FIG. 7 is a schematic diagram of a positioning system provided by an embodiment of the present application;

fig. 8 is a schematic structural diagram of an apparatus according to an embodiment of the present application.

With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

Referring to fig. 1, an embodiment of the present application provides an apparatus a, which can be applied to indoor positioning, including:

the device a comprises a processing circuit 1 and M infrared camera modules 2, wherein the processing circuit 1 is respectively connected with each infrared camera module 2 in the M infrared camera modules 2, and M is an integer larger than 1.

The imaging direction of each infrared imaging module 2 is different. For each infrared camera assembly 2, the infrared camera assembly 2 comprises a first infrared camera 21, a second infrared camera 22 and an infrared light supplement panel 23. The shooting direction of the first infrared camera 21, the shooting direction of the second infrared camera 22 and the infrared light emitting direction of the infrared light supplementing plate 23 are all the same as the shooting direction of the infrared camera assembly 2.

The infrared light compensating plate 23 is used for emitting infrared light, and the first infrared camera 21 and the second infrared camera 22 respectively perform infrared exposure.

Optionally, the M first infrared images and the M second infrared images are normal infrared night vision images.

The processing circuit 1 acquires a first infrared image generated by exposure of a first infrared camera 21 included in each infrared camera assembly 2 and a second infrared image generated by exposure of a second infrared camera 22 included in each infrared camera assembly 2, namely, M first infrared images and M second infrared images, and aligns and packages the M first infrared images and the M second infrared images.

Optionally, referring to fig. 1, for each infrared camera assembly 2, the infrared camera assembly 2 includes a first infrared camera 21 and a second infrared camera 22 arranged in parallel, and an infrared light supplement plate 23 is located between the first infrared camera 21 and the second infrared camera 22.

Optionally, the M infrared camera modules 2 may include three infrared camera modules, and the three infrared camera modules have different shooting directions, so that the three infrared camera modules can be used to perform all-around environmental scanning on the periphery of the device.

Optionally, the wavelength band of the narrowband filter of the first infrared camera 21 and the wavelength band of the narrowband filter of the second infrared camera 22 are both the same as the wavelength band of the infrared light emitted by the infrared light supplement plate 23. For example, the central wavelengths of the three bands are 850nm or 940 nm.

Optionally, the infrared light band emitted by the infrared light compensation plate 23 is different from the natural infrared light band, so that the first infrared camera 21 and the second infrared camera 22 can capture the infrared light emitted by the infrared light compensation plate 23, and thus are not interfered by the ambient light.

Optionally, referring to fig. 2, the processing circuit 1 includes a plurality of signal interfaces, and the first infrared camera 21 and the second infrared camera 22 included in each infrared camera assembly 2 are respectively connected to different signal interfaces of the processing circuit 1.

The processing circuit 1 may send a synchronous exposure signal to the first infrared camera 21 and the second infrared camera 22 included in each infrared camera assembly 2 through each signal interface according to a preset trigger frame rate. For any infrared camera module 2, the first infrared camera 21 and the second infrared camera 22 included in the infrared camera module 2 start to perform infrared exposure after receiving the synchronous exposure signal, so as to obtain a first infrared image and a second infrared image, respectively.

The processing circuit 1 also obtains a first infrared image obtained by exposure of the first infrared camera 21 and a second infrared image obtained by exposure of the second infrared camera 22 included in each infrared camera assembly 2. And processing abnormal phenomena such as frame loss and the like on the first infrared image and the second infrared image of each infrared camera assembly 2.

For example, referring to fig. 1, the apparatus includes three infrared camera assemblies 2, that is, three first infrared cameras 21 and three second infrared cameras 22 are included in total, and the processing circuit 2 may acquire one first infrared image and one second infrared image from each infrared camera assembly 2, so as to obtain six infrared images in total, and perform processing on the six infrared images for abnormal phenomena such as frame loss.

Optionally, the signal interface may be a General-purpose input/output (GPIO) interface.

Optionally, the processing circuit 1 further includes a network interface, and the processing circuit 1 establishes a network connection with the first infrared camera 21 included in each infrared camera module 2 and establishes a network connection with the second infrared camera 22 included in each infrared camera module 2 through the network interface.

The processing circuit 1 obtains a first infrared image obtained by exposure of the first infrared camera 21 and a second infrared image obtained by exposure of the second infrared camera 22 included in each infrared camera assembly 2 through the network interface.

Optionally, the network interface may be an interface such as a Local Area Network (LAN).

Optionally, the device a is installed on a mobile device, and the mobile device may be a mobile service robot or a sweeping robot, etc.

Referring to fig. 2, an embodiment of the present application provides a network architecture, including:

the equipment a is connected with the upper computer b, and the equipment a is connected with the upper computer b. The device a can pack the acquired first infrared image and the acquired second infrared image which are shot by each infrared camera shooting component 2 to obtain a data packet, and the data packet is sent to the upper computer.

Optionally, the device a establishes network connection with the upper computer b through a network interface, and the device a sends the data packet to the upper computer b through the network interface.

Optionally, the processing circuit 1 included in the device a packages the acquired first infrared image and the acquired second infrared image captured by each infrared camera module 2 to obtain a data packet, and sends the data packet to the upper computer through the network interface. For example, if the device a shown in fig. 2 includes three infrared camera assemblies 2, the processing circuit 1 of the device a may acquire six infrared images, pack the six infrared images to obtain a data packet, and send the data packet to the upper computer.

The upper computer receives the data packet, acquires the first infrared images and the second infrared images shot by each infrared camera shooting assembly from the data packet, namely M first infrared images and M second infrared images, and positions the equipment a according to the M first infrared images and the M second infrared images.

The detailed implementation process of the upper computer for positioning the device a will be described in detail in the embodiment shown in fig. 3, and will not be described in detail here.

Optionally, the network architecture further includes a forwarding device c, and the device a establishes a network connection with the upper computer b through the forwarding device c.

Optionally, the processing circuit 1 in the device a establishes a network connection with the first infrared camera 21 included in each infrared camera assembly 2 and establishes a network connection with the second infrared camera 22 included in each infrared camera assembly 2 through the forwarding device c.

Optionally, the upper computer b may be integrated on the device a. For example, the upper computer b may include a processor and a memory, and in the case where the upper computer b is integrated on the device a, the processor of the upper computer b may be connected to the processing circuit 1 in the device a.

Optionally, the upper computer b may be a notebook computer or a desktop computer, and the forwarding device c may be a switch or a router.

Referring to fig. 3, the present application provides a positioning method, where the method is applied to the above-mentioned device a, the device a includes M infrared camera assemblies, each infrared camera assembly has a different shooting direction, each infrared camera assembly includes a first infrared camera, a second infrared camera, and an infrared light supplement panel, the shooting direction of the first infrared camera, the shooting direction of the second infrared camera, and the infrared light emission direction of the infrared light supplement panel are the same, the infrared light supplement panel is used for emitting infrared light, the first infrared camera and the second infrared camera respectively perform infrared exposure, and M is an integer greater than 1; the method comprises the following steps:

step 301: the method comprises the steps of obtaining a first positioning image and a second positioning image, wherein the first positioning image comprises M first infrared images, the second positioning image comprises M second infrared images, the M first infrared images are images obtained by performing infrared exposure on M first infrared cameras in equipment a, and the M second infrared images are images obtained by performing infrared exposure on M second infrared cameras in the equipment a.

Step 302: and positioning the equipment a according to an indoor map, the first positioning image and the second positioning image, wherein the indoor map comprises map information of key frames obtained by shooting an indoor environment.

In this application embodiment, because equipment a includes M infrared camera components, every infrared camera component's shooting direction is different, and infrared camera component includes first infrared camera, second infrared camera and infrared light supplement board, can acquire first location image and second location image through equipment a, and first location image includes M first infrared image, and the second location image includes M second infrared image. Because the first positioning image and the second positioning image are both infrared images, the first positioning image and the second positioning image are not influenced by visible light when being positioned. The key frames in the indoor map are also infrared images, so that the established indoor map can be used for positioning. And the positioning precision is improved. In addition, the M first infrared images and the M second infrared images include environment content of the device a in all directions, so that the device a is located according to the first locating image and the second locating image, and robustness can be better.

Referring to fig. 4, an embodiment of the present application provides a positioning method, where the method may be applied to the network architecture shown in fig. 2, an execution main body of the method may be an upper computer, the method may position a device to be positioned in the network architecture, and the device to be positioned is the device a in the network architecture. In the method, when a device a enters a new indoor environment, the device a is positioned without saving an indoor map of the indoor environment. The method comprises the following steps:

step 401: m first infrared images and M second infrared images which are shot by equipment a for the first time are obtained, and M is an integer larger than 1.

Optionally, the M first infrared images are all left-eye images, and the M second infrared images are all right-eye images. Or the M first infrared images are all right eye images, and the M second infrared images are all left eye images.

Optionally, the shooting time of the M first infrared images and the shooting time of the M second infrared images are aligned.

Optionally, the device a includes M infrared camera assemblies, and each infrared camera assembly captures a first infrared image and a second infrared image. The method comprises the steps that a data packet sent by equipment a is received by an upper computer for the first time, wherein the data packet comprises a first infrared image and a second infrared image which are shot by each infrared camera component, namely the data packet comprises M first infrared images and M second infrared images; and acquiring M first infrared images and M second infrared images from the data packet.

The device a controls each infrared camera assembly to shoot to obtain a first infrared image and a second infrared image based on a preset trigger frame rate. The shooting time of the M first infrared images acquired by the upper computer is aligned with the shooting time of the M second infrared images.

After M first infrared images and M second infrared images are shot for the first time, when the equipment a obtains the M first infrared images and the M second infrared images, a data packet comprising the M newly shot first infrared images and the M second infrared images is sent to the upper computer. The upper computer continuously acquires the M first infrared images and the M second infrared images.

Step 402: and obtaining map information of a first key frame according to the first positioning image and the second positioning image, wherein the first positioning image comprises the M first infrared images, the second positioning image comprises the M second infrared images, and the first key frame is an image obtained by splicing the M first infrared images.

Optionally, the first positioning image is an image obtained by splicing the M first infrared images, or the first positioning image includes the M separate first infrared images. The second positioning image is an image obtained by splicing the M second infrared images, or the second positioning image includes the M separate second infrared images.

The map information of the first keyframe comprises an equipment pose corresponding to the first keyframe, a bag-of-words vector of the first keyframe, a frame identifier of the first keyframe, feature descriptor information of at least one feature point included in the first keyframe, and a three-dimensional space position of a map point corresponding to each feature point in the at least one feature point. For each feature point, the map point corresponding to the feature point is a physical point in the physical space.

The device pose corresponding to the first keyframe is the pose of device a when the first keyframe was captured, which includes the position and pose of device a.

Optionally, the step is implemented by the following operations 4021 to 4023, where the operations of 4021 to 4023 are:

4021: and initializing the pose of the device a as the device pose corresponding to the first keyframe.

Since the device a enters a new indoor environment, the M first infrared images included in the first keyframe are the first images taken by the device a when entering the indoor environment. When the device a is positioned, an indoor map used for a reference key frame required by the positioning device a is not stored, so that the pose of the device a is initialized firstly and used as the device pose corresponding to the first key frame, and the first key frame is used as the reference key frame required by the subsequent positioning of the device a.

4022: and acquiring feature descriptor information of at least one feature point in the first key frame and a bag-of-word vector of the first key frame.

The feature point of the first key frame may be a corner point in the first key frame, where the corner point is an intersection of any two lines in the first key frame. The corner points are typically the vertices of the object image in the first keyframe. For any feature point in the first keyframe, the feature descriptor information of the feature point can be represented by a high-dimensional vector.

Optionally, the feature descriptor information of the feature point is a high-dimensional vector, which is used to describe image information of an image area including the feature point.

The bag-of-words vector of the first key frame is used to describe the image information of the first key frame. The bag-of-words vector for the first keyframe may be represented by a high-dimensional vector. Optionally, the bag-of-words vector of the first keyframe is also a high-dimensional vector.

In the step, at least one feature point is identified in the first key frame through a first specified algorithm, and for each feature point in the at least one feature point, an image area with a specified size is determined based on the feature point, wherein the image area comprises the feature point; and acquiring feature descriptor information of each feature point included in the image area through a first specified algorithm according to the feature point. And acquiring a bag-of-words vector of the first key frame through a second specified algorithm according to the feature descriptor information of each feature point in the at least one feature point.

Alternatively, the feature point may be located at the center position of the image area. The specified size may be NxN, N being an integer greater than 0. For example, the specified size may be 30x30, 40x40, or 50x50, etc.

Optionally, for each first infrared image in the M first infrared images, the feature point in the first infrared image and the feature descriptor information of the feature point may be acquired in the same manner as described above. When the feature points in each first infrared image and the feature descriptor information of the feature points are acquired, the feature points in each first infrared image are used as the feature points of the first keyframe, so that the feature descriptor information of each feature point and each feature point in the first keyframe is acquired.

The first specified algorithm may be BRIEF descriptor (ORB) with Oriented FAST corner and rotation or Scale-invariant feature transform (SIFT) algorithm. The second designated algorithm is a visual bag-of-words vector algorithm, etc.

4023: and acquiring the three-dimensional space position of the map point corresponding to each feature point in the at least one feature point according to the first positioning image, the second positioning image and the equipment pose corresponding to the first key frame.

The first key frame comprises the same image content as the first positioning image, so that the at least one feature point is the feature point in the first positioning image. For each feature point of the at least one feature point, the feature point is referred to as a first feature point for ease of description.

Optionally, in this step, according to the first feature point, a second feature point corresponding to the first feature point in the second positioning image is obtained, and the first feature point and the second feature point correspond to the same map point (physical point). And acquiring the depth of the first characteristic point and the depth of the second characteristic point according to the parallax between the first characteristic point and the second characteristic point. And acquiring the three-dimensional space position of the map point corresponding to the first characteristic point according to the depth of the first characteristic point, the depth of the second characteristic point and the pose of the equipment.

Optionally, in a case that the first positioning image includes M separate first infrared images and the second positioning image includes M separate second infrared images, the first feature point belongs to one first infrared image, one second infrared image corresponding to the one first infrared image is determined, and the one first infrared image and the one second infrared image are captured by the same infrared camera module.

Therefore, the operation of acquiring the second feature point corresponding to the first feature point in the second positioning image may be: and acquiring a second characteristic point corresponding to the first characteristic point in the second infrared image.

Thus, the device pose corresponding to the first keyframe, the bag-of-word vector of the first keyframe, the feature descriptor information of at least one feature point included in the first keyframe, and the three-dimensional space position of the map point corresponding to each feature point in the at least one feature point are obtained. And combining the obtained information and the frame identification of the first key frame to form the map information of the first key frame.

Optionally, the map information of the first key frame is stored in the memory of the upper computer, so that the map information of the first key frame in the indoor map is obtained.

Step 403: acquiring M first infrared images and M second infrared images shot by the device a at the ith time, wherein M is an integer larger than 1, and i is 2, 3, 4 and … ….

In this step, a data packet sent by the device a is received, wherein the data packet comprises a first infrared image and a second infrared image shot by each infrared camera shooting assembly; and acquiring M first infrared images and M second infrared images from the data packet.

Step 404: and detecting whether the third positioning image is a key frame or not according to the map information of the second key frame, wherein the second key frame is a key frame in the indoor map, and the third positioning image comprises M first infrared images shot by the device a at the ith time.

The second key frame is the key frame last saved to the indoor map. Where i is equal to 2, 3 or 4, the second key frame may be the first key frame.

Optionally, the third positioning image is an image obtained by stitching the M first infrared images, or the third positioning image includes the M separate first infrared images.

In this step, feature descriptor information of at least one feature point in the third positioning image is obtained, and according to the feature descriptor information of the feature point in the second keyframe and the feature descriptor information of the feature point in the third positioning image, the number N of the same feature points in the second keyframe and the third positioning image is determined, where N is an integer greater than 0. And calculating a proportion value according to the same feature point number N and the feature point number included in the third positioning image. And when the proportion value exceeds a proportion threshold value, the image content of the second key frame is similar to the image content of the third positioning image, the third positioning image is determined to be a non-key frame, when the proportion value does not exceed the proportion threshold value, the image content of the second key frame is not similar to the image content of the third positioning image, and the third positioning image is determined to be a key frame.

Optionally, the third positioning image includes the M individual first infrared images, feature descriptor information of the feature point in each first infrared image is obtained, and the feature descriptor information of the feature point in each first infrared image is combined into feature descriptor information of the feature point in the third positioning image.

When the ratio value exceeds the ratio threshold, it indicates that the image content of the second key frame is similar to the image content of the third positioning image, the shooting time of the device a for shooting the second key frame and the shooting time of the device a for shooting the third positioning image may be similar, and the time difference between the two shooting times may be smaller. The second key frame and the third positioning image may be images obtained by shooting the same indoor space by the device a, and thus the third positioning image is determined to be a non-key frame.

And if the proportion value does not exceed the proportion threshold value, the image content of the second key frame is not similar to the image content of the third positioning image, and the difference between the two image contents is larger. The shooting time of the device a for shooting the second key frame and the shooting time of the device a for shooting the third positioning image may differ greatly, that is, the time difference between the two shooting times may be great. The second key frame and the third positioning image may be images obtained by shooting different indoor spaces by the device a, and thus the third positioning image is determined to be the key frame.

Optionally, the operation of determining the number of the same feature points in the second keyframe and the third positioning image may be:

for any one feature point in the third positioning image, referred to as a third feature point for convenience of explanation, a feature distance between the third feature point and each feature point in the second key frame is calculated according to the feature descriptor information of the third feature point and the feature descriptor information of each feature point in the second key frame. And selecting a minimum feature distance from the feature distances between the third feature point and each feature point in the second key frame, and when the minimum feature distance is smaller than a distance threshold, determining that the feature point corresponding to the minimum feature distance between the third feature point and the second key frame is the same feature point, namely that the feature point corresponding to the minimum feature distance between the third feature point and the second key frame corresponds to the same map point. According to the mode, all the same feature points in the second key frame and the third positioning image can be obtained, and the same feature points are counted to obtain the number of the same feature points.

Step 405: and under the condition that the third positioning image is a non-key frame, determining the pose of the equipment a according to the same feature points in the second key frame and the third positioning image.

N feature point pairs are obtained through step 404, where each feature point pair includes a third feature point and a fourth feature point, the third feature point is a feature point in the third positioning image, and the fourth feature point is a feature point in the second positioning image. Each feature point pair includes a third feature point and a fourth feature point which are the same feature point.

Alternatively, this step may be realized by the following operations 4051 to 4052. The operations 4051 to 4052 are:

4051: and acquiring the pixel distance corresponding to each characteristic point pair according to each characteristic point pair included in the N characteristic point pairs.

Referring to fig. 5, for each of the N pairs of feature points, the pair of feature points includes a third feature point P_j1And a fourth characteristic point P_j2According to the fourth characteristic point P_j2Obtaining a fourth characteristic point P_j2Three-dimensional spatial position P of the corresponding map point_j. From the three-dimensional spatial position P of the map point_jAcquiring the projection point of the map point in the third positioning image

Wherein T is_iIs a pose parameter, is an unknown value, and K is an intra-camera parameter matrix of the device a. Third characteristic point P included according to the characteristic point pair_j1And the projection point

The pixel distance corresponding to the feature point pair is calculated according to the following first formula.

The first formula is:

in the first formula, e is the corresponding pixel distance of the feature point pair. In a case that the third positioning image includes M individual first infrared images, the map point is located in a shooting direction corresponding to one of the M first infrared images, so obtaining a projection point of the map point in the third positioning image may be obtaining a projection point of the map point in the first infrared image.

4052: and acquiring the equipment pose corresponding to the third positioning image according to the pixel distance corresponding to each feature point pair by the following second formula, and taking the equipment pose as the pose of the equipment a.

The second formula is:

in the second formula, the pose parameter T is continuously adjusted in the second formula_iSuch that the value of the distance parameter T_i ^*And minimum. At a distance parameter value T_i ^*Minimum pose parameter T_iThe value of (d) is the device pose corresponding to the third positioning image.

Step 406: and under the condition that the third positioning image is the key frame, acquiring map information of the third positioning image, and taking the device pose corresponding to the third positioning image as the pose of the device a.

The fact that a co-view relationship exists between any one of the keyframes stored in the indoor map and the other keyframes means that the map points corresponding to the feature points included in one of the two keyframes and the map points corresponding to the feature points included in the other keyframe have the same map points, and the number of the same map points exceeds a number threshold.

The map information of the key frame stored in the indoor map further includes the frame identifier of other key frames having a co-view relationship with the key frame. The two key frames having the common-view relationship may be that the position interval of the device a when the two key frames are captured is within a specified interval range, the image contents of the two key frames are different, but not completely different, and a part or a small part of the same image content exists between the two key frames.

Optionally, the map information of the third positioning image obtained in this step includes an apparatus pose corresponding to the third positioning image, a bag-of-word vector of the third positioning image, a frame identifier of each key frame having a common view relationship with the third positioning image, feature descriptor information of at least one feature point included in the third positioning image, and a three-dimensional space position of a map point corresponding to each feature point in the at least one feature point.

Optionally, this step is implemented by the following operations 4061 to 4067, where the operations 4061 to 4067 are respectively:

4061: and acquiring X key frames which have a common view relation with the second key frame.

Optionally, X key frames having a co-view relationship with the second key frame may be acquired from the map information of the second key frame included in the indoor map.

When the second key frame is the first key frame, there is no key frame having a co-view relationship with the second key frame, that is, X is 0.

4062: and for each key frame in the X key frames, determining at least one feature point pair corresponding to the key frame according to the feature descriptor information of the feature points included in the key frame and the feature descriptor information of the feature points included in the third positioning image.

For each feature point pair in at least one feature point pair corresponding to the key frame, the feature point pair includes a third feature point and a fourth feature point corresponding to the same map point, the third feature point is a feature point in a third positioning image, and the fourth feature point is a feature point in the key frame.

The detailed implementation of operation 4062 can be referred to in step 404, and will not be described in detail here.

Repeatedly performing operation 4062 may obtain at least one feature point pair corresponding to each of the X key frames, assuming that a total of Y feature point pairs are obtained after performing operation 4062, where the Y feature point pairs include a feature point pair corresponding to each of X +1 key frames, and the X +1 key frames include the second key frame and the X key frames having a co-view relationship with the second key frame. Each feature point pair corresponds to one map point, i.e. a total of Y map points are obtained.

In the step, X +2 key frames are obtained in total, the X +2 key frames include the X key frames, the second key frame and the third positioning image, and the third positioning image is the X +2 key frame.

4063: and acquiring at least one pixel distance of the characteristic point pair corresponding to each key frame according to the characteristic point pair corresponding to each key frame in the X +1 key frames.

For any one of the Y feature point pairs, determining a map point j corresponding to the feature point pair and a feature point P corresponding to the map point j in the x-th key frame_jxX is 1, 2, … …, X +2, j is 1, 2, … …, Y. From the three-dimensional spatial position P of the map point j_jAcquiring the projection point of the map point in the x-th key frame

Wherein T is_xThe pose parameter for the xth keyframe is an unknown value. According to the characteristic point P_jxAnd the projection point

The distance of one pixel corresponding to the map point is calculated according to the following third formula.

The third formula is:

and e is the corresponding pixel distance of the characteristic point pair in the third formula.

4064: and acquiring an equipment pose corresponding to the third positioning image according to at least one pixel distance of each characteristic point pair by a fourth formula, and taking the equipment pose as the pose of the equipment a.

The fourth formula is:

in the fourth formula, the pose parameters T corresponding to the X +2 key frames are continuously adjusted in the fourth formula_xAnd three-dimensional spatial position P of Y map points_jSuch that the value of the distance parameter P_jT_i ^*And minimum. Adjusting P_jThe value of (1) is to adjust the three-dimensional space position of the jth map point at the distance parameter value P_jT_i ^*At minimum, the obtained pose parameter T_xThe value of (a) is the device pose corresponding to the xth keyframe, and the three-dimensional spatial positions of the Y map points are obtained. Wherein, T_x+2The value of (d) is the device pose corresponding to the (x + 2) th keyframe, and is also the device pose corresponding to the third positioning image.

Optionally, for any key frame in the first X +1 key frames, partial map points corresponding to the key frame are determined from the Y map points, the three-dimensional spatial positions of the partial map points are respectively updated to the adjusted three-dimensional spatial positions of the partial map points in the map information corresponding to the key frame in the indoor map, and the device pose of the key frame is updated to the adjusted device pose of the key frame.

4065: and acquiring a bag-of-word vector of the third positioning image according to the feature descriptor information of at least one feature point in the third positioning image.

4066: and acquiring the three-dimensional space position of the map point corresponding to each feature point in the at least one feature point according to the third positioning image, the fourth positioning image and the equipment pose corresponding to the third positioning image, wherein the fourth positioning image comprises M second infrared images shot at the ith time.

Optionally, the fourth positioning image is an image obtained by splicing the M second infrared images, or the fourth positioning image includes the M separate second infrared images.

For each feature point in the third positioning image, the feature point is referred to as a first feature point for convenience of explanation.

Optionally, in this step, according to the first feature point, a second feature point corresponding to the fourth positioning image is obtained, and the first feature point and the second feature point correspond to one map point (physical point). And acquiring the depth of the first characteristic point and the depth of the second characteristic point according to the parallax between the first characteristic point and the second characteristic point. And acquiring the three-dimensional space position of the map point corresponding to the first characteristic point according to the depth of the first characteristic point, the depth of the second characteristic point and the pose of the equipment.

4067: and acquiring the key frames which have a common view relation with the third positioning image according to the feature descriptor information of at least one feature point included in the third positioning image and the stored feature descriptor information of the feature point included in each key frame.

In this step, for any one of the key frames in the indoor map, feature descriptor information of at least one feature point included in the key frame and feature descriptor information of at least one feature point in the third positioning image are acquired, the number of the same feature points in the key frame and the third positioning image is determined according to the feature descriptor information of the feature point in the key frame and the feature descriptor information of the feature point in the third positioning image, and if the number of the same feature points exceeds a threshold value, it is determined that a common view relationship exists between the key frame and the third positioning image.

After operation 4067 is performed, the map information of the third positioning image is obtained, the third positioning image is used as a key frame, and the map information of the key frame is stored in the indoor map.

Optionally, the indoor map is stored in the memory of the upper computer, that is, the map information of the key frame is stored in the memory of the upper computer.

When the device a shoots the M first infrared images and the M second infrared images again, the operations 404 to 406 are repeatedly executed until the device a shoots the whole indoor environment, so that the indoor map information of the whole indoor environment is stored in the memory of the upper computer, that is, the indoor map of the whole indoor environment is formed.

Alternatively, the indoor map of the indoor environment may be saved in a storage medium such as a hard disk.

In this application embodiment, because equipment a includes M infrared camera components, every infrared camera component's shooting direction is different, and infrared camera component includes first infrared camera, second infrared camera and infrared light supplement board, can acquire first location image and second location image through equipment a, and first location image includes M first infrared image, and the second location image includes M second infrared image. Since the M first infrared images or the M second infrared images are infrared images in different directions, the device a is located according to the first locating image and the second locating image shot by the multi-view camera, and better robustness can be achieved. In addition, the first positioning image and the second positioning image are both infrared images generated by exposing infrared light generated by an infrared light supplement plate of the device a, so that the positioning is performed according to the first positioning image and the second positioning image without being influenced by visible light, and after the map information of each key frame is positioned, the map information of each key frame can be stored in an indoor map of an indoor environment. Because the map information of each key frame in the indoor map is not influenced by visible light, the stored indoor map can be used for positioning the equipment when the equipment is repositioned at different moments, and the positioning precision is improved.

Referring to fig. 6, an embodiment of the present application provides a positioning method, where the method may be applied to the network architecture shown in fig. 2, an execution main body of the method may be an upper computer, the method may position a device to be positioned in the network architecture, and the device to be positioned is the device a in the network architecture. In this method, when a device a enters an indoor environment in which an indoor map is stored, the device a is located. The method comprises the following steps:

step 501: m first infrared images and M second infrared images which are shot by equipment a for the first time are obtained, and M is an integer larger than 1.

Step 502: and acquiring a bag-of-words vector of the first positioning image according to the first positioning image, wherein the first positioning image comprises the M first infrared images.

Optionally, a detailed implementation process of obtaining the bag-of-words vector of the first positioning image may refer to relevant contents in the operation of the above 4022, and is not described in detail here.

Step 503: and obtaining the map information of the second key frame according to the bag-of-word vector of the first positioning image and the indoor map of the indoor environment.

In this step, according to the bag-of-word vector of the first positioning image and the bag-of-word vector of each key frame included in the indoor map, the image distance between the first positioning image and each key frame is calculated, the key frame corresponding to the minimum image distance is obtained as the second key frame, and the map information of the second key frame is obtained from the indoor map.

Wherein, the smaller the image distance between the first positioning image and the key frame, the more similar the image contents of the first positioning image and the key frame. The image distance between the second key frame and the first positioning image is minimum, the image content of the second key frame and the image content of the first positioning image are the content of the same indoor space, namely the second key frame and the first positioning image are images obtained by shooting the same indoor space by the device a.

Step 504: and determining the pose of the equipment a according to the same characteristic points in the second key frame and the first positioning image.

During implementation, feature descriptor information of at least one feature point in the first positioning image is obtained, the same feature point in the second key frame and the first positioning image is determined according to the feature descriptor information of the feature point in the second key frame and the feature descriptor information of the feature point in the first positioning image, Z feature point pairs are obtained, Z is an integer larger than 0, and each feature point pair comprises two same feature points. And acquiring the pixel distance corresponding to each characteristic point pair according to each characteristic point pair included in the Z characteristic point pairs. And acquiring the equipment pose corresponding to the first positioning image according to the pixel distance corresponding to each characteristic point pair by using a second formula, and taking the equipment pose as the pose of the equipment a.

Step 505: acquiring M first infrared images and M second infrared images shot by the device a at the ith time, wherein M is an integer larger than 1, and i is 2, 3, 4 and … ….

Step 506-508: respectively, as in

steps

404 and 406, and will not be described in detail herein.

In this application embodiment, because equipment a includes M infrared camera components, every infrared camera component's shooting direction is different, infrared camera component includes first infrared camera, second infrared camera and infrared light supplement board, can acquire first location image and second location image through equipment a, first location image includes M first infrared image, second location image includes M second infrared image to first location image and second location image according to the shooting of many mesh camera are fixed a position equipment a, can have better robustness. In addition, the first positioning image and the second positioning image are both infrared images, so that positioning is carried out according to the first positioning image and the second positioning image without being influenced by visible light, the map information of each key frame can be stored in an indoor map of an indoor environment when the map information of each key frame is positioned, the stored indoor map can be used for positioning the equipment when relocation is carried out, and the positioning accuracy is improved.

Referring to fig. 7, the embodiment of the present application provides a positioning system 600, the system 600 includes a device 601 and an upper computer 602, the device 601 includes M infrared camera modules, each infrared camera module has a different shooting direction, each infrared camera module includes a first infrared camera, a second infrared camera and an infrared light compensation plate, the shooting direction of the first infrared camera, the shooting direction of the second infrared camera and the infrared light emission direction of the infrared light compensation plate are the same, the infrared light compensation plate is used for emitting infrared light, the first infrared camera and the second infrared camera respectively perform infrared exposure, and M is an integer greater than 1.

The upper computer can be but is not limited to a processor, a server, a cloud computing platform or a computer and the like.

The upper computer 602 is configured to obtain a first positioning image and a second positioning image, where the first positioning image includes M first infrared images, the second positioning image includes M second infrared images, the M first infrared images are images obtained by performing infrared exposure on M first infrared cameras in the device, and the M second infrared images are images obtained by performing infrared exposure on M second infrared cameras in the device; and positioning the equipment according to an indoor map, the first positioning image and the second positioning image, wherein the indoor map comprises map information of key frames obtained by shooting an indoor environment.

Optionally, the upper computer 602 is configured to:

the upper computer 602 is used for:

acquiring a bag-of-words vector of the first positioning image;

Optionally, the upper computer 602 is further configured to:

Optionally, the central wavelength of the infrared light is 850nm or 940 nm.

In this application embodiment, because equipment a includes M infrared camera components, every infrared camera component's shooting direction is different, infrared camera component includes first infrared camera, second infrared camera and infrared light supplement board, the module of acquireing can acquire first location image and second location image through equipment a, first location image includes M first infrared image, second location image includes M second infrared image to fix a position equipment a according to first location image and the second location image that the many mesh camera was shot, can have better robustness. In addition, the first positioning image and the second positioning image are both infrared images, so that the positioning module is not influenced by visible light when positioning is carried out according to the first positioning image and the second positioning image, and therefore the established indoor map is used for positioning.

Fig. 8 shows a block diagram of an apparatus 700 provided in an exemplary embodiment of the invention. The device 700 may be a portable mobile terminal. Device 700 may also be referred to by other names such as user equipment, portable terminals, laptop terminals, desktop terminals, and the like. Optionally, the device a may be the upper computer b or the device a provided in any of the above embodiments.

In general, the apparatus 700 includes: a processor 701 and a memory 702.

The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 701 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 701 may further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

Memory 702 may include one or more computer-readable storage media, which may be non-transitory. Memory 702 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 702 is used to store at least one instruction for execution by processor 701 to implement a positioning method provided by method embodiments herein.

In some embodiments, the apparatus 700 may further optionally include: a peripheral interface 703 and at least one peripheral. The processor 701, the memory 702, and the peripheral interface 703 may be connected by buses or signal lines. Various peripheral devices may be connected to peripheral interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 704, touch screen display 705, camera 706, audio circuitry 707, positioning components 708, and power source 709.

The peripheral interface 703 may be used to connect at least one peripheral related to I/O (Input/Output) to the processor 701 and the memory 702. In some embodiments, processor 701, memory 702, and peripheral interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral interface 703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The Radio Frequency circuit 704 is used for receiving and transmitting RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuitry 704 communicates with communication networks and other communication devices via electromagnetic signals. The rf circuit 704 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuitry 704 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: the world wide web, metropolitan area networks, intranets, generations of mobile communication networks (2G, 3G, 4G, and 5G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 704 may also include NFC (Near Field Communication) related circuits, which are not limited in this application.

The display screen 705 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to capture touch signals on or over the surface of the display screen 705. The touch signal may be input to the processor 701 as a control signal for processing. At this point, the display 705 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display 705 may be one, providing the front panel of the device 700; in other embodiments, the display 705 may be at least two, each disposed on a different surface of the device 700 or in a folded design; in still other embodiments, the display 705 may be a flexible display disposed on a curved surface or on a folded surface of the device 700. Even more, the display 705 may be arranged in a non-rectangular irregular pattern, i.e. a shaped screen. The Display 705 may be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or the like.

The camera assembly 706 is used to capture images or video. Optionally, camera assembly 706 includes a front camera and a rear camera. Generally, a front camera is disposed on a front panel of the apparatus, and a rear camera is disposed on a rear surface of the apparatus. In some embodiments, the number of the rear cameras is at least two, and each rear camera is any one of a main camera, a depth-of-field camera, a wide-angle camera and a telephoto camera, so that the main camera and the depth-of-field camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize panoramic shooting and VR (Virtual Reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 706 may also include a flash. The flash lamp can be a monochrome temperature flash lamp or a bicolor temperature flash lamp. The double-color-temperature flash lamp is a combination of a warm-light flash lamp and a cold-light flash lamp, and can be used for light compensation at different color temperatures.

The audio circuitry 707 may include a microphone and a speaker. The microphone is used for collecting sound waves of a user and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 701 for processing or inputting the electric signals to the radio frequency circuit 704 to realize voice communication. The microphones may be multiple and placed at different locations on the device 700 for stereo sound acquisition or noise reduction purposes. The microphone may also be an array microphone or an omni-directional pick-up microphone. The speaker is used to convert electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The loudspeaker can be a traditional film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, the speaker can be used for purposes such as converting an electric signal into a sound wave audible to a human being, or converting an electric signal into a sound wave inaudible to a human being to measure a distance. In some embodiments, the audio circuitry 707 may also include a headphone jack.

The positioning component 708 is operative to locate a current geographic Location of the device 700 to enable navigation or LBS (Location Based Service). The Positioning component 708 can be a Positioning component based on the Global Positioning System (GPS) in the united states, the beidou System in china, or the galileo System in russia.

A power supply 709 is used to supply power to the various components in the device 700. The power source 709 may be alternating current, direct current, disposable batteries, or rechargeable batteries. When the power source 709 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, the device 700 also includes one or more sensors 710. The one or more sensors 710 include, but are not limited to: acceleration sensor 711, gyro sensor 712, pressure sensor 713, fingerprint sensor 714, optical sensor 715, and proximity sensor 716.

The acceleration sensor 711 can detect the magnitude of acceleration in three coordinate axes of a coordinate system established with the apparatus 700. For example, the acceleration sensor 711 may be used to detect components of the gravitational acceleration in three coordinate axes. The processor 701 may control the touch screen 705 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 711. The acceleration sensor 711 may also be used for acquisition of motion data of a game or a user.

The gyro sensor 712 may detect a body direction and a rotation angle of the device 700, and the gyro sensor 712 may cooperate with the acceleration sensor 711 to acquire a 3D motion of the device 700 by the user. From the data collected by the gyro sensor 712, the processor 701 may implement the following functions: motion sensing (such as changing the UI according to a user's tilting operation), image stabilization at the time of photographing, game control, and inertial navigation.

Pressure sensors 713 may be disposed on a side bezel of device 700 and/or an underlying layer of touch display 705. When the pressure sensor 713 is disposed on a side frame of the device 700, a user's holding signal of the device 700 may be detected, and the processor 701 may perform right-left hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 713. When the pressure sensor 713 is disposed at a lower layer of the touch display 705, the processor 701 controls the operability control on the UI interface according to the pressure operation of the user on the touch display 705. The operability control comprises at least one of a button control, a scroll bar control, an icon control and a menu control.

The fingerprint sensor 714 is used for collecting a fingerprint of a user, and the processor 701 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 714, or the fingerprint sensor 714 identifies the identity of the user according to the collected fingerprint. When the user identity is identified as a trusted identity, the processor 701 authorizes the user to perform relevant sensitive operations, including unlocking a screen, viewing encrypted information, downloading software, paying, changing settings, and the like. The fingerprint sensor 714 may be disposed on the front, back, or side of the device 700. When a physical key or vendor Logo is provided on the device 700, the fingerprint sensor 714 may be integrated with the physical key or vendor Logo.

The optical sensor 715 is used to collect the ambient light intensity. In one embodiment, the processor 701 may control the display brightness of the touch display 705 based on the ambient light intensity collected by the optical sensor 715. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 705 is increased; when the ambient light intensity is low, the display brightness of the touch display 705 is turned down. In another embodiment, processor 701 may also dynamically adjust the shooting parameters of camera assembly 706 based on the ambient light intensity collected by optical sensor 715.

A proximity sensor 716, also known as a distance sensor, is typically provided on the front panel of the device 700. The proximity sensor 716 is used to capture the distance between the user and the front of the device 700. In one embodiment, the processor 701 controls the touch display 705 to switch from the bright screen state to the dark screen state when the proximity sensor 716 detects that the distance between the user and the front surface of the device 700 is gradually decreased; when the proximity sensor 716 detects that the distance between the user and the front surface of the device 700 is gradually increased, the processor 701 controls the touch display 705 to switch from the breath-screen state to the bright-screen state.

Those skilled in the art will appreciate that the configuration shown in fig. 8 does not constitute a limitation of the device 700 and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components may be employed.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A positioning method is characterized in that the method is used for positioning equipment, the equipment comprises M infrared camera components, the shooting direction of each infrared camera component is different, each infrared camera component comprises a first infrared camera, a second infrared camera and an infrared light supplementing plate, the shooting direction of the first infrared camera, the shooting direction of the second infrared camera and the infrared light transmitting direction of the infrared light supplementing plate are the same, the infrared light supplementing plate is used for transmitting infrared light, and M is an integer larger than 1; the method comprises the following steps:

2. The method of claim 1, wherein said locating the device based on the indoor map, the first positioning image, and the second positioning image comprises:

when the first positioning image and the second positioning image are images shot by the equipment for the first time, obtaining map information of a target key frame from the indoor map according to the first positioning image, wherein the image content in the target key frame and the image content in the first positioning image belong to the same indoor space;

3. The method of claim 2, wherein the map information of the target key frame comprises a bag-of-words vector of the target key frame, the bag-of-words vector describing image content of the target key frame;

acquiring a bag-of-words vector of the first positioning image;

4. The method of claim 1, wherein after locating the device based on the indoor map, the first positioning image, and the second positioning image, further comprising:

5. The method of claim 4, wherein the map information of the first localization image comprises a pose obtained by locating the device, a bag of words vector of the first localization image, feature descriptor information of at least one feature point included in the first localization image, a three-dimensional spatial position of a map point corresponding to each of the at least one feature point, and a frame identification of a key frame having a co-view relationship with the first localization image.

6. The method of any of claims 1 to 5, wherein the M first infrared images and the M second infrared images are both infrared night vision images.

7. The method of any of claims 1 to 5, wherein the infrared light has a center wavelength of 850nm or 940 nm.

8. A positioning system is characterized by comprising equipment and an upper computer, wherein the equipment comprises M infrared camera components, the shooting direction of each infrared camera component is different, each infrared camera component comprises a first infrared camera, a second infrared camera and an infrared light supplementing plate, the shooting direction of the first infrared camera, the shooting direction of the second infrared camera and the infrared light transmitting direction of the infrared light supplementing plates are the same, the infrared light supplementing plates are used for transmitting infrared light, and M is an integer greater than 1;

9. The system of claim 8, wherein the upper computer is to:

10. The system of claim 9, wherein the map information of the target key frame includes a bag-of-words vector of the target key frame, the bag-of-words vector describing image content of the target key frame;

the upper computer is used for:

acquiring a bag-of-words vector of the first positioning image;

11. The system of claim 8, wherein the host computer is further configured to:

12. The system of claim 11, wherein the map information of the first positioning image comprises a pose at which the device is positioned, a bag of words vector of the first positioning image, feature descriptor information of at least one feature point comprised by the first positioning image, a three-dimensional spatial position of a map point corresponding to each of the at least one feature point, and a frame identification of a keyframe having a co-view relationship with the first positioning image.