CN112204345A

CN112204345A - Indoor positioning method of mobile equipment, mobile equipment and control system

Info

Publication number: CN112204345A
Application number: CN202080003090.3A
Authority: CN
Inventors: 不公告发明人
Original assignee: Shankou Shenzhen Intelligent Technology Co ltd
Current assignee: Shankou Shenzhen Intelligent Technology Co ltd
Priority date: 2020-01-20
Filing date: 2020-01-20
Publication date: 2021-01-08
Also published as: WO2021146862A1

Abstract

The application discloses indoor positioning method, mobile device (6) and control system (7,8) of mobile device (6), indoor positioning method includes: acquiring a first image (C1) and a second image (C2) which are respectively captured at different positions by a camera (601,801); wherein, the second image (C2) has a second image feature (b1, b3) matching the first image feature (a1, a2) in the first image in the pixel unit, and the first image feature (a1, a2) and the second image feature (b1, b3) form an image feature pair; determining position change information of the mobile device (6) between the different positions depending on current spatial scale parameters and pixel position offsets of the image feature pairs in the first image (C1) and the second image (C2). The indoor positioning method can accurately position the mobile robot, and improves the positioning precision of the mobile robot.

Description

Indoor positioning method of mobile equipment, mobile equipment and control system

Technical Field

The present application relates to the field of positioning technologies, and in particular, to an indoor positioning method for a mobile device, and a control system.

Background

A mobile robot is a machine device that automatically performs work. It can accept human command, run the program programmed in advance, and also can operate according to the principle outline action made by artificial intelligence technology. The mobile robot can be used indoors or outdoors, can be used for industry or families, can be used for replacing security patrol, replacing people to clean the ground, and can also be used for family companions, auxiliary office work and the like.

Depending on the application fields of different mobile robots, the mobile robots used in the respective fields may be moved in different ways, for example, the mobile robots may be wheeled, walking, chain, or the like. With the updating iteration of the mobile technology of the mobile robot, the mobile information provided by the sensor is utilized to perform instant positioning and Mapping (SLAM) so as to provide more accurate navigation capability for the mobile robot, and the mobile robot can move autonomously more effectively.

However, for a robot cleaner, the distance traveled by the wheels varies from one floor to another. When, for example, the mobile robot is jammed or the wheels slip, a movement information error is caused, so that a large difference may occur in the positioning of the sweeping robot.

Disclosure of Invention

In view of the above-mentioned shortcomings of the related art, an object of the present application is to provide an indoor positioning method for a mobile device, a mobile device and a control system, which are used to solve the problem of inaccurate positioning of the mobile device in the prior art.

To achieve the above and other related objects, a first aspect of the present application discloses an indoor positioning method for a mobile device, where the mobile device includes a depth detection device and a camera device, where the camera device captures an image of an indoor environment, and the depth detection device provides depth information corresponding to at least one pixel unit in the image in an indoor space, where the depth information is used to determine a spatial scale parameter; the indoor positioning method comprises the following steps: acquiring a first image and a second image which are respectively shot at different positions by the camera device; the pixel unit of the second image is provided with a second image feature matched with the first image feature in the first image, and an image feature pair is formed between the first image feature and the second image feature; and determining position change information of the mobile equipment between different positions according to the current spatial scale parameter and the pixel position offset of the image feature pair in the first image and the second image.

In some embodiments of the first aspect, the current spatial scale parameter is a proportional relationship determined according to depth information of image features in corresponding pixel units detected by the depth detection device and depth information of corresponding image features in a map coordinate system of the camera device.

In some embodiments of the first aspect, the current spatial scale parameter is set according to depth information corresponding to the pixel unit in the first image or the second image; or the current spatial scale parameter is obtained according to historical depth information of the depth detection device or acquired from historical data of the spatial scale parameter.

In some embodiments of the first aspect, a relative positional relationship between the imaging device and the depth detection device is preset.

In some embodiments of the first aspect, a previous image of the first and second images is captured by a camera during movement of the mobile device; alternatively, the prior image is from a map database of the indoor space.

In some embodiments of the first aspect, the map database includes landmark information; wherein the landmark information includes: the prior image, and the matching image features located in the prior image.

In some embodiments of the first aspect, the positioning method further comprises: and determining the position information of the mobile equipment in the indoor space according to the position change information and a preset map database of the indoor space.

In some embodiments of the first aspect, the positioning method further comprises the steps of: based on the determined location change information and at least one of: the first image, the second image, and the depth information, updating a map database of the indoor space.

In some embodiments of the first aspect, the positioning method further comprises: updating a positional relationship between the mobile device and a preset navigation route based on the determined position change information.

In some embodiments of the first aspect, the positioning method further comprises: and during the continuous movement of the mobile equipment, identifying that the determined position change information corresponds to the entity object with the suspension space according to the depth information sequence provided by the depth detection device at different positions.

In some embodiments of the first aspect, the physical object comprises at least one of: door, furniture.

In some embodiments of the first aspect, the mobile device further comprises an inertial navigation detection apparatus, and the indoor positioning method further comprises: and correcting the inertial navigation data provided by the inertial navigation detection device by using the determined position change information.

A second aspect of the present application provides a mobile device comprising: an image pickup device for picking up an image; depth detection means for detecting depth information; and the processing device is connected with the depth detection device and the camera device and is used for executing the indoor positioning method in the first aspect.

In some embodiments of the second aspect, the camera device is spaced from the depth detection device by no more than 3 cm.

In some embodiments of the second aspect, an axis of the depth detection device is parallel to an optical axis of the image capture device, and an angle between the axis and the optical axis is in a range from 0 degrees to a maximum angle of view of the image capture device, or an angle between the axis and the optical axis is in a range from 0 degrees to a minimum angle of view of the image capture device.

In some embodiments of the second aspect, the depth detection device comprises a laser ranging device, or a single point ToF sensor, or an ultrasonic sensing device.

In some embodiments of the second aspect, the mobile device comprises any of: mobile robot, intelligent terminal.

In some embodiments of the second aspect, the mobile robot is a sweeping robot; correspondingly, the sweeping robot further comprises a cleaning device for executing cleaning operation.

A third aspect of the present application provides a control system of a mobile device, comprising: the interface device is used for connecting the depth detection device and the camera device in the mobile equipment; the axis of the depth detection device and the optical axis of the camera device have a preset angle relationship, so that the depth information measured by the depth detection device corresponds to a pixel unit in an image shot by the camera device; storage means for storing at least one program; and the processing device is connected with the interface device and the storage device and used for calling and executing the at least one program so as to coordinate the storage device, the depth detection device and the camera device to execute and realize the indoor positioning method according to any one of the first aspect.

A fourth aspect of the present application provides a control system of a mobile device, comprising: the camera shooting device is used for shooting an image of an indoor environment; the depth detection device is used for providing depth information of at least one pixel unit in the image in the indoor space; the axis of the depth detection device and the optical axis of the camera device have a preset angular relationship, so that the depth information measured by the depth detection device corresponds to a pixel unit in an image shot by the camera device; the interface device is connected with the depth detection device and the camera device; storage means for storing at least one program; and the processing device is connected with the interface device and the storage device and used for calling and executing the at least one program so as to coordinate the storage device, the depth detection device and the camera device to execute and realize the indoor positioning method according to any one of the first aspect.

A fifth aspect of the present application provides a computer-readable storage medium, in which at least one program is stored, and the at least one program executes and implements the indoor positioning method according to any one of the first aspects when being called.

To sum up, the indoor positioning method of the mobile device, the mobile device and the control system provided by the application take at least two images through the camera device, match the image characteristics in the two images, and acquire the position change information of the mobile device in the indoor space according to the pixel position offset of the matched image characteristics and the space scale parameter obtained according to the depth information of the feature object of the indoor space corresponding to the image characteristics acquired by the depth detection device, so that the indoor positioning of the mobile device is realized, and the positioning accuracy is improved.

Drawings

Fig. 1A is a schematic diagram illustrating a determination of depth information for a sweeping robot in a scene according to the present application.

FIG. 1B shows a schematic diagram of determining depth information for a service robot in a scenario according to the present application.

Fig. 1C shows a schematic diagram of determining depth information for the intelligent terminal in a scenario according to the present application.

Fig. 2 is a flowchart illustrating an indoor positioning method for a mobile device according to an embodiment of the present disclosure.

Fig. 3 is a schematic diagram illustrating image feature matching according to the present application.

FIG. 4 is a schematic diagram of image features of the captured image of FIG. 1A in an image pixel coordinate system.

Fig. 5 is a schematic view illustrating a scenario of an indoor positioning method for a mobile device according to an embodiment of the present disclosure.

Fig. 6 is a schematic structural diagram of a mobile device according to an embodiment of the present application.

Fig. 7 is a schematic structural diagram of a control system of a mobile device according to an embodiment of the present application.

Fig. 8 is a schematic structural diagram of a control system of another mobile device according to an embodiment of the present application.

Fig. 9 shows a mapping relationship between the map coordinate system and a pixel point p' in an Image1 captured by the imaging device.

Detailed Description

The following description of the embodiments of the present application is provided for illustrative purposes, and other advantages and capabilities of the present application will become apparent to those skilled in the art from the present disclosure.

In the following description, reference is made to the accompanying drawings that describe several embodiments of the application. It is to be understood that other embodiments may be utilized and that mechanical, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present disclosure. The following detailed description is not to be taken in a limiting sense, and the scope of embodiments of the present application is defined only by the claims of the issued patent. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. Spatially relative terms, such as "upper," "lower," "left," "right," "lower," "below," "lower," "above," "upper," and the like, may be used herein to facilitate describing one element or feature's relationship to another element or feature as illustrated in the figures.

Although the terms first, second, etc. may be used herein to describe various elements or parameters in some instances, these elements or parameters should not be limited by these terms. These terms are only used to distinguish one element or parameter from another element or parameter. For example, a first image may be referred to as a second image, and similarly, a second image may be referred to as a first image, without departing from the scope of the various described embodiments. The first image and the second image are both describing one image, but they are not the same image unless the context clearly indicates otherwise. Similar situations also include the first image feature and the second image feature.

Also, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and/or "including," when used in this specification, specify the presence of stated features, steps, operations, elements, components, items, species, and/or groups, but do not preclude the presence, or addition of one or more other features, steps, operations, elements, components, species, and/or groups thereof. The terms "or" and/or "as used herein are to be construed as inclusive or meaning any one or any combination. Thus, "A, B or C" or "A, B and/or C" means "any of the following: a; b; c; a and B; a and C; b and C; A. b and C ". An exception to this definition will occur only when a combination of elements, functions, steps or operations are inherently mutually exclusive in some way.

At present, the depth detection is widely applied to various fields such as smart home, portable electronic equipment, games, and later optimization.

Taking the smart terminal as an example, for example, the head-mounted device may utilize an Augmented Reality (AR) technology to implement an immersive field of view experience such as an AR game, an AR decoration, a motion sensing game, or a holographic interaction through the environment image and the depth information provided by the depth camera. How to acquire the position change information of the body limb of the operator or the position change information of the operating device is crucial to whether a more realistic immersive field of view experience can be provided for the operator.

Taking a mobile robot as an example, in general, the mobile robot performs positioning and motion control by using an Inertial Measurement Unit (IMU), which includes a gyroscope, an accelerometer, and the like, and obtains a pose of the mobile robot by measuring an angular velocity and an acceleration of the mobile robot. However, there is an error in positioning in this way. For example, when the traveling wheels of the mobile robot slip or walk on the ground made of different materials, the data provided by the inertial measurement unit is different from the actual data.

In view of this, the present application provides an indoor positioning method for a mobile device, which accurately obtains a position of the mobile device by using an image captured by an image capturing device and depth information provided by a depth detection device, and improves positioning accuracy of the mobile device.

The mobile device refers to a mobile device that can be displaced within a physical space. The displacement refers to an offset between a position of the mobile equipment at the previous moment and a position at the next moment, the offset has a direction, and the direction points to the position at the next moment from the position at the previous moment. The displacement is used for describing the change of the position of the mobile device or describing the motion track or mode of the mobile device. For example, the movement pattern of the mobile device includes, but is not limited to, a linear movement, a polygonal movement, a curved movement, and the like.

In some embodiments, the mobile device is controlled and displaced by its own control system, and the mobile device includes but is not limited to: one or more of devices capable of autonomous movement, such as an unmanned aerial vehicle, an industrial robot, a home-companion mobile device, a medical mobile device, a cleaning robot, a smart vehicle, and a patrol mobile device.

In some embodiments, the mobile device is displaced by an external actuation. For example, the mobile device may be worn on a human body and displaced by the activity of the human, or the mobile device may be mounted on a device capable of autonomous movement (e.g., mounted on a vehicle) and displaced by the movement of the device. In this embodiment, the mobile device includes, but is not limited to: one or more of a wearable electronic device such as a Head Mounted Display (HMD), smart glasses, and a bracelet, a smart phone, a tablet, and a notebook.

It should be noted that the positioning method provided by the embodiment of the present application is applicable to an indoor space. The indoor space is a physical space where a boundary exists, such as an indoor environment of a home, a public place (e.g., an office, a mall, a hospital, a parking lot, and a bank), and the like.

In some embodiments, the indoor space may also include room dividers therein. The room partition is a facade in a physical space, such as a wall, partition, French window, ceiling, etc., that forms a boundary of the space.

It should be clear to those skilled in the art that the coordinate system established according to the indoor space is a world coordinate system. For example, the origin, x-axis, and y-axis of the world coordinate system may be set on the ground, and the height direction of the indoor space may be taken as the z-axis. Typically, the world coordinate system has units of meters (m); of course, the world coordinate system is not limited thereto, and may be in units of decimeters (dm), centimeters (cm), millimeters (mm), or the like, for example, according to the actual situation. It should be understood that the world coordinate system is only an example and not a limitation.

The mobile device comprises a camera device for taking an image of the indoor space. In some embodiments, the imaging device includes, but is not limited to: cameras, video cameras, camera modules integrated with optical systems or CCD chips, and camera modules integrated with optical systems and CMOS chips, and the like. According to the requirement of actual imaging, the lens that can be adopted by the camera or the video camera includes but is not limited to: standard lenses, telephoto lenses, fisheye lenses, wide-angle lenses, and the like. For convenience of description, the embodiment of the present application takes the image pickup apparatus as an example of a video camera. Those skilled in the art will appreciate that the examples do not limit the scope of the specific embodiments.

The image pickup device can be used for picking up one or more of a single image, a continuous image sequence, a discontinuous image sequence, a video and the like. In some embodiments, the mobile device stores the captured image in a local storage medium or transmits the captured image to an external device for storage, wherein the communication connection comprises a wired or wireless communication connection. The storage medium may include read-only memory, random-access memory, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, a usb disk, a removable hard disk, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed.

In some embodiments, the external device may be a server located in a network, and the server includes, but is not limited to, one or more of a single server, a server cluster, a distributed server cluster, a cloud server, and the like. In a specific implementation, the cloud server may be a cloud computing platform provided by a cloud computing provider. Based on the architecture of the cloud server, the types of the cloud server include but are not limited to: Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS). Based on the nature of the cloud server, the types of the cloud server include, but are not limited to: public Cloud (Public Cloud) server, Private Cloud (Private Cloud) server, Hybrid Cloud (Hybrid Cloud) server, and the like.

In some embodiments, the public Cloud service is, for example, Amazon's elastic computing Cloud (Amazon EC2), IBM's Blue Cloud, google's appeengine, and Windows' Azure service platform; the private cloud service end is, for example, an aristoson cloud computing service platform, an Amazon cloud computing service platform, a hundredth cloud computing platform, an Tencent cloud computing platform, and the like.

The position of the camera on the mobile device can be determined according to the type of the mobile device and/or the application scene. In some embodiments, when the mobile device is, for example, a mobile robot, the camera may be disposed on a top surface (e.g., a central region of the top surface, a front end of the top surface opposite the central region, a rear end of the top surface opposite the central region), a side surface, or an intersection of the top surface and the side surface of the mobile robot, so as to capture images of a working environment of the mobile device for subsequent possible object recognition, map construction, real-time positioning, or virtual simulation, and the like. In some embodiments, when the mobile device is, for example, a smart terminal, the camera device may be disposed on an outer surface of the smart terminal, such as a region near an edge on a display side, a central region on the display side, a region near an edge on a back side of the display side, a central region on the back side, and the like. Alternatively, the camera device may be telescopically disposed inside the smart terminal, and extend out of the surface of the smart terminal when an image needs to be captured, and so on. Correspondingly, the number of the camera devices can be set according to actual requirements; in some embodiments, the camera may also be movable, for example, to adjust its optical axis direction and to be positioned at a movable position.

In some embodiments, the field angle of the imaging device is determined by parameters of the imaging device. The parameters of the image pickup device include internal parameters, external parameters, and distortion parameters, wherein the internal parameters include but are not limited to: one or more of a focal length, a physical size corresponding to each pixel, a pixel center, etc., including but not limited to one or more of a position of the camera on the mobile device, a rotational orientation, a translation matrix, etc.

In some embodiments, the range of field angles of the camera device includes, but is not limited to: 10 to 120 degrees. For example, the angle of view is 10 degrees, 20 degrees, 30 degrees, 40 degrees, 50 degrees, 60 degrees, 70 degrees, 80 degrees, 90 degrees, 100 degrees, 110 degrees, 120 degrees. The above range is only an example, and the precision of the viewing angle is not limited to be within a range of 10 degrees, and the precision of the viewing angle may be higher according to actual design requirements, such as 1 degree, 0.1 degree, 0.01 degree or more.

In some embodiments, the range of the optical axis of the camera device includes, but is not limited to: the included angle of the indoor space relative to the height direction is 0 degree to +/-30 degrees, or 60 degrees to 120 degrees relative to the ground. For example, the optical axis of the imaging device may be at an angle of-30 degrees, -29 degrees, -28 degrees, -27 degrees … … -2 degrees, -1 degree, 0 degrees, 1 degree, 2 degrees … … 29 degrees, or 30 degrees with respect to the vertical. For another example, the included angles of the optical axis of the imaging device with respect to the ground plane are 60 degrees, 61 degrees, 62 degrees … 89 degrees, 90 degrees, 91 degrees … 119 degrees, and 120 degrees. It should be noted that the angle between the optical axis of the above-mentioned camera device and the vertical line or the horizontal line is only an example, and is not limited to the range of the accuracy of the angle being 1 degree, and the accuracy of the angle can be higher according to the actual design requirement, such as reaching 0.1 degree, more than 0.01 degree, and the like.

In some embodiments, the optical axis of the camera is determined by the orientation of the camera. The orientation of the camera may be preset, for example, at a fixed angle according to the structure of the mobile device. The orientation of the camera device can also be adjusted manually according to actual needs, or by a control system of the mobile device, for example, the camera device adjusts the horizontal and pitch angles by means of a pan-tilt.

According to the parameters of the camera device, the world coordinate system is subjected to similarity transformation to form a map coordinate system. Wherein the similarity transformation comprises a scale transformation, a rotation transformation, and a translation transformation. Here, the unit of the map coordinate system may be a custom voxel unit, which may or may not be related to a length unit. The length unit is, for example, meter, decimeter, centimeter, millimeter, etc. Here, the map coordinate system may be a coordinate system in a virtual space or a map coordinate system constructed by the mobile device for the physical space.

Taking the map coordinate system as an example of the coordinate system in the virtual space, each pixel point in the image shot by the camera device is mapped to the virtual three-dimensional space of the map coordinate system. In some examples, the map coordinate system is co-ordinated with a camera at an initial position of the mobile deviceThe systems are overlapped, and a map of a virtual space corresponding to a physical space in which the mobile device is located is constructed by analyzing features in images photographed at preset intervals and preset voxel units. For example, please refer to fig. 9, which shows a map coordinate diagram illustrating the position relationship of the mobile device at the position O and the position a respectively relative to the measuring point M on the lamp in the map coordinate system, wherein the coordinate system XYZ is a map coordinate system, O is an origin of the map coordinate system, and during the movement in the physical space, the mobile device captures two images including a same real measurement point M' (not shown) of the lamp in the physical space at a predetermined time interval, wherein, the distance moved by the mobile equipment at the preset time interval is corresponding to the unit distance D in the map coordinate system, and the posture change between the position O and the position A of the mobile equipment in the map coordinate system is determined by utilizing the matched image features feature1 and feature2 respectively corresponding to the measuring point M' in the two pictures Pic1 and Pic2, thereby determining the coordinate (X) of the position A in the map coordinate system._A,Y_A,Z_A) Then, the coordinate (X) of the position M of the real measuring point M' of the lamp in the map coordinate system is calculated by utilizing the triangle_M,Y_M,Z_M)。

It should be noted that, in calculating the pose change, a transformation matrix equation may also be constructed by using a plurality of matched image feature pairs, for example, at least four matched image feature pairs, in the two maps Pic1 and Pic2, so as to obtain the pose change.

It should be noted that, at preset time intervals, based on the known positions, a map of the physical space can be constructed in the virtual map coordinate system by using the matched image features in the two maps, which is different from the measured constructed map in that the unit distance D is independent of the actual moving distance of the mobile device.

Taking the map coordinate system as an example of an indoor space coordinate system constructed by the mobile device by using the measurement during the movement of the previous time, the mobile device determines the position coordinates of each pixel point in the currently captured image in the map coordinate system by matching the currently captured image with the image captured during the movement of the previous time and the position coordinates captured during the movement of the previous time.

It should be clear to those skilled in the art that the coordinates of the corresponding points in the world coordinate system and the map coordinate system can be converted and calculated according to parameters such as internal parameters and external parameters of the camera device. In order to construct the association relationship between the map coordinate system and the world coordinate system of the image pickup apparatus, before the image pickup apparatus starts to pick up an image, a step of calibrating the image pickup apparatus may be further included. And the calibration is to determine the position of a pixel point mapped to the image by shooting the geometric pattern of the standard point in the measurement space. Wherein, the calibration method includes but is not limited to: one or more of a conventional camera calibration method, an active vision camera calibration method, a camera self-calibration method, and the like.

The actual object is mapped as a set of pixel points in the image, each pixel point corresponding to a voxel point coordinate (x, y, z) in the map coordinate system. If a physical length of the position of the actual object corresponding to the voxel point in the indoor space is known or measurable, and a known proportional relationship can be established between the physical length and a coordinate value on any axis of the coordinates (x, y, z) of the voxel point (for example, the physical length is parallel to/consistent with the direction of the axis, or a component of the physical length on the axis can be calculated), the spatial scale parameter of the coordinate conversion between the map coordinate system and the world coordinate system of the indoor space can be deduced according to the proportional relationship because the proportion between x, y, z is unchanged. In some embodiments, the pixel points may be selected from image features to facilitate recognition.

The physical length can be obtained by means of ranging. Correspondingly, the mobile device further comprises a depth detection device. In some embodiments, the depth detection device includes, but is not limited to: one or more of an infrared distance measuring device, a laser distance measuring device, an ultrasonic sensing device and the like.

Specifically, the infrared distance measuring device continuously emits modulated infrared light, forms reflection after the infrared light irradiates an object, receives the reflected light, and calculates depth information by calculating the time difference or phase difference between the emission and the reception of the infrared light. In some embodiments, the depth detection device may be a single-point ToF (time of flight) sensor, and the number of ToF sensors may be one or more. For example, the ToF sensor is one in number and is disposed on one side of the imaging device. For another example, the number of ToF sensors is two, and the ToF sensors are symmetrically disposed on two opposite sides of the imaging device. By adopting the single-point ToF sensor, the product cost can be effectively reduced; of course, the depth detection device may also use other types of ToF sensors, and can also measure depth information and further calculate spatial scale parameters.

Specifically, the laser ranging device is provided with a laser signal transmitter and a laser signal receiver, a beam of laser is transmitted by the laser signal transmitter and reflected after irradiating an object, the reflected laser is received by the laser signal receiver, and depth information is calculated according to time difference data of laser transmission and reception. In some embodiments, the number of the laser ranging devices may be one or more, for example, the number of the laser ranging devices may be four, six or eight, and the laser ranging devices are respectively symmetrically arranged on two opposite sides of the image capturing device.

Specifically, the ultrasonic sensing device emits ultrasonic waves to a certain direction, and starts timing at the same time of emission time, the ultrasonic waves collide with an obstacle to form reflection during propagation in air, and the ultrasonic sensing device receives reflected waves and stops timing; and calculating to obtain depth information according to the time recorded by the timer.

The depth detection device provides depth information of an indoor space corresponding to at least one pixel unit in the image. The image block imaged by the pixel unit is obtained by reflecting light from an entity (or called an object) in the indoor space. The depth information of the corresponding pixel unit provided by the depth detection device is used for determining the distance from the mobile equipment to the corresponding entity. The distance is expressed in units of length. Wherein, according to the type of the selected depth detection device, the objects detectable by the depth detection device include opaque objects such as ceilings, houses, door frames, tables and the like; or the object detectable by the depth detection means comprises a transparent/translucent object such as glass.

The position of the pixel unit in the image is fixedly set in advance according to the angles of the camera device and the depth detection device. For example, the detection position of the depth detection device can be located in the central area of the field angle of the image capture device by calibration, and correspondingly, the pixel unit is located in the central area of the captured image. Alternatively, the pixel unit can be made to fall on other areas in the image by calibration.

The range size of the pixel unit may be determined according to a measurement range of the depth detection device. In some embodiments, the range of transmission and reception of the axis (e.g., near infrared light, laser light, etc.) of the depth detection device includes, but is not limited to: 0 to + -5 degrees with respect to the vertical line, etc. For example, the axis of the depth detection device is at an angle of 0 degrees with respect to the vertical. Or the included angle of the axis of the depth detection device relative to the vertical line is-5 degrees, -4 degrees, -3 degrees, -2 degrees, -1 degree, 2 degrees, 3 degrees, 4 degrees or 5 degrees. It should be noted that, the included angle between the axis of the depth detection device and the vertical line or the horizontal line is only an example, and is not limited to the range of the included angle precision of 1 degree, and the precision of the included angle may be higher, such as reaching 0.1 degree, more than 0.01 degree, and the like according to the design requirement of the actual mobile device. The pixel units can approximately occupy square pixel areas such as 3 × 3 pixels, 4 × 4 pixels, 5 × 5 pixels, 6 × 6 pixels, 7 × 7 pixels or 8 × 8 pixels according to the range radiated by the depth detection device; or the pixel unit can approximately occupy rectangular pixel areas such as 3 × 4 pixels, 4 × 5 pixels, 5 × 7 pixels and the like.

It should be noted that, in an actual scene, the emitted beams of the selected infrared distance measuring devices, ultrasonic sensing devices and the like can be concentrated as much as possible, so that the landing range is small; on one hand, the obtained depth information of the calibration point is more accurate; on the other hand, for example, with a single-point TOF sensor, the cost is much lower than that of other types of depth detection devices.

In addition, the pixel unit may be one or more, and when the pixel unit is multiple, it means that the depth detection apparatus can simultaneously detect the depth information between the mobile device and multiple physical locations of the indoor space. Determining the spatial dimension parameter using the plurality of depth information may improve positioning accuracy. In some embodiments, the relative positional relationship between the image pickup device and the depth detection device may be set in advance. For example, the depth detection device may be one, and is disposed on one side of the image pickup device, and the measured depth information corresponds to one pixel unit in the picked-up image of the image pickup device. For another example, the number of the depth detection devices may be two, and the two depth detection devices are respectively and symmetrically arranged on two sides of the camera device, and the two pieces of measured depth information correspond to two pixel units which are axisymmetric in the captured image of the camera housing. Of course, not limited thereto, the above-described relative positional relationship is merely an example, and does not limit the scope thereof. For example, the depth detection device may be provided in plurality, and the depth detection device may be symmetrically provided around the imaging device, and the four pieces of measured depth information may correspond to four pixel units in the captured image of the imaging device.

And in the moving process of the mobile equipment, the position of the pixel unit in the image is fixed. When the camera device detects that the pixel unit contains the image feature in the current image, the depth detection device acquires the depth information corresponding to the image feature at the current moment. In some embodiments, the image features include, but are not limited to: shape features, and grayscale features, etc. The shape features include, but are not limited to, one or more of corner features, edge features, line features, curve features, and the like. The grayscale features include, but are not limited to: the image processing method comprises the following steps of obtaining an image frame, wherein the image frame comprises a preset gray scale range, and the image frame comprises one or more of a gray scale jump characteristic, a gray scale value higher or lower than a gray scale threshold value, a region size comprising the preset gray scale range in the image frame and the like.

For example, in the case where the depth detection device is directed toward a ceiling in an indoor space, the obtained depth information may be distance information of the depth detection device from the ceiling. When the depth detection device is a single point ToF sensor pointing vertically towards the ceiling, the resulting depth information is the vertical height of the depth detection device from the ceiling. In some embodiments, the detection position is located on an object, such as a room divider or other indoor object in an indoor space, and the obtained depth information is the distance information between the depth detection device and the surface of the object.

When the image feature is extracted in the pixel unit, the depth detection device can acquire the depth information of the indoor object corresponding to the image feature.

The depth detection device determines a spatial scale parameter according to the detected depth information of the image feature and the position information of the image feature corresponding to the map coordinate system. The space scale parameter is used for determining the proportional relation between the height information of the image features corresponding to the indoor space and the depth information of the image features corresponding to the map coordinate system.

For example, assume that the mobile device is a sweeping robot. Referring to fig. 1A, a schematic diagram of the determining depth information of the sweeping robot in a scene according to the present application is shown. As shown in the figure, it is assumed that the corner points on a square object (e.g., a dome lamp) in the indoor space are imaged in pixel units of an image, and an image feature p0 is detected, which has coordinates p0(x0, y0, z0) in a map coordinate system.

Taking a mobile device as an example of a sweeping robot, and taking an installed depth detection device as an example of pointing to a ceiling vertically, the x, y, and z axes of the map coordinate system may be parallel to the X, Y, Z axis of the world coordinate system, respectively, and therefore, the current spatial scale parameter may be obtained by obtaining a ratio of the height information z0 and the depth information H0 of the image feature p0 corresponding to the detection position of the depth detection device. From this spatial scale parameter, the image feature p0 can be mapped into the world coordinate system.

Of course, the mobile device may also be a service robot or the like with a certain height. Please refer to fig. 1B, which is a schematic diagram illustrating a service robot determining depth information in a scenario according to the present application. As shown, a corner point on the dome lamp corresponds to an image feature p1 on the image, and the coordinates of the image feature in the map coordinate system are p1(x1, y1, z 1).

In this case, the depth information detected by the service robot 11 is the distance H1 from the detection position to the depth detection device on the top of the service robot. Thus, the depth information may be the sum of the distance H1 and the service robot's own height H2. Similarly, according to the proportional relationship between the depth information (i.e. H1+ H2) and the height information z1 of the image feature p1, the current spatial scale parameter can be obtained.

As another example, the mobile device may be a smart terminal (e.g., AR glasses). Please refer to fig. 1C, which is a schematic diagram illustrating a scenario of determining depth information by an intelligent terminal according to the present application. As shown, it is assumed that the depth detection device points horizontally to a room partition (e.g., a wall surface), a corner point on a wall painting on the wall corresponds to an image feature P2 on the image, and the coordinate of the corresponding position of the image feature under the world coordinate system is P2(X2, Y2, Z2). In this case, the depth information detected by the smart terminal 12 is the distance H3 from the detected position to the depth detection device on the smart terminal 12.

As mentioned earlier, the map coordinate system (e.g., x, y, z axes shown by dashed arrows in fig. 1A, 1B, and 1C) may be transformed with the world coordinate system (e.g., X, Y, Z axes shown by solid arrows in fig. 1A, 1B, and 1C) using spatial scale parameters. Therefore, the coordinate position in the map coordinate system corresponding to the image feature is mapped to the coordinate position in the world coordinate system, that is, the coordinate of the image feature in the world coordinate system is obtained.

If a deflection angle exists between the axis of the depth detection device and the height direction of the indoor space, the height information corresponding to each point on the object can be obtained through mathematical calculation according to the detected depth information and the deflection angle; furthermore, the spatial scale parameter can be calculated by one or more of known depth detection devices, and position data predetermined by the camera device (for example, an angle between the axes of the two devices and a height direction of the indoor space, and the like), and is not derived here.

In some embodiments, the angular relationship between the optical axis of the camera and the axis of the depth detection device may also be pre-set, e.g., parallel between the optical axis of the camera and the axis of the depth detection device. For another example, an angle between the depth detection device and an optical axis of the image pickup device is in a range from 0 degree to a maximum angle of view of the image pickup device. For example, when the maximum viewing angle of the image capturing device is 120 degrees, the included angle between the depth detection device and the optical axis of the image capturing device may be 0 degrees, 1 degree, 2 degrees … … 58 degrees, 59 degrees, 60 degrees … … 118 degrees, 119 degrees, 120 degrees. It should be understood that the included angles are exemplary only and do not limit the scope thereof. For another example, the included angle between the depth detection device and the optical axis of the image pickup device is in a range from 0 degrees to the minimum value of the field angle of the image pickup device. For example, when the minimum field angle of the image capturing device is 30 degrees, the included angle between the depth detection device and the optical axis of the image capturing device may be 0 degrees, 1 degree, 2 degrees, … … 13 degrees, 14 degrees, 15 degrees … … 28 degrees, 29 degrees, and 30 degrees.

Of course, not limited thereto, and the above-described angular relationships are merely examples, and do not limit the scope thereof. For example, the angular relationship between the camera device and the depth detection device may be that the optical axis of the camera device is parallel to the vertical line, and the axis of the depth detection device forms a certain angle with the vertical line, which is not exhaustive.

Please refer to fig. 2, which is a flowchart illustrating an indoor positioning method of a mobile device according to an embodiment of the present disclosure. As shown in the figure, the indoor positioning method includes:

s201, acquiring a first image and a second image which are respectively shot by the camera device at different positions; and the second image features matched with the first image features in the first image are arranged in the pixel units of the second image, and image feature pairs are formed between the first image features and the second image features.

As mentioned above, the image captured by the image capturing device may be one or more of a single frame image, a continuous image frame sequence, a discontinuous image frame sequence, or a video. For example, the first image and the second image may be two frame images photographed separately. Alternatively, the first image and the second image may be two-frame images or the like arbitrarily extracted from a captured video or image sequence.

It should be noted that the "first" and "second" do not mean the order of image capturing, and are only used for distinguishing two images; the previous image represents the earlier of the first image and the second image at the moment of capture.

The mobile device may directly acquire a captured image as the first image or the second image from the imaging apparatus, or may acquire a history captured image as the first image or the second image from a storage medium. In some embodiments, the camera in the mobile device buffers the captured images of the room in a preset format in a storage medium and is acquired by a control system of the mobile device.

In some embodiments, a previous image of the first and second images is captured by a camera during movement of the mobile device.

For example, the mobile device captures a first image C1 at time t1 and a second image C2 at time t2 during the movement at position a. Alternatively, the mobile device takes the second image C2 at time t1 at position A, the first image C1 at time t2 at position B, and so on. As another example, the mobile device captures a first image C1 during movement, and extracts a previously captured image from a map database stored in a storage medium as a second image C2; or the mobile device takes a second image C2 during movement, extracts a previously taken image from a map database stored in a storage medium as a first image C1, and so on.

In some embodiments, the prior image is from a map database of the indoor space and the subsequent image is obtained by a camera of the mobile device.

The map database includes landmark information. The landmark information refers to a feature easily distinguished from other objects in the environment, and for example, the landmark information may be a table corner, an outline feature of a dome lamp on a ceiling, a line between a wall and the ground, and the like. Wherein the landmark information includes, but is not limited to: the mobile device comprises a first image, a second image, a map data storage unit, a display unit and a display unit, wherein the first image and the second image are one or more of a previous image in the first image and the second image, map data of a physical space where the mobile device is located when an image feature is shot, the image feature matched with the previous image, the position of the image feature in a corresponding image when the image feature is shot, the position and the posture of the mobile device when the image feature is shot, and the like.

In some embodiments, the landmark information also has coordinate information. For example, the landmark information has a position of the mobile device when the landmark is ingested, i.e., coordinates of the mobile device in the world coordinate system. For another example, when the landmark information is an image feature, the landmark information has a position of the image feature in a corresponding image.

In some embodiments, the map database may be stored in a storage medium together with an image of an indoor environment acquired by the mobile device through the camera, or uploaded to a server or a cloud for storage.

When the pixel unit of the second image has a second image feature which is matched with the first image feature in the first image, an image feature pair is formed between the first image feature and the second image feature.

The first image feature matches the second image feature, i.e. the first image feature is similar to the second image feature. In some embodiments, by calculating a distance (e.g., euclidean distance, cosine distance, etc.) between the feature vector of the first image feature and the feature vector of the second image feature, a closer distance indicates a better match between the two image features. Here, the first image feature in the first image may be matched with the second image feature in the second image by an image matching algorithm, which includes but is not limited to: SIFT (Scale-innovative Feature Transform), ORB (organized FAST and Rotated BRIEF), FLANN (FAST Library for Approximate neuron neighbors), and the like.

To enable accurate localization, the number of matched image features may be multiple. For this purpose, image features which can be matched are searched from the identified image features according to the positions of the pixel units corresponding to the identified image features in the image, and the matched image features in different images may belong to the same object or part.

Please refer to fig. 3, which is a diagram illustrating matching of image features according to the present application. As shown, after identifying image features in the images, it is determined that image features a1 and a2 are included in the first image C1, image features b1, b2 and b3 are included in the second image C2, and image features a1 and b1 belong to a matching image feature pair and image features a2 and b3 belong to a matching image feature pair.

Thus, it may be determined that the image feature a1 in the first image C1 is located to the left of the image feature a2 and is spaced by d1 pixels; it was also determined that image feature b1 in second image C2 was to the left of image feature b3 and was spaced d1 'apart, and that image feature b2 was to the right of image feature b3 and was spaced d 2' apart. Matching the pixel pitches of the image features a1 and a2 according to the position relations of the image features b1 and b3 and the position relations of the image features b2 and b3 and the position relations of the image features a1 and a2 respectively, and the pixel pitches of the image features b1 and b3 and the pixel pitches of the image features b2 and b3 respectively, so as to obtain that the image feature a1 in the first image C1 is matched with the image feature b1 in the second image C2, and the image feature a2 is matched with the image feature b 3.

And so on to determine pairs of image features that match the first image to the second image. And determining each matched image feature pair so as to position and pose the mobile equipment according to the vector of the pixel position offset of the image corresponding to each image feature. Wherein, the position of the mobile device can be obtained according to the displacement change in the image pixel coordinate system, and the posture can be obtained according to the angle change in the image pixel coordinate system.

In some embodiments, the pixel position offset amount of the image features in the two images or the physical position offset amount of the image features in the physical space can be determined according to the corresponding relation, and the relative position and the posture of the mobile device from the time t2 of taking the second image C2 to the time t1 of taking the first image C1 can be calculated by integrating any obtained position offset information. For example, through coordinate transformation, the position and posture of the mobile device from the time t1 when the first image C1 is captured to the time t2 when the second image C2 is captured are obtained as follows: moved by d length on the ground and rotated by θ degrees to the left, or equivalently, the mobile device moved dsin θ and dcos θ, respectively, in the direction of the X, Y coordinate axis on the ground.

S202, determining position change information of the mobile equipment between different positions according to the current spatial scale parameter and the pixel position offset of the matched image features in the first image and the second image.

The current spatial scale parameter is a proportional relation determined according to the depth information of the image features in the corresponding pixel units detected by the depth detection device at the current moment and the depth information of the corresponding image features in the map coordinate system of the camera device.

In some embodiments, the current spatial scale parameter is set according to depth information corresponding to the pixel unit in the first image or the second image. The depth detection device may obtain depth information immediately when positioning is needed, for example, the depth information may be depth information of a feature corresponding to a pixel unit in any one of a first image or a second image obtained by the mobile device. Alternatively, the depth information of the feature corresponding to the pixel unit in the first image and the depth information of the image feature corresponding to the pixel unit in the second image may be acquired separately, and the average of the two pieces of depth information may be used as the final depth information.

Of course, the present spatial scale parameter is not limited to this, and in some embodiments, the present spatial scale parameter may also be obtained according to historical depth information of the depth detection apparatus. For example, the depth information may be randomly chosen by the mobile device among a plurality of historical depth information previously acquired and stored. Or, the depth information may be a depth information whose date is closest to the mobile device in a chronological order, from a plurality of historical depth information acquired and stored by the mobile device, as a final depth information. Alternatively, the depth information may be a mean or a weighted sum of a plurality of historical depth information previously acquired and stored by the mobile device, or the like.

And acquiring position change information between the position of the mobile equipment when the first image is shot and the position of the mobile equipment when the second image is shot according to the acquired current space scale parameter and the pixel position offset of one or more matched image feature pairs in the first image and the second image. The pixel position offset amount refers to an offset distance and angle of the second image feature with respect to the first image feature, or an offset distance and angle of the first image feature with respect to the second image feature.

The position change information reflects the relative position and posture change of the mobile device from the previous position to the current position. In some embodiments, the location change information includes a displacement distance, a movement direction, and the like of the mobile device.

Referring again to fig. 1A, as shown, a corner point on the dome lamp corresponds to an image feature p0, which has coordinates p0(x0, y0, z0) in the map coordinate system. The mobile robot 10 detects the depth information of the image feature by a depth detection device (e.g., a ToF sensor, not shown) as H0.

If the imaging device of the mobile robot 10 is set such that the optical axis thereof is set in the height direction of the indoor space, the depth information H0 is used as the height information H of the current space between the mobile robot 10 and the object corresponding to the image feature located in the pixel unit in the currently captured image. Therefore, the current spatial scale parameter is the ratio of the height direction (assuming that the Z axis is consistent) Z0 of the image feature p0 in the map coordinate system to the depth information H0.

In some embodiments, the mobile robot 10 may be as shown in the figure, when the mobile robot is installed on the ground, its upper surface is a plane parallel to the ceiling surface, and the camera device is installed vertically upward on the upper surface of the mobile robot 10, so that its axis can also be consistent with the height direction of the indoor space; of course, this is merely an example, and in other embodiments, the upper surface of the mobile device may not be a plane, which is not limited to this embodiment.

In the present embodiment, the mobile device captures a first image by the camera at a previous position w 1; after moving d meters in the direction indicated by the dashed arrow, a second image is taken at the current position w 2.

Please refer to fig. 4, which is a diagram illustrating image features of the captured image in the image pixel coordinate system in fig. 1A. As shown, under the same image pixel coordinate system (u, v), the coordinates of the first image feature in the first image captured by the mobile device are P1(u1, v1), and the coordinates of the second image feature in the second image captured are P2(u2, v 2). Thus, the pixel position shift amount of the matched first image feature P1 and second image feature P2 can be calculated

Vector of the pixel position offset

Indicating that the first image feature P1 has moved m pixels in the direction of the second image feature P2 (indicated by the dashed arrow) as a result of movement of the mobile device. In other words, the position of the same image feature in the first image and the change in position in the second image reflect the change in pose during movement of the mobile device.

After the pixel position offsets between the second image feature P2 and the first image feature P1, and P2 and P1 respectively corresponding to the local Q on the object are obtained, the position change information of the mobile device relative to the first image when the second image is captured can be calculated and obtained according to the current spatial scale parameters.

Taking the depth information in the current spatial dimension parameter as the depth information of the second image feature P2 and the corresponding object part Q relative to the mobile device as an example, in some examples, the mobile device obtains the position relationship between itself and the object part Q, and determines the position relationship between the current position of the mobile device relative to the shooting position when the first image is shot by using the matched pixel position relationship of the first image feature P1 and the second image feature P2, wherein the position relationship and the displacement relationship are included. In still other examples, the mobile device infers a plurality of image feature pairs matched in the first image and the second image with the current spatial scale parameters, and utilizes each image feature pair to determine a positional relationship between the current position of the mobile device relative to a previous captured position when the first image was captured, to improve positioning accuracy. For example, positional relationship calculation is performed separately using each pair of image features, and the positional relationship between the current position and the previous shooting position is determined by a weighted sum, that is, positional change information is obtained.

It should be noted that the above-mentioned determination method of the current spatial scale parameter is only an example, and may also be a weighted value obtained by the mobile device according to multiple measurements during the movement. For example, during the movement, the mobile device acquires depth information corresponding to the time of consecutive shots and performs a weighted average to obtain depth information of image features in pixel units of an image during the consecutive shots and thereby determine the current spatial scale parameter.

Taking the mobile device as an example, the mobile device is an intelligent terminal with Augmented Reality (AR) or Mixed Reality (MR) functions, and the intelligent terminal may be a mobile phone, a head-mounted device, intelligent glasses, and the like. When the camera device provides the first image and the second image during movement and the pixel unit in the second image contains the image feature matched with the first image during movement of the intelligent terminal, the intelligent terminal can align the coordinates of the key part of the virtual object displayed on the screen with the coordinates in the physical space according to the depth information of the image feature in the pixel unit corresponding to the second image provided by the depth detection device, so as to determine the plane reference of the virtual object in the real scene and the corresponding relation of the coordinates of the key part of the virtual object and the coordinates in the physical space. When the player operates the intelligent terminal, the interaction with the virtual object in the actual space can be realized.

In some embodiments, when the previous image is from a map database pre-stored by the mobile device, the mobile device is further capable of performing the following steps of the indoor positioning method using the determined location change information of the mobile device: and determining the position information of the mobile equipment in the indoor space according to the position change information and a preset map database of the indoor space. Therefore, the current position information of the mobile equipment is determined by acquiring the position change information of the mobile equipment, so that the real-time indoor positioning of the mobile equipment is realized.

For example, taking the mobile device as a mobile robot as an example, the mobile robot captures a first image C1 at a position a at time t1, and matches landmark information in the first image C1 with landmark information stored in the map database, thereby determining a position of the mobile robot in the map corresponding to the position a in the real world. Next, the mobile robot captures a second image C2 at the B position at time t2, and determines, by the indoor positioning method, that the position change with respect to the a position when the mobile robot is at the B position is: moved a distance D in the southeast direction. And matching the landmark information in the second image C2 with the landmark information stored in the map database so as to obtain the position of the mobile robot in the map corresponding to the B position of the real world. Thus, it is possible to obtain the position information of the mobile robot in the real world, the position change information of the mobile robot in the map, and the map positions at the time t1 and the time t2, respectively.

In other embodiments, the mobile device may further construct a correspondence between an image pixel coordinate system and a world coordinate system according to the location change information, so as to construct a map of the indoor space. Wherein, the constructed map includes but is not limited to: grid maps, or topological maps, etc. For example, a robot scanner constructs a feature map or the like from captured images of landmarks and coordinate information thereof during a cleaning movement.

It is easily understood that the feature included in the landmark information is usually fixed, but in actual applications, the feature included in the landmark information is not necessarily the same. For example, the feature included as landmark information is an outline feature of the lamp, and when the lamp is replaced, the corresponding feature disappears. When the mobile device needs to be positioned by means of the feature, the feature cannot be found.

To this end, in some embodiments, the indoor positioning method further comprises: updating a map database of the indoor space based on the determined location change information and one or more of the first image, the second image, and the depth information. For example, when image features that have not been stored by the mobile device are found based on similar or identical location change information (e.g., location and orientation), the latest image feature correspondence of the first image and/or the second image may be saved in the map database. As another example, when an image feature that has been stored but cannot be matched with a newly matched image feature is found based on a similar or identical position and orientation, the redundant features stored in the corresponding map database are deleted.

In some embodiments, new landmark information may also be added based on the number of image features of the currently matched first image and second image being above a preset threshold. Where the threshold may be a fixed value or set based on the corresponding number of features at the marked location in the map. For example, when the number of newly matched features is found to be more than the number of features stored in the storage medium at the corresponding position based on similar or identical positions and attitudes, the new features may be added to the constructed landmark information.

In some embodiments, the latest depth information may also be stored in the map database based on the currently acquired depth information. Or when the depth information is acquired but the depth information corresponding to the position cannot be found in the existing map database, the depth information is supplemented into the map database. It should be noted that, as will be understood by those skilled in the art, the above-mentioned manner of updating the map database based on the location change information is only an example, and is not a limitation to the present application. For example, a map database or the like may be updated based on the image features.

In some embodiments, the mobile device is capable of correcting or updating the constructed map of the indoor space based on the location change information of the mobile device determined by the positioning method in combination with inertial navigation data provided by an inertial navigation detection device. For example, the mobile device obtains position change information of itself as D1 through the inertial navigation detection device, and obtains position change information after processing according to data obtained by the image pickup device and the depth detection device as D2. Therefore, whether the position change information is in an error range or whether the position change information is blocked or whether the walking wheels slip or the like can be judged according to the difference of the position change information obtained in the two modes.

Thus, in some embodiments, the mobile device further comprises an inertial navigation detection device for acquiring inertial navigation data of the mobile device. The inertial navigation detection device includes but is not limited to one or more of a gyroscope, a odometer, a light flow meter, an accelerometer, and the like. The inertial navigation data includes, but is not limited to, one or more of motion velocity data, acceleration data, and movement distance of the mobile device. For example, when the mobile device is a mobile robot, the inertial navigation data may be one or more of speed data, acceleration data, the number of turns of a wheel, a moving distance, a deflection angle of the wheel, and the like of the mobile robot. For another example, when the mobile device is an electronic device such as a mobile phone and a head-mounted device, the inertial navigation data may be one or more of displacement data, a displacement direction, acceleration data, velocity data, and the like of the electronic device in a three-dimensional space.

Therefore, the indoor positioning method can further comprise the step of correcting the inertial navigation data provided by the inertial navigation detection device by using the determined position change information. For example, the current position of the mobile device is obtained according to the position change information of the mobile device obtained by the indoor positioning method, and is compared and corrected with the position information provided by the inertial navigation detection device. In some embodiments, the inertial navigation data provided by the inertial navigation detection device may be replaced by the determined position change information. Alternatively, the position change information and the inertial navigation data provided by the inertial navigation detection device may be integrated, and the final inertial navigation data may be obtained by calculating a weighted sum or other mathematical operations, for example.

The positioning mode of the embodiment of the application can be used for positioning with sufficient matched features. For example, during navigation of the movement of the mobile device, acquiring the relative position and posture change in the above manner can quickly determine whether the current movement route of the mobile device is deviated, and perform subsequent navigation adjustment based on the determination result.

Accordingly, in some embodiments, the indoor positioning method further comprises: a step of updating a positional relationship between the mobile device and a preset navigation route based on the determined position change information. Taking the mobile device as a sweeping robot as an example, when the sweeping robot establishes a map, the robot can be helped to determine whether the map is on a preset navigation route according to the obtained position and posture. When the sweeping robot is not positioned on the preset navigation route, the sweeping robot can be helped to determine the relative displacement and the relative rotation angle between the current position and the previous position according to the obtained position and posture, and the data is used for carrying out mapping or planning the navigation route again based on the current position and the like.

Please refer to fig. 5, which is a schematic view illustrating a scene of an embodiment of an indoor positioning method of a mobile device according to the present application. Taking the mobile device as a floor sweeping robot as an example, the floor sweeping robot performs a cleaning task indoors. Assuming that the direction in which the sweeping robot moves forward (the direction indicated by the dotted arrow a1 in the figure) is the forward direction, the camera device (for example, a camera) is arranged at the top front end of the sweeping robot. The right side of the camera is provided with a depth detection device, such as a single-point ToF sensor. For convenience of description, the optical axis direction of the imaging device is the same as the vertical direction, and the axis of the single-point ToF sensor is directed vertically to the ceiling (the direction indicated by the dashed arrow B in the figure); the detected depth information is then the vertical height of the single point ToF sensor to the roof room in this case.

As shown in the figure, a world coordinate system (X, Y, Z) is formed by assuming that the extending direction of the wall in the front of the sweeping robot is an X axis, the extending direction of the wall on the right side of the sweeping robot is a Y axis, and the vertical upward direction is a Z axis. The sweeping robot can successively shoot images of at least two ceilings through the camera in a moving or static state, and it is assumed that at least two images all contain indoor objects T. Meanwhile, the sweeping robot acquires the depth information of the object T in the indoor space through the single-point ToF sensor. According to the matched image features T1 and T2 in the two images corresponding to the object T, the position change information of the sweeping robot at the current position relative to the previous position can be obtained by calculating the pixel position offset of the matched image features T1 and T2 and according to the current space scale parameters obtained by the depth information and the height of the image features under the map coordinate system. And combining the position information of the last position of the sweeping robot with the position change information, and comparing and calculating the position information with the map of the indoor space stored in the map database, so as to obtain the current position of the sweeping robot in the map. Further, whether the sweeping robot deviates from a preset navigation route or not can be judged according to the previous position and the current position of the sweeping robot.

Also taking the sweeping robot as an example, for example, the sweeping robot may further combine the constructed indoor map and the position change information obtained according to the positioning method to pre-determine the distance from the current position to the obstacle marked on the indoor map, and facilitate timely adjustment of the sweeping strategy. Wherein the barrier may be described by a single marking or marked as a wall, table, sofa, wardrobe, etc. based on characteristics of shape, size, etc.

In some embodiments, the indoor positioning method further comprises: and during the continuous movement of the mobile equipment, identifying that the determined position change information corresponds to the entity object with the suspension space according to the depth information sequence provided by the depth detection device at different positions. The suspension space refers to a space formed by a horizontal plane at least partially suspended above the ground. The solid object comprises one or more of a door, a bed body, a tea table, a table and the like. For example, for a door body, since the height of the upper door frame is generally lower than the height of the ceiling, and the door body generally has a certain thickness, the space below the upper door frame to the ground forms the suspension space. As another example, for a table, the table top is suspended from the legs, whereby the space below the table top to the floor forms the suspended space.

Referring to fig. 5 again, as shown in the figure, when the sweeping robot crosses below the door frame in the direction indicated by the dotted arrow a2 in the figure (the door body is not shown), the sweeping robot performs detection (for example, continuously performs detection, or periodically performs detection at certain intervals, which is not limited herein) by using the single-point ToF sensor, and obtains depth information of each position of the ceiling, so as to form at least one depth information sequence. It is easy to understand that if the detection position of the single-point ToF sensor of the sweeping robot is always an area without a ceiling (a plane area of the ceiling), each depth information in the depth information sequence should be the same, or the difference between each depth information is within an error range. For example, when the detection position of the single-point ToF sensor of the sweeping robot is always an area without a ceiling (a plane area of the ceiling), the at least one depth information sequence detected by the single-point ToF sensor may be {3.02m,2.99m,3.01m,2.98m,3.00m }, and so on.

And when the robot of sweeping the floor was located the door frame below, promptly, when the robot of sweeping the floor was located the unsettled space under the door frame, what the single-point ToF sensor detected is the degree of depth information of door frame lower surface. Obviously, the difference between the depth information and other depth information in the depth information sequence is larger than an error range, so that the situation that the upper part of the sweeping robot corresponds to a door frame with a suspension space can be judged.

Also taking the mobile device as a sweeping robot for example, if the axis of the single-point ToF sensor is directed vertically to the ceiling, in this case, the obtained depth information is the vertical height of the depth detection device to the ceiling. If the vertical height of the ceiling is 3m and the vertical height of the door body is 2m, the obtained vertical height of the lower surface of the door frame can be 2m if the thickness of the sweeping robot is ignored. When the detected depth information may be maintained in a range of about 3.00m ± 0.02m (the error range of 0.02m is merely an example and is not a limitation in the embodiment) before the sweeping robot passes under the door frame in the direction indicated by the dotted arrow a2, the obtained depth information sequence may be {3.01m,2.99m, … …,2.98m,3.00m }. When the sweeping robot is located right below the door frame, the depth information detected by the single-point ToF sensor is height information of the lower surface of the door frame from the ground, for example, the depth information may be 2.00m at this time. Obviously, the depth information has a larger difference compared with other depth information in the depth information sequence, and the sweeping robot can recognize that the ceiling is not above the current position. And according to the depth information obtained by the detection at the moment, the entity object with the suspension space can be further identified as the door frame. For example, depth information templates of a plurality of physical objects are stored in the map database in advance, and the detected depth information is compared or matched with the templates, so as to determine whether the physical object corresponding to the depth information is a certain physical object in the templates.

Of course, the mobile device may also be a service robot or the like with a certain height. In this case, the depth information detected by the service robot may be a distance of the detection position from a depth detection device on the top of the service robot, or the like. Thus, the depth information may be depth information obtained by subtracting the height of the service robot itself from the actual vertical height of the physical object. For example, assuming that the vertical height of the ceiling is 3m and the height of the service robot itself is 1m, the depth information sequence acquired may be {2.01m,1.99m, … …,1.98m,2.00m }. Of course, not limited thereto, but merely as exemplifications of one possible embodiment thereof.

In some examples where the mobile device is a sweeping robot, the sweeping robot may select a sweeping strategy that does not perform sweeping based on the identified low physical objects, such as beds, end tables, and the like. For example, when the sweeping robot measures the position relation between the sweeping robot and the bed according to the depth information sequence, the sweeping robot selects to turn around to sweep other areas.

When the mobile equipment moves into or out of other suspended spaces, the corresponding entity object can be identified according to the change of the depth information in the depth information sequence. For example, when the mobile device moves to a suspended space under a table or leaves the suspended space under a tea table, the depth information may also change accordingly, so as to identify that the entity above is the table, the tea table, and the like, which is not described herein again.

According to the indoor positioning method of the mobile equipment, at least two images are shot through the camera device, image features in the two images are matched, and position change information of the mobile equipment in the indoor space is obtained according to pixel position offset of the matched image features and space scale parameters obtained according to depth information of features of the indoor space corresponding to the image features, which is obtained through the depth detection device, so that the indoor positioning of the mobile equipment is achieved. The indoor positioning method is used for positioning the mobile equipment, so that the influence of ground conditions on a positioning result is avoided, and the positioning accuracy is improved. Meanwhile, the indoor positioning method provided by the application can realize multiple functions of map construction, navigation, positioning and the like through the camera device, and cost is saved.

Fig. 6 is a schematic structural diagram of a mobile device according to an embodiment of the present application. As shown, the mobile device 6 includes a camera 601, a depth detection device 602, and a processing device 603. It should be understood that the devices may be disposed on a circuit board of the mobile device, and the devices are directly or indirectly electrically connected to each other to realize data transmission or interaction.

It should be noted that, in consideration of the detection range of the depth detection device, the mobile device is generally displaced in an indoor space. The indoor space is a physical space where a boundary exists, such as an indoor environment of a home, a public place (e.g., an office, a mall, a hospital, a parking lot, and a bank), and the like.

The image capturing apparatus 601 is used to capture an image. In some embodiments, the imaging device includes, but is not limited to: cameras, video cameras, camera modules integrated with optical systems or CCD chips, and camera modules integrated with optical systems and CMOS chips, and the like. According to the requirement of actual imaging, the lens that can be adopted by the camera or the video camera includes but is not limited to: standard lenses, telephoto lenses, fisheye lenses, wide-angle lenses, and the like. For convenience of description, the embodiment of the present application is described by taking the image capturing apparatus as an example. Those skilled in the art will appreciate that the examples do not limit the scope of the specific embodiments.

The image pickup device can be used for picking up one or more of a single image, a continuous image sequence, a discontinuous image sequence, a video and the like. In some embodiments, the mobile device caches the captured image in a local storage medium or transmits the captured image to an external device connected with a communication link for storage, wherein the communication link comprises a wired or wireless communication link. The storage medium may include read-only memory, random-access memory, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, a usb disk, a removable hard disk, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed.

In some embodiments, the field angle of the imaging device is determined by parameters of the imaging device itself. The parameters of the image pickup device include internal parameters, external parameters, and distortion parameters, wherein the internal parameters include but are not limited to: one or more of a focal length, a physical size corresponding to each pixel, a pixel center, etc., including but not limited to one or more of a position of the camera on the mobile device, a rotational orientation, a translation matrix, etc.

In some embodiments, the range of field angles of the camera device includes, but is not limited to: 10 to 120 degrees. For example, the angle of view is 10 degrees, 20 degrees, 30 degrees, 40 degrees, 50 degrees, 60 degrees, 70 degrees, 80 degrees, 90 degrees, 100 degrees, 110 degrees, 120 degrees. For another example, the included angles of the optical axis of the imaging device with respect to the ground plane are 60 degrees, 61 degrees, 62 degrees … 89 degrees, 90 degrees, 91 degrees … 119 degrees, and 120 degrees. The above range is only an example, and the precision of the viewing angle is not limited to be within a range of 10 degrees, and the precision of the viewing angle may be higher according to actual design requirements, such as 1 degree, 0.1 degree, 0.01 degree or more.

According to the parameters of the camera device, the world coordinate system is subjected to similarity transformation to form a map coordinate system. Here, the unit of the map coordinate system may be a custom voxel unit, which may or may not be related to a length unit. The length unit is, for example, meter, decimeter, centimeter, millimeter, etc. Here, the map coordinate system may be a coordinate system in a virtual space or a map coordinate system constructed by the mobile device for the physical space.

Taking the map coordinate system as an example of the coordinate system in the virtual space, each pixel point in the image shot by the camera device is mapped to the virtual three-dimensional space of the map coordinate system.

In some examples, the map coordinate system overlaps with a camera coordinate system at an initial position of the mobile device, and a map of a virtual space corresponding to a physical space in which the mobile device is located is constructed by analyzing a feature in an image photographed at a preset interval and a preset voxel unit. For example, please refer to FIG. 9, which is shown in map coordinatesA map coordinate diagram describing the position relationship of the mobile device relative to the measurement point M on the lamp at the position O and the position a respectively, wherein the coordinate system XYZ is a map coordinate system, O is the origin of the map coordinate system, during the movement in the physical space, the mobile device takes two images containing the same real measurement point M '(not shown) of the lamp in the physical space at a preset time interval, wherein the distance moved by the mobile device at the preset time interval is corresponding to the unit distance D in the map coordinate system, and the posture change between the position O and the position a of the mobile device in the map coordinate system is determined by using the matched image features feature1 and feature2 respectively corresponding to the measurement point M' in the two images Pic1 and Pic2, thereby determining the coordinate (X) of the position a in the map coordinate system (X is the position b) of the mobile device in the map coordinate system_A,Y_A,Z_A) Then, the coordinate (X) of the position M of the real measuring point M' of the lamp in the map coordinate system is calculated by utilizing the triangle_M,Y_M,Z_M)。

The infrared distance measuring device continuously emits modulated infrared light, forms reflection after the infrared light irradiates an object, receives the reflected light, and calculates depth information by calculating the time difference or phase difference between the emission and the reception of the infrared light. In some embodiments, the depth detection device may be a single-point ToF (time of flight) sensor, and the number of ToF sensors may be one or more. For example, the ToF sensor is one in number and is disposed on one side of the imaging device. For another example, the number of ToF sensors is two, and the ToF sensors are symmetrically disposed on two opposite sides of the imaging device. By adopting the single-point ToF sensor, the product cost can be effectively reduced; of course, the depth detection device may also use other types of ToF sensors, which can also measure depth information and further calculate spatial scale parameters, but may increase the cost.

The laser ranging device is provided with a laser signal transmitter and a laser signal receiver, a beam of laser is transmitted by the laser signal transmitter and forms reflection after irradiating an object, the reflected laser is received by the laser signal receiver, and depth information is calculated according to time difference data of the transmission and the reception of the laser. In some embodiments, the number of the laser ranging devices may be one or more, for example, the number of the laser ranging devices may be four, six or eight, and the laser ranging devices are respectively symmetrically arranged on two opposite sides of the image capturing device.

The ultrasonic sensing device transmits ultrasonic waves to a certain direction, timing is started at the same time of transmitting time, the ultrasonic waves collide with an obstacle to form reflection in the air in the process of propagation, and the ultrasonic sensing device receives reflected waves and stops timing; and calculating to obtain depth information according to the time recorded by the timer.

The depth detection device 602 of the mobile device is configured to obtain depth information of a position of at least one detection position in the indoor space, where the depth information represents a distance between the detection position and the depth detection device. The detection position corresponds to a pixel unit in the image, and the pixel unit comprises one or more corresponding pixel points of the detection position mapped in the image.

The position of the pixel unit in the image is fixedly set in advance according to the angles of the camera device and the depth detection device. For example, the detection position of the depth detection device can be located in the central area of the field angle of the image capture device by calibration, and correspondingly, the pixel unit is located in the central area of the captured image. Alternatively, the pixel unit can be located at another known position in the image by calibration.

In a possible implementation manner, the detection position of the depth detection device may fall in the field of view of the image pickup device, so as to directly obtain the depth information of the pixel unit corresponding to the detection position in the image; alternatively, the detected position may not be in the image, but the physical structure relationship between the detected position and the corresponding shooting area in the image in the indoor space is known in advance, and the depth information of one or more pixel units in the image may be calculated accordingly.

For example, the image is taken corresponding to an area a of the ceiling in the indoor space, and the detection position a of the depth detection device may be located in the area a, so as to obtain the depth information of the pixel unit corresponding to the position a in the image; or, the depth detection device may detect that the position B may be located in a B region of the ceiling outside the a region, and then it may be inferred that the depth information of the pixel unit corresponding to the position a' located on the same plane as the position B in the a region is the depth information of B.

The range size of the pixel unit is determined according to the measuring range and the measuring angle of the depth detection device. In some embodiments, the range of transmission and reception of the axis (e.g., near infrared light, laser light, etc.) of the depth detection device includes, but is not limited to: 0 to + -5 degrees with respect to the vertical line, etc. For example, the axis of the depth detection device is at an angle of 0 degrees with respect to the vertical. Or the included angle of the axis of the depth detection device relative to the vertical line is-5 degrees, -4 degrees, -3 degrees, -2 degrees, -1 degree, 2 degrees, 3 degrees, 4 degrees or 5 degrees. It should be noted that, the included angle between the axis of the depth detection device and the vertical line or the horizontal line is only an example, and is not limited to the range of the included angle precision of 1 degree, and the precision of the included angle may be higher, such as reaching 0.1 degree, more than 0.01 degree, and the like according to the design requirement of the actual mobile device. The pixel units can approximately occupy square pixel areas such as 3 × 3 pixels, 4 × 4 pixels, 5 × 5 pixels, 6 × 6 pixels, 7 × 7 pixels or 8 × 8 pixels according to the range radiated by the depth detection device; or the pixel unit can approximately occupy rectangular pixel areas such as 3 × 4 pixels, 4 × 5 pixels, 5 × 7 pixels and the like.

In some embodiments, the distance between the camera and the depth detection device is no greater than 3 cm. For example, the distance between the imaging device and the depth detection device is 1.0cm, 1.1cm, 1.2cm, 1.3cm, 1.4cm, 1.5cm, 1.6cm, 1.7cm, 1.8cm, 1.9cm, 2.0cm, 2.1cm, 2.2cm, 2.3cm, 2.4cm, 2.50cm, 2.6cm, 2.7cm, 2.8cm, 2.9cm, 3.0 cm. It should be noted that the distance between the image capturing device and the depth detecting device is only an example, and is not limited to the range of 0.1 degree in accuracy. According to the actual design requirement of the mobile device, the precision of the included angle may be adjusted, for example, to a precision range of 1 degree, or the precision may be increased to reach 0.01 degree or more than 0.001 degree.

The processing means 604 is connected to the depth detection means, the camera means and the moving means. It should be understood that the various devices described may be disposed on a circuit board of a mobile device, and that the various devices are electrically connected to each other, directly or indirectly, to enable data transmission or interaction. The data transmission includes wireless network transmission (such as one or more of TDMA, CDMA, GSM, PHS, Bluetooth, etc.), wired network transmission (such as one or more of dedicated network, ADSL network, cable modem network, etc.), or interface transmission (such as obtained from a storage medium such as flash memory, usb disk, removable hard disk, optical disk, and floppy disk through an interface), etc.

The processing device 604 is configured to perform an indoor positioning method according to the embodiment corresponding to fig. 2: acquiring a first image and a second image which are respectively shot at different positions by the camera device; and determining position change information of the mobile equipment between different positions according to the current spatial scale parameter and the pixel position offset of the matched image features in the first image and the second image. It should be noted that, please refer to the embodiment corresponding to fig. 2 in the present application for details of the process and technical effects of the indoor positioning method, which are not described herein again.

In some embodiments, the mobile device is a mobile robot. The mobile robot is a machine device which automatically executes specific work, can receive the command of people, can run a pre-arranged program, and can perform an action according to a principle formulated by an artificial intelligence technology. The mobile robot can be used indoors or outdoors, can be used for industry or families, can be used for replacing security patrol, replacing people to clean the ground, and can also be used for family companions, auxiliary office work and the like. The mobile robot includes but is not limited to: one or more of devices capable of autonomous movement, such as an unmanned aerial vehicle, an industrial robot, a home-companion mobile device, a medical mobile device, a cleaning robot, a smart vehicle, and a patrol mobile device.

The mobile robot also comprises a mobile device, and the mobile device is used for receiving mobile control information and executing corresponding mobile operation. The mobility control information includes, but is not limited to: one or more of distance moved, direction of movement, speed of movement, acceleration, etc. For example, executing the mobility control information may cause the mobile device to: walking d meters in the southeast direction.

The mobility control information may be sent by the processing device, or may be generated by receiving an instruction transmitted by a server or a cloud. For example, by using the indoor positioning method, the mobile device determines that the mobile device itself deviates from a preset navigation route, and therefore, the processing device generates the movement control information according to the distance between the current position and the navigation route and the offset direction, and sends the movement control information to the mobile device to control the mobile device to perform a corresponding movement operation. For another example, the mobile device receives movement control information containing the forward distance and the direction from a server, and executes a movement operation according to the movement control information.

In some embodiments, the mobile device comprises at least one drive unit, such as a left wheel drive unit for driving a left side drive wheel of the mobile device and a right wheel drive unit for driving a right side drive wheel of the mobile device. The drive unit may contain one or more processors (CPUs) or micro-processing units (MCUs) dedicated to controlling the drive motor. For example, the micro-processing unit is used for converting information or data provided by the processing device into an electric signal for controlling a driving motor, and controlling the rotating speed, the steering direction and the like of the driving motor according to the electric signal so as to adjust the moving speed and the moving direction of the mobile equipment. The information or data is as determined by the processing means. The processor in the drive unit may be shared with the processor in the processing device or may be provided independently. For example, the drive unit functions as a slave processing device, the processing apparatus functions as a master device, and the drive unit performs movement control based on control of the processing apparatus. Or the drive unit may be shared with a processor in the processing device. The driving unit receives data provided by the processing device through the program interface. The driving unit is used for controlling the driving wheel based on the movement control information provided by the processing device.

The type of the mobile robot is different according to different application scenes. With the development of science and technology and the improvement of living standard, intelligent household appliances are widely applied. Thus, in some embodiments, the mobile robot is a sweeping robot. The sweeping robot can be called as an autonomous cleaner, an automatic sweeper, an intelligent dust collector and the like, is one of intelligent household appliances, and can complete cleaning, dust collection and floor wiping. Specifically, the floor sweeping robot can be controlled by a person (an operator holds a remote controller by hand or through an APP loaded on an intelligent terminal) or automatically complete floor cleaning work in a room according to a certain set rule, and can clean floor impurities such as hair, dust and debris on the floor.

Thereby, the sweeping robot further comprises a cleaning device for performing a cleaning operation. In some embodiments, the cleaning device includes, but is not limited to: the sweeping robot comprises a sweeping robot, and is characterized by comprising one or more of a side brush (or called side brush, side brush and the like) arranged on at least one side of the bottom of the sweeping robot, a rolling brush (or called cleaning roller, middle brush and the like) arranged near the center of the bottom of the sweeping robot, a fan used for providing adsorption force to sundries on the ground, a dust collecting device used for collecting the sundries on the ground, and the like. In an exemplary scenario, the sweeping robot stirs or adsorbs the hair, dust, debris and other ground impurities by the rolling brush during moving, and then sucks the ground impurities into the dust suction port arranged above the rolling brush by means of the suction force of the fan, so that the ground impurities are collected to complete the cleaning operation.

In some embodiments, the mobile device is a smart terminal including, but not limited to: one or more of Head Mounted Display (HMD), smart glasses, smart band, smart phone, tablet, and notebook.

For example, the smart terminal is an AR device (e.g., an AR headset, a smart phone, etc.), the AR device obtains data of the physical space through a camera device and a depth detection device, performs analysis and reconstruction through a processing device, and updates position change information of the device in the physical space in real time through the camera device, the depth detection device, an Inertial Measurement Unit (IMU), and other sensors on the augmented reality device, so as to perform fusion of a virtual scene and a real scene, thereby providing a real immersive viewing angle experience for an operator. For related description, please refer to the embodiment corresponding to fig. 1C, which is not repeated herein.

In some embodiments, the intelligent terminal further comprises an interaction device, wherein the interaction device is used for collecting interaction instructions (the forms of the interaction instructions include, but are not limited to, one or more of voice instructions, key instructions, touch instructions, eye movement instructions, gesture instructions and the like) of an operator and generating control signals to realize human-computer interaction. The interaction means include, but are not limited to: eye tracker, infrared inductor, camera, microphone, and various sensors etc.

The present application further provides a control system of a mobile device, configured to control the mobile device according to any embodiment corresponding to fig. 7 to perform the indoor positioning method according to any embodiment corresponding to fig. 2.

The control system of the mobile device may be implemented by software and hardware contained in a computer device. The computer device may be any computing device with mathematical and logical operations, data processing capabilities, including but not limited to: personal computer equipment, single server, server cluster, distributed server, cloud server, etc.

In a specific implementation, the cloud server may be a cloud computing platform provided by a cloud computing provider. Based on the architecture of the cloud server, the types of the cloud server include but are not limited to: Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS). Based on the nature of the cloud server, the types of the cloud server include, but are not limited to: public Cloud (Public Cloud) server, Private Cloud (Private Cloud) server, Hybrid Cloud (Hybrid Cloud) server, and the like. In some embodiments, the public Cloud service is, for example, Amazon's elastic computing Cloud (Amazon EC2), IBM's Blue Cloud, google's appeengine, and Windows' Azure service platform; the private cloud service end is, for example, an aristoson cloud computing service platform, an Amazon cloud computing service platform, a hundredth cloud computing platform, an Tencent cloud computing platform, and the like.

Please refer to fig. 7, which is a schematic structural diagram of a control system of a mobile device according to an embodiment of the present application. As shown, the control system 7 of the mobile device comprises an interface means 701, a storage means 702 and a processing means 703.

The devices may be disposed on a circuit board of the mobile device, and the devices are directly or indirectly electrically connected to each other to implement data transmission or interaction. The data transmission includes wireless network transmission (such as one or more of TDMA, CDMA, GSM, PHS, Bluetooth, etc.), wired network transmission (such as one or more of dedicated network, ADSL network, cable modem network, etc.), or interface transmission (such as obtained from a storage medium such as flash memory, usb disk, removable hard disk, optical disk, and floppy disk through an interface), etc.

It should be understood that the control system described in the embodiments of the present application is only one example of an application, and that the components of the system may have more or fewer components than shown, or a different configuration of components; the imaging device, the depth detection device, the interface device, the storage device, and the processing device are not necessarily separate components; for example, part or all of the camera device and the depth detection device may be integrated with the interface device, the storage device, and the processing device, and for example, part or all of the interface device and the storage device may be integrated with the processing device, and the like, which is not limited herein.

The interface device 701 is configured to connect a depth detection device and a camera device in the mobile device, so as to perform data transmission with the depth detection device and the camera device. The axis of the depth detection device and the optical axis of the camera device have a preset angular relationship, so that the depth information measured by the depth detection device corresponds to a pixel unit in an image shot by the camera device. For detailed description and technical effects, reference is made to the foregoing embodiments, which are not repeated herein.

The storage 702 is used to store at least one program. The processing device 703 executes the program after receiving the execution instruction. In some examples, the storage 702 may also include memory remote from the one or more processors, such as network-attached memory accessed via RF circuitry or external ports and a communication network, which may be the internet, one or more intranets, Local Area Networks (LANs), wide area networks (WLANs), Storage Area Networks (SANs), and the like, or suitable combinations thereof. The memory controller may control access to the memory by other components of the device, such as the CPU and peripheral interfaces. The memory optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory is optionally controlled by a memory controller by other components of the device, such as a CPU and peripheral interfaces. The Memory may also include Volatile Memory (Volatile Memory), such as Random Access Memory (RAM); the Memory may also include a Non-Volatile Memory (Non-Volatile Memory), such as a Read-Only Memory (ROM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid-State Drive (SSD).

The storage 702 may include at least one software module stored in the storage 702 in the form of software or Firmware (Firmware). The software module is used for storing images shot by the camera device, a map of an indoor space where the mobile equipment is located, a map database and various programs which can be executed by the mobile equipment, such as a path planning program of the mobile equipment; accordingly, the processing device 703 is configured to execute the program, so as to control the mobile device to perform the operation.

The processing device 703 is electrically connected to the storage device 702 and the interface device 701 through one or more communication buses or signal lines, and is configured to call and execute the at least one program, so as to coordinate the execution of the storage device, the depth detection device, and the camera device and implement the indoor positioning method according to the embodiment corresponding to fig. 2.

In some embodiments, the processing device 703 comprises an integrated circuit chip having signal processing capabilities; or include a general-purpose processor such as a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), discrete gate or transistor logic, discrete hardware components, or the like, that may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application. The general purpose processor may be a microprocessor, or any conventional processor such as a Central Processing Unit (CPU).

For details of the process and technical effects of the processing device 703 for coordinating the method executed by the storage device 702 and the interface device 701, please refer to the above embodiments, which is not described herein again.

It should be understood that the control system of the mobile device described in the embodiments of the present application is only one example of an application, and that the components of the device may have more or fewer components than shown, or a different configuration of components. The various components of the depicted figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

Fig. 8 is a schematic structural diagram of a control system of another mobile device according to an embodiment of the present application. As shown, the control system 8 includes an image pickup device 801, a depth detection device 802, an interface device 803, a storage device 804, and a processing device 805.

The devices may be disposed on a circuit board of the mobile device, and the devices are electrically connected to each other directly or indirectly to implement data transmission. The data transmission includes wireless network transmission (such as one or more of TDMA, CDMA, GSM, PHS, Bluetooth, etc.), wired network transmission (such as one or more of dedicated network, ADSL network, cable modem network, etc.), or interface transmission (such as obtained from storage media such as flash memory, usb disk, removable hard disk, optical disk, and floppy disk through an interface), etc.

The control system of the mobile device may be implemented by software, hardware or a combination of software and hardware comprised in a computer device, including one or more signal processing and/or application specific integrated circuits. The computer device may be any computing device with mathematical and logical operations, data processing capabilities, including but not limited to: personal computer equipment, single server, server cluster, distributed server, cloud server, etc.

The image pickup device 801 is used for picking up an image of an indoor environment. In some embodiments, the imaging device includes, but is not limited to: cameras, video cameras, camera modules integrated with optical systems or CCD chips, and camera modules integrated with optical systems and CMOS chips, and the like. According to the requirement of actual imaging, the lens that can be adopted by the camera or the video camera includes but is not limited to: standard lenses, telephoto lenses, fisheye lenses, wide-angle lenses, and the like. It should be understood that the foregoing embodiments can be referred to for the description of the image capturing apparatus, and the principle and technical effects thereof are similar, and are not described herein again.

The depth detection means 802 of the mobile device is used to detect depth information between it and a detection location (e.g., one or more detection points) in the indoor space. For example, when the depth detection device is oriented at an angle toward a ceiling plate in an indoor space, the obtained depth information is information of a distance from the depth detection device to the ceiling plate. For another example, when the depth detection device is vertically directed to the ceiling, the obtained depth information is the vertical height between the depth detection device and the ceiling. For another example, when the detection position is located on a room partition of the indoor space, for example, the detection position is located on a wall elevation, the obtained depth information is distance information from the depth detection device to the wall elevation. As another example, when the depth detection device is located on the surface of another object in the indoor space, the obtained depth information is information on the distance from the depth detection device to the surface of the object, and so on.

The depth detection device is used for acquiring depth information corresponding to the position of the detection position of at least one pixel unit in the image in the indoor space, and the depth information represents the distance between the detection position and the depth detection device. The detection position corresponds to a pixel unit in the image, and the pixel unit comprises one or more corresponding pixel points of the detection position mapped in the image.

The axis of the depth detection device 802 and the optical axis of the image capturing device 801 have a predetermined angular relationship, so that the depth information measured by the depth detection device 802 corresponds to a pixel unit in the image captured by the image capturing device 801.

It should be understood that the depth detection device and the relationship between the image capturing device and the depth detection device can be described with reference to the foregoing embodiments, and the principles and technical effects thereof are similar and will not be described herein again.

The interface unit 803 is connected to the depth detection unit 802 and the image pickup unit 801 to perform data transmission with the depth detection unit 802 and the image pickup unit 801.

The storage 804 is used for storing at least one program. In some examples, the storage 804 may also include memory remote from the one or more processors, such as network-attached memory accessed via RF circuitry or external ports and a communication network, which may be the internet, one or more intranets, Local Area Networks (LANs), wide area networks (WLANs), storage local area networks (SANs), and the like, or suitable combinations thereof. The memory controller may control access to the memory by other components of the device, such as the CPU and peripheral interfaces. The memory optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory is optionally controlled by a memory controller by other components of the device, such as a CPU and peripheral interfaces. The Memory may also include Volatile Memory (Volatile Memory), such as Random Access Memory (RAM); the Memory may also include a Non-Volatile Memory (Non-Volatile Memory), such as a Read-Only Memory (ROM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid-State Drive (SSD).

The storage 804 may include at least one software module stored in the storage 804 in the form of software or Firmware (Firmware). The software module is used for storing images shot by the camera device, a map of an indoor space where the mobile equipment is located, a map database and various programs which can be executed by the mobile equipment, such as a path planning program of the mobile equipment; accordingly, the processing device 803 is configured to execute the program, so as to control the mobile device to perform the operation.

The processing device 805, the interface device and the storage device may be electrically connected through one or more communication buses or signal lines, and configured to call and execute the at least one program, so as to coordinate the execution of the image capturing device 801, the depth detection device 802, the interface device 803 and the storage device 804, and implement the indoor positioning method according to the embodiment corresponding to fig. 2.

In some embodiments, the processing device 803 comprises an integrated circuit chip having signal processing capabilities; or include a general-purpose processor such as a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), discrete gate or transistor logic, discrete hardware components, or the like, that may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application. The general purpose processor may be a microprocessor, or any conventional processor such as a Central Processing Unit (CPU).

It should be understood that the control system described in the embodiments of the present application is only one example of an application, and that the components of the system may have more or fewer components than shown, or a different configuration of components. For example, part or all of the camera device, the depth detection device, the interface device, the storage device, and the processing device may be integrated into a positioning module, so as to be more conveniently embedded into a mobile device such as a mobile robot or a smart terminal.

The present application also provides a computer readable and writable storage medium storing a computer program which, when executed, implements the indoor positioning method of the above example with respect to the mobile device described in the embodiment of fig. 2.

Those of ordinary skill in the art will appreciate that the various illustrative functional blocks (e.g., the fig. 7 embodiment) and method steps (e.g., the fig. 2 embodiment) described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The computer program may be stored in a computer-readable storage medium when it is sold or used as a stand-alone product. Based on such understanding, the technical solutions of the present application may be embodied in the form of software products, or portions thereof, which substantially or wholly contribute to the prior art; in implementation, the computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the methods described in the examples of the present application.

In the examples provided herein, the computer-readable and writable storage medium may comprise read-only memory, random-access memory, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, a USB flash drive, a removable hard disk, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer.

Also, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium.

It should be understood, however, that computer-readable-writable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are intended to be non-transitory, tangible storage media. Disk and disc, as used in this application, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

The flowcharts and block diagrams in the figures described above of the present application illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various examples of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).

It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The above embodiments are merely illustrative of the principles and utilities of the present application and are not intended to limit the application. Any person skilled in the art can modify or change the above-described embodiments without departing from the spirit and scope of the present application. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical concepts disclosed in the present application shall be covered by the claims of the present application.

Claims

1. An indoor positioning method of a mobile device is characterized in that the mobile device comprises a depth detection device and a camera device, wherein the camera device takes an image of an indoor environment, the depth detection device provides depth information of an indoor space corresponding to at least one pixel unit in the image, and the depth information is used for determining a spatial scale parameter; the indoor positioning method comprises the following steps:

acquiring a first image and a second image which are respectively shot at different positions by the camera device; the pixel unit of the second image is provided with a second image feature matched with the first image feature in the first image, and an image feature pair is formed between the first image feature and the second image feature;

and determining position change information of the mobile equipment between different positions according to the current spatial scale parameter and the pixel position offset of the image feature pair in the first image and the second image.

2. The indoor positioning method of claim 1, wherein the current spatial scale parameter is a proportional relationship determined according to depth information of image features in corresponding pixel units detected by the depth detection device and depth information of corresponding image features in a map coordinate system of the camera device.

3. The indoor positioning method of claim 1, wherein the current spatial scale parameter is set according to depth information corresponding to the pixel unit in the first image or the second image; or the current spatial scale parameter is obtained according to historical depth information of the depth detection device or acquired from historical data of the spatial scale parameter.

4. The indoor positioning method of a mobile device according to claim 1, wherein a relative positional relationship between the camera and the depth detection means is set in advance.

5. The indoor positioning method of a mobile device according to claim 1, characterized in that a preceding image of the first and second images is taken by a camera during movement of the mobile device; alternatively, the prior image is from a map database of the indoor space.

6. The indoor positioning method of a mobile device according to claim 4, wherein the map database includes landmark information; wherein the landmark information includes: the prior image, and the matching image features located in the prior image.

7. The indoor positioning method of a mobile device according to claim 1 or 6, further comprising: and determining the position information of the mobile equipment in the indoor space according to the position change information and a preset map database of the indoor space.

8. The indoor positioning method of a mobile device according to claim 1, further comprising the steps of: based on the determined location change information and at least one of: the first image, the second image, and the depth information, updating a map database of the indoor space.

9. The indoor positioning method of a mobile device according to claim 1 or 4, further comprising: updating a positional relationship between the mobile device and a preset navigation route based on the determined position change information.

10. The indoor positioning method of a mobile device according to claim 1, further comprising: and during the continuous movement of the mobile equipment, identifying that the determined position change information corresponds to the entity object with the suspension space according to the depth information sequence provided by the depth detection device at different positions.

11. The indoor positioning method of a mobile device according to claim 10, wherein the physical object comprises at least one of: door, furniture.

12. The indoor positioning method of a mobile device according to claim 1, wherein the mobile device further comprises an inertial navigation detection device, the indoor positioning method further comprising: and correcting the inertial navigation data provided by the inertial navigation detection device by using the determined position change information.

13. A mobile device, comprising:

an image pickup device for picking up an image;

depth detection means for detecting depth information;

processing means, connected to the depth detection means and the camera means, for performing the indoor positioning method according to any one of claims 1 to 12.

14. The mobile device of claim 13, wherein the camera and the depth detection device are spaced no more than 3cm apart.

15. The mobile device according to claim 13, wherein an axis of the depth detection device is parallel to an optical axis of the camera device, and an included angle between the axis and the optical axis is in a range from 0 degrees to a maximum value of a field angle of the camera device, or an included angle between the axis and the optical axis is in a range from 0 degrees to a minimum value of the field angle of the camera device.

16. The mobile device of claim 13, wherein the depth detection device comprises a laser ranging device, or a single point ToF sensor, or an ultrasonic sensing device.

17. The mobile device of claim 13, wherein the mobile device comprises any of: mobile robot, intelligent terminal.

18. The mobile device of claim 17, wherein the mobile robot is a sweeping robot; correspondingly, the sweeping robot further comprises a cleaning device for executing cleaning operation.

19. A control system for a mobile device, comprising:

the interface device is used for connecting the depth detection device and the camera device in the mobile equipment; the axis of the depth detection device and the optical axis of the camera device have a preset angle relationship, so that the depth information measured by the depth detection device corresponds to a pixel unit in an image shot by the camera device;

storage means for storing at least one program;

processing means connected to the interface means and the storage means for calling and executing the at least one program to coordinate the storage means, the depth detection means, and the camera means to execute and implement the indoor positioning method according to any one of claims 1 to 12.

20. A control system for a mobile device, comprising:

the camera shooting device is used for shooting an image of an indoor environment;

the depth detection device is used for providing depth information of at least one pixel unit in the image in the indoor space; the axis of the depth detection device and the optical axis of the camera device have a preset angular relationship, so that the depth information measured by the depth detection device corresponds to a pixel unit in an image shot by the camera device;

the interface device is connected with the depth detection device and the camera device;

storage means for storing at least one program;

21. A computer-readable storage medium, characterized by storing at least one program which, when invoked, executes and implements an indoor positioning method according to any one of claims 1-12.