WO2024005380A1

WO2024005380A1 - Electronic device for generating map of space and control method therefor

Info

Publication number: WO2024005380A1
Application number: PCT/KR2023/007198
Authority: WO
Inventors: 조호식; 미탈아카쉬; 사디사지드
Original assignee: 삼성전자주식회사
Priority date: 2022-06-30
Filing date: 2023-05-25
Publication date: 2024-01-04
Also published as: KR20240002835A

Abstract

An electronic device for generating a map of a space and a control method therefor are provided. The control method for an electronic device for generating a map of a space comprises the steps of: while the electronic device recognizes that a mobile device having the electronic device mounted thereon travels through a space, receiving, from the mobile device, information regarding the distance traveled, and acquiring direction information of the electronic device via at least one sensor; acquiring a first map of the space on the basis of the information regarding the distance traveled and the direction information; while the electronic device recognizes that the mobile device having the electronic device mounted thereon travels through the space, receiving wireless signals from a plurality of APs located within the space; and acquiring a second map of the space on the basis of the information regarding the distance traveled, the direction information, and the received wireless signals.

Description

Electronic device for generating a map of space and method for controlling the same

The present disclosure relates to an electronic device that generates a map of space and a control method thereof, and more specifically, to an electronic device that generates a map of space based on information collected while being mounted on and moved in a mobile device and its control. It's about method.

Recently, the location of the user who currently owns the electronic device is identified, and various services are provided based on the identified location. For example, electronic devices use GPS information, wireless signals, etc. to identify the location information of the user who owns the electronic device, and provide various services (e.g., route finding service, weather service) based on the identified location information. services, etc.) are provided.

In particular, in order to estimate the user's location in an indoor space such as inside a building, conventional electronic devices use pre-stored maps of the indoor space and various signals (e.g., Wi-Fi signals, geomagnetic signals, etc.) to estimate the user's location in an indoor space such as inside a building. The user's location in space was estimated.

According to an embodiment of the present disclosure, a method of controlling an electronic device that generates a map of space includes moving the mobile device equipped with the electronic device while the electronic device recognizes that the mobile device is moving in space. Receiving information about a moving distance from a device and obtaining direction information of the electronic device through at least one sensor; Obtaining a first map of the space based on the information about the moving distance and the direction information; Receiving wireless signals from a plurality of APs located within the space while the electronic device recognizes that a mobile device equipped with the electronic device is moving through the space; And, it includes; obtaining a second map of the space based on the information about the moving distance, the direction information, and the received wireless signal.

According to an embodiment of the present disclosure, an electronic device that generates a map of space includes a camera, at least one sensor, at least one communication interface, a memory, and the camera, the at least one sensor, and the at least one communication interface. and at least one processor that controls the electronic device in conjunction with a memory. At this time, the at least one processor receives information about the moving distance from the mobile device through the at least one communication interface while the electronic device recognizes that the mobile device equipped with the electronic device is moving in space. . And, the at least one processor obtains direction information of the electronic device through the at least one sensor. And, the at least one processor obtains a first map of the space based on the information about the moving distance and the direction information. And, the at least one processor receives wireless signals from a plurality of APs located in the space through the at least one communication interface while the electronic device recognizes that the mobile device equipped with the electronic device is moving in the space. do. And, the at least one processor obtains a second map of the space based on the information about the moving distance, the direction information, and the received wireless signal.

A computer-readable recording medium including a program for executing a control method of an electronic device that generates a map of space, wherein the control method includes moving a mobile device equipped with the electronic device through space. During recognition, receiving information about a moving distance from the mobile device and obtaining direction information of the electronic device through at least one sensor; Obtaining a first map of the space based on the information about the moving distance and the direction information; Receiving wireless signals from a plurality of APs located within the space while the electronic device recognizes that a mobile device equipped with the electronic device is moving through the space; And, it includes; obtaining a second map of the space based on the information about the moving distance, the direction information, and the received wireless signal.

1 is a diagram illustrating a map generating system including an electronic device and a mobile device according to an embodiment of the present disclosure;

2 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure;

FIG. 3 is a diagram illustrating a method for an electronic device to obtain a first map and a second map, according to an embodiment of the present disclosure;

4 is a diagram illustrating a waypoint to which an electronic device moves, according to an embodiment of the present disclosure;

5 is a diagram illustrating a first map generated based on waypoints, according to an embodiment of the present disclosure;

6 is a diagram illustrating a third map generated by an electronic device according to an embodiment of the present disclosure;

7 is a flowchart illustrating a method for identifying a user location according to an embodiment of the present disclosure;

8 and 9 are diagrams for explaining a method of obtaining a fourth map indicating information about objects placed in space, according to an embodiment of the present disclosure;

FIG. 10 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

Below, various embodiments of the present disclosure are described. However, this is not intended to limit the technology of the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure. .

In this document, expressions such as “have,” “may have,” “includes,” or “may include” refer to the existence of the corresponding feature (e.g., a numerical value, function, operation, or component such as a part). , and does not rule out the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” includes (1) at least one A, (2) at least one B, or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in this document can modify various components regardless of order and/or importance, and refer to one component. It is only used to distinguish from other components and does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, a first component may be renamed a second component without departing from the scope of rights described in this document, and similarly, the second component may also be renamed to the first component.

Terms such as “module,” “unit,” and “part” used in this document are terms to refer to components that perform at least one function or operation, and these components are implemented in hardware or software. Alternatively, it can be implemented through a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except in cases where each needs to be implemented with individual specific hardware, and is integrated into at least one processor. It can be implemented as:

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), the component and the It may be understood that no other component (e.g., a third component) exists between other components.

As used in this document, the expression “configured to” depends on the situation, for example, “suitable for,” “having the capacity to.” ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

Terms used in this document are merely used to describe specific embodiments and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in this document, have an ideal or excessively formal meaning. It is not interpreted as In some cases, even terms defined in this document cannot be interpreted to exclude embodiments of this document.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings. However, in describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted. In connection with the description of the drawings, similar reference numbers may be used for similar components.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

In order to estimate the user's location indoors, such as inside a building, the electronic device uses a map of the inside of the building and various signals (e.g., Wi-Fi signals, geomagnetic signals, etc.) to estimate the user's location indoors. can do. In other words, in order for an electronic device to estimate the user's location indoors, a map of the indoor space is needed. However, conventionally, in order to obtain a map of the indoor space, a map (e.g., floor plan) provided by the building owner was used, but if the building owner did not have a floor plan, the user's location indoors was estimated. There was a problem that made it difficult to do.

The present disclosure has been devised to solve the above-described problems. Through the present disclosure, an electronic device mounted on a mobile device acquires various information, obtains various maps, and uses the obtained maps to determine the user's location in the indoor space. It can be estimated.

FIG. 1 is a diagram illustrating a map generating system including a mobile device and an electronic device, according to an embodiment of the present disclosure. According to one embodiment of the present disclosure, a map creation system includes an electronic device 100 and a mobile device 200. According to an embodiment of the present disclosure, the electronic device 100 may be a smartphone, but this is only an embodiment and may be implemented as various devices such as a tablet PC, a laptop PC, etc.

The electronic device 100 can be mounted on the mobile device 200 and move within an indoor space (for example, inside a shopping mall, etc.). In particular, while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in space, the electronic device 100 moves from the mobile device 200 to the mobile device 200. You can receive information about distance. Additionally, the electronic device 100 may obtain information on the direction in which the electronic device 100 moves through at least one sensor provided in the electronic device 100. Additionally, the electronic device 100 may obtain a first map (or floor map) of the space based on information about the movement distance and direction information.

In addition, while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in space, the electronic device 100 can receive wireless signals from a plurality of APs located in the space. there is. The electronic device 100 may obtain a second map (or wireless signal map) of the space based on information about the moving distance, direction information, and received wireless signals.

The electronic device 100 may learn a neural network model (or artificial intelligence model) for estimating the user's location in an indoor space using the second map. Additionally, the electronic device 100 can estimate the user's location using the learned neural network model. Additionally, the electronic device 100 may identify the user's location by applying the estimated user's location and the first map to a sensor fusion algorithm.

The mobile device 200 may include at least one wheel 210, a handle for the user to hold and move the mobile device 200, and an odometer. At this time, the odometer can obtain information about the wheel 210 that can move in the indoor space and the number of rotations of the wheel 210. The mobile device 200 can move around the indoor space via wheels 210 while the electronic device 100 is mounted, and information about the distance traveled by the mobile device 200 while the mobile device 200 is moving. can be transmitted to the electronic device 100. At this time, the information about the moving distance may be information about the number of rotations of the wheel 210 provided in the moving device 200, but this is only an example and is calculated based on the number of rotations of the wheel 210. May include information about travel distance.

Meanwhile, the mobile device 200 may be moved within an indoor space by a person, but this is only an example, and may be moved within an indoor space in an autonomous driving manner. Additionally, as shown in FIG. 1, the mobile device 200 may be implemented in the form of a bar, but this is only an example, and of course, it may be implemented in various forms. In addition, as shown in FIG. 1, the electronic device 100 may be mounted to face upward based on the moving direction of the mobile device, but this is only an example, and the electronic device 100 may be mounted as a mobile device. It can be mounted to face the front based on the direction of movement.

As described above, the electronic device 100 mounted on the mobile device acquires various maps to identify the user's location, so there is no need for a separate floor plan for the indoor space, and the location of furniture or objects existing in the indoor space is Even if it changes, it is possible to obtain a map to identify the user's location.

Figure 2 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 2, the electronic device 100 includes at least one sensor 110, at least one communication interface 120, memory 130, camera 140, display 150, and at least one processor. It may include (160). Meanwhile, the configuration of the electronic device 100 is not limited to the configuration shown in FIG. 2, and of course, configurations that are obvious to those skilled in the art may be added or deleted.

At least one sensor 110 is configured to sense various information about the electronic device 100 or the environment surrounding the electronic device 100. At least one sensor 110 may include an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. In particular, the electronic device 100 may use an acceleration sensor or a gyro sensor to obtain information about the direction of movement of the electronic device 100 while the electronic device 100 is mounted on the mobile device 200 and moved. Additionally, the geomagnetic sensor can detect magnetic signals around the electronic device 100. In addition, the electronic device 100 may include various sensors to detect the environment around the electronic device 100 or the user's status.

At least one communication interface 120 includes at least one circuit and can communicate with various types of external devices or servers. At least one communication interface 120 includes a BLE (Bluetooth Low Energy) module, a Wi-Fi communication module, a cellular communication module, a 3G (3rd generation) mobile communication module, a 4G (4th generation) mobile communication module, and a 4th generation LTE (Long It may include at least one of a Term Evolution) communication module and a 5G (5th generation) mobile communication module.

In particular, the electronic device 100 may communicate with the mobile device 200 using a first communication module (eg, BLE module). At this time, the electronic device 100 may receive information about the moving distance of the electronic device (or mobile device) using the first communication module. Additionally, the electronic device 100 uses a second communication module (e.g., Wi-Fi communication module) to receive a wireless signal (e.g., Wi-Fi communication module) from at least one Access Point (AP) disposed in an indoor space. signal) can be received.

The memory 130 may store an operating system (OS) for controlling the overall operation of the components of the electronic device 100 and instructions or data related to the components of the electronic device 100. In particular, the memory 130 may include various modules to generate a map of the indoor space and identify the user's location using the generated map.

In addition, the memory 130 includes an object recognition model, which is a neural network model for recognizing objects included in an image, and a location estimation model, which is a neural network model for estimating the user's location using a wireless signal (for example, a Wi-Fi signal). etc. can be saved. However, this is only an example, and at least one of the object recognition model and the location estimation model may be stored in an external server.

Meanwhile, the memory 130 may be implemented as non-volatile memory (ex: hard disk, solid state drive (SSD), flash memory), volatile memory (which may also include memory within the processor 160), etc.

The camera 140 is configured to acquire images taken around the electronic device 100. The camera 140 may be placed on the front and rear of the electronic device 100 and may include a plurality of image sensors (eg, a telephoto image sensor, a wide-angle image sensor, an ultra-wide-angle image sensor, etc.). At least one processor 160 may input an image captured through the camera 140 into a neural network model to obtain information about an object included in the captured image.

The display 150 can display various images received from the outside or captured by the camera 140. Meanwhile, the display 150 may be implemented as a liquid crystal display panel (LCD), an organic light emitting diode (OLED), etc., and the display 130 may also be implemented as a flexible display, a transparent display, etc., depending on the case. . However, the display 150 according to the present disclosure is not limited to a specific type.

In particular, the display 150 can display a map of an indoor space and display information about a user's location identified by various information on the map.

At least one processor 160 may be electrically connected to the memory 130 to control the overall functions and operations of the electronic device 100. At least one processor 160 may generate a map of the indoor space stored in the non-volatile memory and load data for performing an operation to identify the user's location into the volatile memory. Here, loading refers to an operation of loading and storing data stored in non-volatile memory in volatile memory so that at least one processor 190 can access it.

In particular, the at least one processor 160 communicates with the mobile device 200 via at least one communication interface 120 while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in space. Information about the moving distance can be received from (200). Additionally, at least one processor 160 may obtain direction information of the electronic device through at least one sensor 110. And, at least one processor 110 may obtain (or generate) a first map of space based on information about the moving distance and direction information.

In particular, the at least one processor 160 allows the mobile device 200 to move in space based on information about the moving distance received from the mobile device 200 and direction information obtained through at least one sensor 110. You can obtain information about waypoints along the way. Additionally, at least one processor 160 may correct the path to the waypoint using image processing. In addition, at least one processor 160 may obtain (or generate) a first map including information on the obstacle area and information on the movement area in the space based on the corrected path. At this time, information about the moving distance may be obtained based on the number of rotations of the wheels provided in the moving device 200.

And, the at least one processor 160 is positioned in the space through at least one communication interface 120 while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in the space. Wireless signals can be received from multiple APs. And, at least one processor 160 may obtain (or generate) a second map of the space based on information about the moving distance, direction information, and the received wireless signal. At this time, the wireless signal may be a Wi-Fi signal, and at least one processor 160 may obtain (or generate) a second map of the space by mapping information about the waypoint and the Wi-Fi signal.

In addition, at least one processor 160 may acquire information about magnetic signals through a geomagnetic sensor while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in space. there is. Additionally, at least one processor 160 may obtain (or generate) a third map of the space based on information about the moving distance, direction information, and magnetic signals.

Additionally, at least one processor 160 may use the obtained second map to train a neural network model (or location estimation model) that estimates the location using a wireless signal.

Additionally, at least one processor 160 may identify the location of a user equipped with an electronic device in space using a neural network model and a sensor fusion algorithm.

Additionally, at least one processor 160 may acquire an image using the camera 140 while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in space. there is. At least one processor 160 may obtain information about objects included in the image by inputting the acquired image into a learned neural network model. In addition, at least one processor 160 may obtain a fourth map indicating the location of the object placed in space based on information about the object and waypoint included in the image.

Meanwhile, in addition to the configuration shown in FIG. 2, the electronic device 100 may include various configurations such as a microphone, speaker, etc.

Hereinafter, with reference to FIGS. 3 to 9 , a method for the electronic device 100 to generate a plurality of maps and identify the user's location using the plurality of maps will be described in detail.

FIG. 3 is a diagram illustrating a method for an electronic device to obtain a first map and a second map, according to an embodiment of the present disclosure.

First, the electronic device 100 can run an application for creating a map. Additionally, the electronic device 100 can move around the indoor space while being mounted on the mobile device 200.

While the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in an indoor space (i.e., while the application for creating a map is running and the map recognition function is running) The device 100 may receive information about the moving distance from the mobile device 200 and obtain direction information of the electronic device 100 through at least one sensor 110 (310). Specifically, the electronic device 100 may receive information about the number of wheel rotations of the mobile device 200 from the mobile device 200 while the electronic device 100 is mounted on the mobile device 200 and moves, Information about the moving distance can be obtained based on information about the number of wheel rotations of the mobile device 200. In addition, the electronic device 100 uses an IMU sensor (Inertial Measurement Unit sensor) (e.g., an acceleration sensor, a gyro sensor) to provide information about the direction of movement while the electronic device 100 is mounted on the mobile device 200 and moves. can be obtained. Additionally, the electronic device 100 may store information about the movement distance and information about the direction of movement by mapping them to a timestamp.

The electronic device 100 may obtain information about a waypoint while the mobile device 200 moves through space based on information about the movement distance and information about the direction of movement (320). At this time, the information about the waypoint may be information about the point to which the electronic device 100 moved while the mobile device 200 moved in space. That is, as shown in FIG. 4, the electronic autonomy 100 can acquire information 410 about the waypoint, which is information about the point where the electronic device 100 is mounted and moved by the mobile device 200. there is.

The electronic device 100 may correct the path to the waypoint (330). Specifically, since the electronic device 100 (or mobile device 200) cannot pass through all movement paths in space, the electronic device 100 may correct the path to the waypoint through an image processing operation. For example, the electronic device 100 may correct a path to a waypoint through image dilation of the waypoint.

The electronic device 100 may identify an obstacle area and a movement area within the space based on the corrected path (340). Specifically, the electronic device 100 may identify the area in which the electronic device 100 (or the mobile device 200) moved as a movement area based on the corrected path, and the electronic device 100 (or the mobile device 200) The area where (200)) cannot move can be identified as an obstacle area.

The electronic device 100 may acquire the first map based on the identified obstacle area and movement area. Specifically, as shown in FIG. 5 , the electronic device 100 may obtain a first map 510 (floor map) divided into a movement area 520 and an obstacle area 530.

Meanwhile, while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in an indoor space, the electronic device 100 can receive wireless signals from a plurality of APs located in the space. Can (360). At this time, the electronic device 100 may receive a wireless signal including a timestamp.

Additionally, the electronic device 100 can map waypoints and wireless signals (370). That is, the electronic device 100 can map waypoints and wireless signals based on timestamps. For example, the electronic device 100 can map waypoints and wireless signals, as shown in Table 1 below.

X 좌표X coordinate	Y 좌표Y coordinate	AP1AP1	AP2AP2	AP3AP3
10.010.0	12.512.5	-45-45	-43-43	-12-12
11.011.0	13.513.5	-50-50	-35-35	-20-20

That is, the electronic device 100 can obtain information about the strength of Wi-Fi signals received from a plurality of APs at each moving point while the electronic device 100 moves. Then, the electronic device 100 performs mapping. Based on the results, a second map (or wireless signal map) can be obtained (380).

The electronic device 100 may learn a neural network model based on the acquired second map (390). At this time, in order to estimate the user's location in space, the neural network model may input the strength of the Wi-Fi signal obtained by the electronic device from a plurality of APs and output information about the user's location in space.

Meanwhile, in the above-described embodiment, it has been described that the electronic device 100 generates a second map by collecting wireless signals from a plurality of APs. However, this is only an embodiment, and the electronic device 100 uses a geomagnetic sensor to detect a magnetic field. A third map can be generated by collecting signals.

Specifically, while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in an indoor space, the electronic device 100 may acquire a magnetic signal from a geomagnetic sensor. At this time, the electronic device 100 may acquire a magnetic signal including a timestamp.

Additionally, the electronic device 100 can map waypoints and magnetic signals. That is, the electronic device 100 can map waypoints and magnetic signals based on timestamps. That is, the electronic device 100 may obtain information about the strength of the magnetic signal obtained at each moving point while the electronic device 100 moves. And, the electronic device 100 may obtain the third map 610 as shown in FIG. 6 based on the mapping result. At this time, the third map may be a map that represents the strength of the magnetic signal detected at each point in color or brightness based on the first map. Specifically, the third map may be displayed darker as the strength of the magnetic signal becomes stronger, and may be displayed brighter as the strength of the magnetic signal becomes weaker.

The electronic device 100 may learn a neural network model based on the acquired third map. At this time, the neural network model can output information about the user's location in space by inputting the strength of the magnetic signal detected by the electronic device to estimate the user's location in space.

Figure 7 is a flowchart illustrating a method for identifying a user's location according to an embodiment of the present disclosure. In particular, FIG. 7 is a diagram for explaining a method in which the electronic device 100 acquires a plurality of maps, learns a neural network model based on the acquired maps, and then identifies the user's location using the neural network model.

The electronic device 100 may acquire a wireless signal (S710). At this time, the wireless signal may be a Wi-Fi signal, but is not limited thereto.

The electronic device 100 may estimate user location information by inputting information about the wireless signal into a neural network model (S720). That is, as described in FIG. 3, the electronic device 100 can estimate information about the user's location within the current indoor space by inputting information about the currently detected wireless signal into the neural network model learned by the second map. there is.

The electronic device 100 may identify the user's location using a sensor fusion algorithm (S730). Specifically, the electronic device 100 may not always detect wireless signals but may detect wireless signals at regular time intervals (for example, 20 seconds, etc.). In this case, it is difficult to identify the user's location between the times when the wireless signal is detected. To overcome this, the electronic device 100 may identify the user's location using a sensor fusion algorithm such as a particle filter. At this time, the particle filter is one of the prediction technologies through simulation based on trial and error, and is also called the SMC (Sequential Monte Carlo) method.

Meanwhile, in FIG. 7, it is explained that the user's location is estimated and identified using a wireless signal, but this is only an example, and the user's location can be estimated and identified using a magnetic signal. For example, the electronic device 100 detects a magnetic signal at the current location and inputs information about the detected magnetic signal into a neural network model learned by the third map to provide information about the user's current location within the indoor space. It can be estimated. Additionally, the electronic device 100 may identify the user's location using a sensor fusion algorithm. In addition, of course, the electronic device 100 can identify the location of the user using the electronic device 100 using both wireless signals and magnetic signals.

Figures 8 and 9 are diagrams for explaining a method of obtaining a fourth map indicating information about objects placed in space, according to an embodiment of the present disclosure.

First, the electronic device 100 can acquire an image using the camera 140 provided in the electronic device 100 (S810). Specifically, while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in an indoor space (i.e., the application for creating a map is executed and the map recognition function is executed) (while) the electronic device 100 may acquire an image using the camera 140 provided in the electronic device 100. At this time, the acquired image may include at least one object. Additionally, the electronic device 100 may store a timestamp along with the image.

The electronic device 100 may obtain information about objects included in the image by inputting the image into a learned neural network model (eg, object recognition model) (S820). Specifically, the electronic device 100 can acquire the first image 910, as shown on the left side of FIG. 9, and input the first image 910 into the learned neural network model to display the first image 910 on the right side of FIG. 9. As shown, information 920 about objects included in the image can be obtained. At this time, the information 920 about the object may include the name, capacity, image, number, etc. of the object.

The electronic device 100 can map information about objects included in the image and information about waypoints (S830). At this time, the electronic device 100 may map information about the object included in the image and information about the waypoint based on the timestamp stored when acquiring the object.

The electronic device 100 may obtain a fourth map indicating the location of the object placed in space based on the mapping result (S840). That is, the electronic device 100 may obtain a fourth map indicating the location of an object placed in space on the first map.

First, the electronic device 100 determines the moving distance from the mobile device 200 while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in space. Information about and acquires direction information of the electronic device 100 through at least one sensor 110 (S1010). At this time, the information about the moving distance may be information about the number of rotations of the wheels provided in the moving device, but is not limited thereto.

Then, the electronic device 100 obtains a first map of the space based on information about the moving distance and direction information (S1020). Specifically, the electronic device 100 may obtain information about a waypoint while the mobile device 200 moves through space based on information about the movement distance and direction information. Additionally, the electronic device 100 may correct the path to the waypoint using image processing. Additionally, the electronic device 100 may obtain the first map including information on the obstacle area and information on the movement area in the space based on the corrected path.

The electronic device 100 receives wireless signals from a plurality of APs located in the space while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in the space (S1030). At this time, the wireless signal may be a Wi-Fi signal.

The electronic device 100 obtains a second map of the space based on information about the moving distance, direction information, and the received wireless signal (S1040). Specifically, the electronic device 100 may obtain a second map of space by mapping information about waypoints and Wi-Fi signals.

Meanwhile, the electronic device 100 may acquire information about magnetic signals through a geomagnetic sensor while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in space. Additionally, the electronic device 100 may obtain a third map of space based on information about the moving distance, direction information, and magnetic signals.

Meanwhile, the electronic device 100 can use the acquired second map to learn a neural network model that estimates the location using wireless signals. Additionally, the electronic device 100 may identify the location of a user equipped with the electronic device 100 in space using a neural network model and a sensor fusion algorithm.

Meanwhile, the electronic device 100 uses the camera 140 provided in the electronic device 100 while the electronic device 100 recognizes that the mobile device 200 equipped with the electronic device 100 is moving in space. Thus, the image can be obtained. Additionally, the electronic device 100 may obtain a fourth map indicating the location of the object placed in space based on information about the object and waypoint included in the image. At this time, information about the object can be obtained by inputting the acquired image into a learned neural network model.

According to an embodiment of the present disclosure as described above, the electronic device 100 can generate a map of an indoor space without a separate floor plan and estimate the user's location using the generated map.

Meanwhile, functions related to artificial intelligence (eg, learning function and inference function for a neural network model) according to the present disclosure are operated through the processor and memory of the electronic device.

The processor may consist of one or multiple processors. At this time, one or more processors may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a Neural Processing Unit (NPU), but are not limited to the examples of the processors described above.

CPU is a general-purpose processor that can perform not only general calculations but also artificial intelligence calculations, and can efficiently execute complex programs through a multi-layer cache structure. CPUs are advantageous for serial processing, which allows organic connection between previous and next calculation results through sequential calculations. The general-purpose processor is not limited to the above-described examples, except where specified as the above-described CPU.

GPU is a processor for large-scale operations such as floating-point operations used in graphics processing, and can perform large-scale operations in parallel by integrating a large number of cores. In particular, GPUs may be more advantageous than CPUs in parallel processing methods such as convolution operations. Additionally, the GPU can be used as a co-processor to supplement the functions of the CPU. The processor for mass computation is not limited to the above-described example, except for the case where it is specified as the GPU.

NPU is a processor specialized in artificial intelligence calculations using artificial neural networks, and each layer that makes up the artificial neural network can be implemented in hardware (e.g., silicon). At this time, the NPU is designed specifically according to the company's requirements, so it has a lower degree of freedom than a CPU or GPU, but can efficiently process artificial intelligence calculations requested by the company. Meanwhile, as a processor specialized for artificial intelligence calculations, NPU can be implemented in various forms such as TPU (Tensor Processing Unit), IPU (Intelligence Processing Unit), and VPU (Vision processing unit). The artificial intelligence processor is not limited to the examples described above, except where specified as the NPU described above.

Additionally, one or more processors may be implemented as a System on Chip (SoC). At this time, in addition to one or more processors, the SoC may further include memory and a network interface such as a bus for data communication between the processor and memory.

If the SoC (System on Chip) included in the electronic device includes a plurality of processors, the electronic device uses some of the processors to perform artificial intelligence-related operations (for example, learning of an artificial intelligence model). or operations related to inference) can be performed. For example, an electronic device can perform operations related to artificial intelligence using at least one of a plurality of processors, a GPU, NPU, VPU, TPU, or hardware accelerator specialized for artificial intelligence operations such as convolution operation, matrix multiplication operation, etc. there is. However, this is only an example, and of course, calculations related to artificial intelligence can be processed using general-purpose processors such as CPUs.

Additionally, electronic devices can perform calculations on functions related to artificial intelligence using multiple cores (eg, dual core, quad core, etc.) included in one processor. In particular, electronic devices can perform artificial intelligence operations such as convolution operations and matrix multiplication operations in parallel using multi-cores included in the processor.

One or more processors control input data to be processed according to predefined operation rules or artificial intelligence models stored in memory. Predefined operation rules or artificial intelligence models are characterized by being created through learning.

Here, being created through learning means that a predefined operation rule or artificial intelligence model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself that performs the artificial intelligence according to the present disclosure, or may be performed through a separate server/system.

An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value, and the operation of the layer is performed using the operation result of the previous layer and at least one defined operation. Examples of neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), and Deep Neural Network (BRDNN). There are Q-Networks (Deep Q-Networks) and Transformer, and the neural network in this disclosure is not limited to the above-described examples except where specified.

A learning algorithm is a method of training a target device (eg, a robot) using a large number of learning data so that the target device can make decisions or make predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in the present disclosure is specified. Except, it is not limited to the examples described above.

Additionally, methods according to various embodiments of the present disclosure may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smartphones) or online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily.

Methods according to various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage media that can be read by a machine (e.g., a computer). The device stores information stored from the storage medium. A device capable of calling a command and operating according to the called command may include an electronic device (eg, the robot 100) according to the disclosed embodiments.

Meanwhile, a storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is semi-permanently stored in a storage medium and temporary storage media. It does not distinguish between cases where it is stored as . For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

When the instruction is executed by a processor, the processor may perform the function corresponding to the instruction directly or using other components under the control of the processor. Instructions may contain code generated or executed by a compiler or interpreter.

In the above, preferred embodiments of the present disclosure have been shown and described, but the present disclosure is not limited to the specific embodiments described above, and may be used in the technical field to which the disclosure pertains without departing from the gist of the disclosure as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical ideas or perspectives of the present disclosure.

Claims

In a method of controlling an electronic device that generates a map of space,

While the electronic device recognizes that the mobile device equipped with the electronic device is moving in space, it receives information about the moving distance from the mobile device, and receives direction information of the electronic device through at least one sensor. Obtaining a;

Obtaining a first map of the space based on the information about the moving distance and the direction information;

Receiving wireless signals from a plurality of APs located within the space while the electronic device recognizes that a mobile device equipped with the electronic device is moving through the space;

A control method including; acquiring a second map of the space based on the information about the moving distance, the direction information, and the received wireless signal.
According to paragraph 1,

The step of acquiring the first map is,

Obtaining information about a waypoint while the mobile device moves in the space based on the information about the movement distance and the direction information; and

correcting the path to the waypoint using image processing;

A control method including; obtaining the first map including information on an obstacle area and a movement area in the space based on the corrected path.
According to paragraph 2,

The wireless signal is a Wi-Fi signal,

The step of acquiring the second map is,

A control method for obtaining a second map of the space by mapping information about the waypoint and the Wi-Fi signal.
According to paragraph 1,

The control method is,

acquiring information about magnetic signals through a geomagnetic sensor while the electronic device recognizes that the mobile device equipped with the electronic device is moving in space;

A control method including; acquiring a third map of the space based on the information about the moving distance, the direction information, and the magnetic signal.
According to paragraph 1,

Information on the moving distance is,

A control method characterized in that it is obtained based on the number of rotations of the wheels provided in the mobile device.
According to paragraph 1,

A control method comprising: training a neural network model for estimating a location using the wireless signal using the obtained second map.
According to clause 6,

A control method comprising: identifying the location of a user equipped with an electronic device within the space using the neural network model and the sensor fusion algorithm.
According to paragraph 2,

The control method is,

acquiring an image using a camera provided in the electronic device while the electronic device recognizes that the mobile device equipped with the electronic device is moving in space;

A control method including; acquiring a fourth map indicating the location of an object placed in the space based on information about the object included in the image and information about the waypoint.
According to clause 8,

The control method is,

A control method further comprising: inputting the acquired image into a learned neural network model to obtain information about an object included in the image.
In an electronic device that generates a map of space,

camera;

at least one sensor;

at least one communication interface;

Memory; and

At least one processor that controls the electronic device in conjunction with the camera, the at least one sensor, the at least one communication interface, and the memory,

The at least one processor,

Receiving information about a moving distance from the mobile device through the at least one communication interface while the electronic device recognizes that the mobile device equipped with the electronic device is moving in space,

Obtaining direction information of the electronic device through the at least one sensor,

Obtaining a first map of the space based on the information about the moving distance and the direction information,

Receiving wireless signals from a plurality of APs located in the space through the at least one communication interface while the electronic device recognizes that a mobile device equipped with the electronic device is moving in the space,

An electronic device that obtains a second map of the space based on the information about the moving distance, the direction information, and the received wireless signal.
According to clause 10,

The at least one processor,

Obtaining information about a waypoint while the mobile device moves in the space based on the information about the movement distance and the direction information,

Correcting the path to the waypoint using image processing,

An electronic device, characterized in that, based on the corrected path, the first map including information on an obstacle area and a movement area in the space is acquired.
According to clause 11,

The wireless signal is a Wi-Fi signal,

The at least one processor,

An electronic device characterized in that it obtains a second map of the space by mapping information about the waypoint and the Wi-Fi signal.
According to clause 10,

The at least one sensor includes a geomagnetic sensor,

The at least one processor,

Obtaining information about magnetic signals through the geomagnetic sensor while the electronic device recognizes that the mobile device equipped with the electronic device is moving in space,

An electronic device characterized in that it obtains a third map of the space based on the information about the moving distance, the direction information, and the magnetic signal.
According to clause 10,

Information on the moving distance is,

An electronic device characterized in that it is obtained based on the number of rotations of a wheel provided in the mobile device.
According to clause 10,

The at least one processor,

An electronic device characterized in that a neural network model for estimating a location using the wireless signal is trained using the obtained second map.