KR20220030037A - Method for determining an azimuth angle using image and an electronic device supporting the same - Google Patents

Abstract

An electronic device according to one embodiment disclosed in the present document includes a camera module, a memory, and a processor, wherein the processor photographs a plurality of first images at a first angular interval, recognizes a specified type of objects in each of the plurality of first images, combines the numbers or proportions associated with the recognized objects to create a first arrangement, obtains location information where the plurality of first images have been photographed, obtains a plurality of second arrangements based on the location information, and may determine an azimuth corresponding to an arrangement matching the first arrangement among the plurality of second arrangements as a candidate group with respect to the direction of the electronic device. In addition to this, various embodiments identified through the specification are possible.

Description

Method for determining an azimuth angle using image and an electronic device supporting the same}

Various embodiments disclosed in this document relate to a method of determining an azimuth using an image and an electronic device supporting the same.

An electronic device such as a smartphone or a tablet PC may capture an image using a camera. The electronic device may provide location information or an azimuth based on an image captured using a camera, so that the user can easily recognize the current location or support the user to quickly move to a desired location.

When the electronic device provides an azimuth using an image, the electronic device determines the azimuth by comparing the feature points of the captured image with the feature points of the corresponding image on the 3D map. In this case, the amount of computation increases as all of the plurality of images are compared, and it may be difficult to detect the azimuth in real time.

An electronic device according to various embodiments of the present disclosure includes a camera module, a memory, and a processor, wherein the processor captures a plurality of first images at a first angular interval, and a specified type of object in each of the plurality of first images recognizes, generates a first array by combining the number or ratio related to the recognized object, obtains location information at which the plurality of first images are captured, and forms a plurality of second arrays based on the location information obtained, and an azimuth corresponding to an arrangement matching the first arrangement among the plurality of second arrangements may be determined as a candidate group with respect to the direction of the electronic device.

The electronic device according to various embodiments disclosed in this document may reduce candidate groups for calculating an azimuth by using an arrangement related to an object recognized in a captured image. Through this, the amount of computation for calculating the azimuth may be reduced.

The electronic device according to various embodiments disclosed in this document may reduce the candidate group for calculating the azimuth by comparing the arrangement calculated from the captured image with the arrangement calculated from the virtual image.

The electronic device according to various embodiments of the present disclosure may increase the accuracy of calculating an azimuth by using a margin related to a similarity between arrays or a margin related to an angle between images.

1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure;
2 illustrates a method of calculating an azimuth using an image according to various embodiments of the present disclosure.
3 illustrates a 3D virtual map according to various embodiments.
4 illustrates a comparison of a first arrangement and a second arrangement according to various embodiments.
5 illustrates a comparison with a first arrangement through conversion of an azimuth database according to various embodiments of the present disclosure;
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. However, it is not intended to limit the technology described in this document to specific embodiments, and it should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of this document. . In connection with the description of the drawings, like reference numerals may be used for like components.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 may be included. In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120 . It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be stored in the volatile memory 132 , and may process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 may use less power than the main processor 121 or may be set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The auxiliary processor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the co-processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which artificial intelligence is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176 ). The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used in a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 , or an external electronic device (eg, a sound output module 155 ) connected directly or wirelessly with the electronic device 101 . A sound may be output through the electronic device 102 (eg, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 includes various technologies for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to an embodiment, the wireless communication module 192 may include a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 is to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 illustrates a method of calculating an azimuth using an image according to various embodiments of the present disclosure.

Operations 210 to 250 according to various embodiments may be performed by at least one component (eg, the processor 120 of FIG. 1 ) of the electronic device 101 of FIG. 1 .

Referring to FIG. 2 , in operation 210 , the processor 120 uses a camera module (or a camera, a camera device, and a photographing device) (eg, the camera module 180 of FIG. 1 ) to a first angular interval (eg: A plurality of first images may be photographed at an angle of about 30 degrees).

For example, when a map application or a virtual reality application is executed, the processor 120 may output a message guiding photo taking in a manner specified by the user (eg, press the capture button on the screen and rotate 360 degrees). . The processor 120 may detect the rotation of the electronic device 101 using a sensor module (eg, the sensor module 176 of FIG. 1 ). The processor 120 may store a plurality of first images in a memory (eg, the memory 130 of FIG. 1 ) by taking pictures at first angular intervals (eg, about 30 degrees).

In operation 220, the processor 120 may recognize an object (eg, a building) of a specified type from each of the plurality of first images. For example, the processor 120 may recognize and classify objects (eg, sky, buildings, or roads) using various object recognition algorithms (eg, object detection or instance segmentation). The processor 120 may be used to calculate an azimuth by using one type (eg, a building) among the recognized objects.

In operation 230 , the processor 120 may generate the first arrangement by combining a number or a ratio related to the recognized object. For example, the processor 120 may generate the first arrangement by extracting the number of buildings from each of the plurality of first images. As another example, the processor 120 may generate a first arrangement corresponding to a distance from a first point where the building starts to a second point where the building ends in each of the plurality of first images. As another example, the processor 120 may generate a first arrangement corresponding to the angle between the edge of the first point where the building starts and the edge of the second point where the building ends in each of the plurality of first images. there is.

As another example, the processor 120 may calculate a ratio of pixels occupied by the sky in each of the plurality of first images, and generate the first array using the ratio of pixels occupied by the sky to all pixels. .

Hereinafter, a case in which the first arrangement is generated by extracting the number of buildings will be mainly discussed, but the present invention is not limited thereto.

According to various embodiments, the number of elements in the first arrangement may be determined according to an angular interval at which the plurality of first images are captured. For example, in the case of 12 images captured at intervals of 30 degrees, the processor 120 may generate a 1 by 12 first arrangement. As another example, in the case of 24 images captured at intervals of 15 degrees, the processor 120 may generate a 1 by 24 first arrangement.

In operation 235, the processor 120 may acquire location information at which the plurality of first images are captured.

According to various embodiments, the processor 120 obtains location information (eg, latitude and longitude) in which a plurality of first images are captured using a wireless communication circuit (eg, the wireless communication circuit 192 of FIG. 1 ). can The wireless communication module 192 provides location information from a base station included in a global positioning system (GNSS) (eg, global positioning system) and/or a cellular network (eg, the second network 199 of FIG. 1 ). can receive

According to various embodiments, operation 235 may be performed before operation 210 or may be performed together with operation 210 .

In operation 240 , the processor 120 may acquire a plurality of second arrangements based on location information at which the plurality of first images are captured. Each of the plurality of second arrangements may correspond to one azimuth.

According to various embodiments, the processor 120 may extract a plurality of second arrays using a 3D virtual map. 3D virtual maps are rendered based on elevation data (e.g. height above sea level), and/or textual information such as the site shape (or width/length) or height of the building, rather than an actual photographed image. It may be a generated map (see FIG. 3 ).

According to various embodiments, the processor 120 may store the number of observable virtual buildings in a database (hereinafter, azimuth database) for each second angular interval (eg, about 5 degrees) for each point of the 3D virtual map. . For example, the processor 120 acquires 72 virtual images at intervals of 5 degrees for each point of the 3D virtual map, calculates the number of buildings included in each virtual image, and stores an array of 1 by 72 can

According to various embodiments, the processor 120 sets a virtual camera having the same specification as that of the camera module 180 of the electronic device 101 at each point of the 3D virtual map, and sets the virtual camera at a second angular interval (eg: It is possible to acquire virtual images while rotating 360 degrees (about 5 degrees).

According to various embodiments, the processor 120 may extract a plurality of second arrays from a point (hereinafter, a virtual point) on a 3D virtual map corresponding to location information at which a plurality of first images are captured using an azimuth database. can

In operation 250 , the processor 120 may store an azimuth corresponding to an arrangement matching the first arrangement among the plurality of second arrangements (hereinafter, referred to as a matching arrangement) as a candidate group for determining the direction of the electronic device 101 . there is.

According to an embodiment, the processor 120 may determine the second arrangement having the highest similarity score with the first arrangement as the matching arrangement.

According to another embodiment, the processor 120 may determine the second arrangement having the lowest difference score from the first arrangement as the matching arrangement.

According to various embodiments, when a plurality of candidate azimuth angles are included in the candidate group, the processor 120 compares the feature points of the first image captured through the camera module 180 and the virtual image obtained from the 3D virtual map, One azimuth can be determined.

According to various embodiments, when the azimuth of the electronic device 101 is determined, the processor 120 may update the 3D virtual map based on the data of the first arrangement.

According to various embodiments, when the azimuth of the electronic device 101 is determined, the processor 120 uses a sensor module (eg, the sensor module 176 of FIG. 1 ) (eg, an accelerometer, a gyroscope, and/or a magnetometer). The azimuth can be updated in real time.

For example, if the array of the number of buildings of 1 by 72 in the virtual point is [1 1 2 2 2 2 2 2 2 2 2 2 2 1 1 1 ... 1 1 1 1], the number of recognized buildings is 1 to 2 , the processor 120 may determine that the azimuth is changed by 10 degrees, and may calibrate a sensing value of the sensor module for the 10 degree change.

According to various embodiments, at least some operations of the processor 120 of FIG. 2 may be performed through a server (eg, the server 108 of FIG. 1 ). For example, the processor 120 may transmit the location information to the server 108 and receive from the server 108 a plurality of second arrangements corresponding to the location information.

3 illustrates a 3D virtual map according to various embodiments.

Referring to FIG. 3 , the processor 120 or the server 108 of the electronic device 101 may generate a 3D virtual map 301 . The 3D virtual map 301 may be a map generated by being rendered based on text information such as elevation data (eg, the highest height above sea level) and/or artificial structure data rather than an actual photographed image.

According to various embodiments, the 3D virtual map 301 may not include information about detailed shapes or colors of facilities and/or buildings. For example, all the buildings included in the 3D virtual map 301 have the same type of figure (eg, a rectangular parallelepiped), a shape similar to the shape of the site, or a simplified shape of facilities and/or buildings, but text information may be rendered to have different sizes and/or heights based on the .

According to an embodiment, the altitude data may be text including height information for a specific point on the ground. The altitude data may be a numerical model applied to one coordinate system.

According to an embodiment, the artificial structure data may be text including information about a site area or height of a building or facility. The artificial structure data may be managed through a separate server, and the processor 120 may receive and store the artificial structure data by requesting the server (eg, the server 108 or an additional server (not shown)). For example, man-made structure data may include information about the location of infrastructure (eg, roads, parks, and/or railroads) and/or buildings (eg, addresses), site information (eg, area, shape), and height information (eg, number of floors of the building).

According to various embodiments, the processor 120 or the server 108 places a virtual camera at each point of the 3D virtual map 301 to display a plurality of second images (eg, 5 degrees) at a second angular interval (eg, 5 degrees). Hereinafter, a plurality of virtual images) may be acquired.

According to various embodiments, the processor 120 or the server 108 calculates the number of virtual buildings included in each of a plurality of virtual images, and generates a one-dimensional array (hereinafter, raw data) regarding the number of virtual buildings. can be saved.

4 illustrates a comparison of a first arrangement and a second arrangement according to various embodiments. 4 is illustrative and not limited thereto.

Referring to FIG. 4 , the processor 120 may acquire a plurality of first images (hereinafter, a plurality of photographed images) 410 at a first angular interval by using the camera module 180 . For example, the processor 120 may acquire 12 photographed images 410 scanned by 360 degrees at intervals of about 30 degrees.

The processor 120 may generate the first array 415 by using the plurality of captured images 410 . The processor 120 may recognize the number of buildings in each of the plurality of photographed images 410 . The processor 120 may calculate the number of buildings in each captured image using an object recognition algorithm (eg, object detection or instance segmentation). For example, the processor 120 may generate a 1 by 12 first array 415 by combining the number of buildings calculated from each of the 12 photographed images 410 .

According to various embodiments, the processor 120 may store the azimuth database 420 in the memory 130 . The azimuth database 420 may include a plurality of second arrangements 421a ... 425a .... The plurality of second arrangements 421a ... 425a ... may correspond to different azimuth angles. For example, the first directional arrangement 421a may correspond to an azimuth angle of 0 degrees. The fifth direction arrangement 425a may correspond to an azimuth angle of 120 degrees.

According to an embodiment, the plurality of second arrays 421a ... 425a ... are virtual images 421 obtained at a virtual point corresponding to location information at which the plurality of photographed images 410 are photographed. .. 425 ...) can be extracted.

For example, the azimuth database 420 may include first to twelfth directions corresponding to an angular interval of 30 degrees. The first directional arrangement 421a may correspond to an azimuth angle of 0 degrees. The first direction arrangement 421a may have 12 elements corresponding to the first arrangement 415 . Each element may represent the number of observable virtual buildings in each of the virtual images 421 ... 425 ... facing different directions. The arrangement in the second direction to the arrangement in the fourth direction (not shown) may correspond to azimuth angles of 30 degrees, 60 degrees, and 90 degrees, respectively. The fifth direction arrangement 425a may correspond to an azimuth angle of 120 degrees. The fifth direction arrangement 425a may have 12 elements corresponding to the first arrangement 415 . Each element may represent the number of observable virtual buildings in each of the virtual images 425 facing different directions. The sixth to twelfth directions (not shown) may correspond to azimuth angles of 150 degrees, 180 degrees, 210 degrees, 240 degrees, 270 degrees, 300 degrees, and 330 degrees, respectively.

According to various embodiments, the processor 120 may compare the first array 415 with the arrays in the first to twelfth directions included in the azimuth database 420 . For example, the processor 120 may set an azimuth angle of 120 degrees corresponding to the fifth directional arrangement 425 having the highest similarity score (or the lowest inter-array difference score) as a candidate for determining the azimuth of the electronic device 101 . You can save it to a group.

According to various embodiments, when a plurality of candidate azimuths are included in the candidate group for determining the azimuth of the electronic device 101 , the processor 120 selects a feature point (or a feature vector) between the physically captured image and the virtual image. By comparison, one azimuth can be determined.

According to various embodiments, when the angular spacing of the plurality of second arrays 421a ... 425a ... is smaller than that of the first array 415 in the azimuth database 420 , the processor 120 ) may be compared with the first array 415 by extracting some data from among the plurality of second arrays 421a ... 425a ... so that the angular intervals match.

5 illustrates a comparison with a first arrangement through conversion of an azimuth database according to various embodiments of the present disclosure; In FIG. 5 , a case in which the first angular interval is 30 degrees and the second angular interval is 5 degrees is exemplarily illustrated, but the present invention is not limited thereto.

Referring to FIG. 5 , the processor 120 may generate the first array 510 by using a plurality of captured images (eg, 12 captured images). Each element of the first array 510 may indicate the number of buildings recognized in images taken in different directions at a first angular interval (eg, 30 degrees).

According to various embodiments, the processor 120 may store raw data 505 extracted from a virtual point corresponding to location information at which a plurality of photographed images are photographed in an azimuth database. The processor 120 may convert the raw data 505 into matrix data 520 corresponding to the first array 510 .

For example, the raw data 505 may be an array E1 to E72 storing elements corresponding to each of 72 virtual images acquired at an angular interval of 5 degrees. Each element may represent the number of observable virtual buildings in the corresponding virtual image.

When the first array 510 is extracted from images captured at an angular interval of 30 degrees, the processor 120 may convert the raw data 505 into 12 by 6 matrix data 520 . The processor 120 may convert the first to sixth elements E1 to E6 of the raw data 505 into a first row R1 of the matrix data 520 . The processor 120 may convert the seventh to twelfth elements of the raw data 505 into the second row R2 of the matrix data 520 . In a similar manner, the third to eleventh rows of the matrix data 520 may be transformed. The processor 120 may convert the 67th to 72nd elements E67 to E72 into elements of the twelfth row R12 of the matrix data 520 that is the last row.

In the matrix data 520 , the first row R1 may maintain an angular distance of 30 degrees from the second row R2 . The first column C1 may maintain an angular distance of 5 degrees from the second column C2. For example, the first column C1 may correspond to an azimuth angle of 0 degrees, and the second column C2 may correspond to an azimuth angle of 5 degrees. The sixth column C6 may correspond to an azimuth angle of 25 degrees.

According to various embodiments, the processor 120 may cyclic shift the first array 510 to compare it with all combinations in each column of the matrix data 520 . For example, the processor 120 may compare the first array 510 with each column while shifting 12 cycles. The processor 120 may store, as a candidate group, an azimuth corresponding to a column having the lowest similarity difference score compared within all columns R1 to R6.

According to various embodiments, the processor 120 may set a first margin related to a difference score between the first arrangement and the second arrangement. For example, when the first margin is 0, the processor 120 may store an azimuth corresponding to an array having a minimum similarity difference score as a candidate group. As another example, when the first margin is 1, the azimuth angle candidate group corresponding to the array having the minimum score+1 may be stored. As another example, when the first margin is 2, the azimuth angle candidate group corresponding to the array having the minimum score+2 score may be stored.

As the first margin increases, the candidate azimuth angles included in the candidate group may increase, and the accuracy of the azimuth angle calculation may increase. On the other hand, the amount of calculation or the calculation time of the azimuth angle calculation may be increased.

According to various embodiments, the processor 120 may set a second margin (or guard) for the angle. For example, if the second margin is 1 and the difference score is the minimum score of 120 degrees is determined as the candidate azimuth, the processor 120 selects 115 degrees and 125 degrees with an angular difference of a specified angular interval (eg, 5 degrees) as candidates. It can be saved as an azimuth. As another example, if the second margin is 2 and the difference score is the minimum score of 120 degrees is determined as the candidate azimuth, the processor 120 has an angular difference within twice the specified angular interval (eg, 5 degrees). 110 degrees, 115 degrees, 120 degrees, 125 degrees or 130 degrees can be stored as candidate azimuths.

As the second margin increases, the candidate azimuth angles included in the candidate group may increase, and the accuracy of the azimuth angle calculation may increase. On the other hand, the amount of calculation or the calculation time of the azimuth angle calculation may be increased.

According to various embodiments, the processor 120 may determine the candidate azimuth by combining the first margin and the second margin. When the first margin is 0, the processor 120 may determine the candidate azimuth by using the second margin as a first reference value (eg, +1). When the first margin is not 0, the processor 120 may determine the candidate azimuth by using the second margin as a second reference value (eg, 0). The second reference value may be smaller than the first reference value.

According to various embodiments, the processor 120 may generate a first arrangement by acquiring a plurality of captured images through a scan in a range smaller than 360 degrees. For example, the processor 120 may scan 240 degrees to acquire 8 photographed images at intervals of 30 degrees, and generate a first array having 8 elements. In this case, according to comparison with the second arrangement, the number of candidate azimuth angles may be increased compared to the case of the 360 degree scan. The processor 120 may change the first margin or the second margin so that the number of candidate azimuths is calculated to be less than or equal to a specified number (eg, five).

The electronic device (eg, the electronic device 101 of FIG. 1 ) according to various embodiments includes a camera module, a memory (eg, the memory 130 of FIG. 1 ), and a processor (eg, the processor 120 of FIG. 1 ) and, the processor (eg, the processor 120 of FIG. 1 ) captures a plurality of first images at a first angular interval, recognizes a specified type of object in each of the plurality of first images, and recognizes the A first arrangement is generated by combining a number or a ratio related to the object, obtains location information at which the plurality of first images are captured, obtains a plurality of second arrangements based on the location information, and the plurality of An azimuth corresponding to an arrangement matching the first arrangement among the second arrangements may be determined as a candidate group with respect to a direction of the electronic device (eg, the electronic device 101 of FIG. 1 ).

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 ) determines a virtual point of the 3D virtual map corresponding to the location information, and uses a plurality of second images obtained from the virtual point. Thus, the plurality of second sequences may be extracted.

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 ) may combine the number or ratio of virtual objects corresponding to the objects calculated from each of the plurality of second images to combine the plurality of second images. Arrays can be extracted.

According to various embodiments, the plurality of second images may correspond to the first angular interval.

According to various embodiments, the plurality of second images may correspond to a second angular interval smaller than the first angular interval.

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 ) may convert the plurality of second arrays to correspond to the first angular interval to match the first array.

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 ) may convert the plurality of second arrays into a matrix form to correspond to the first angular interval.

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 ) may cycle shift the first array to match the plurality of second arrays.

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 ) transmits the location information to an external server (eg, the server 108 of FIG. 1 ), and the external server (eg, the processor 120 of FIG. 1 ) The plurality of second arrangements corresponding to the location information may be received from the server 108 .

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 ) may determine an azimuth of one of the candidate groups by using at least one feature point among the plurality of first images.

According to various embodiments, the object is a building, and the processor (eg, the processor 120 of FIG. 1 ) generates the first arrangement based on the number of buildings included in each of the plurality of first images. can do.

According to various embodiments, the object is a building, and the processor (eg, the processor 120 of FIG. 1 ) calculates the distance from the first point to the second point of the building in each of the plurality of first images. Based on this, the first array may be created.

According to various embodiments, the object is a building, and the processor (eg, the processor 120 of FIG. 1 ) may display an edge of a first point and an edge of a second point of the building in each of the plurality of first images. The first arrangement may be generated based on the angle between them.

According to various embodiments, the object is the sky, and the processor (eg, the processor 120 of FIG. 1 ) selects the first arrangement based on a ratio of pixels occupied by the sky in each of the plurality of first images. can create

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 ) may generate a first margin related to a similarity score between the first array and the plurality of second arrays, or a second margin related to an angle. Based on the matching arrangement, it is possible to determine the matching arrangement.

The azimuth angle calculation method according to various embodiments is performed in an electronic device (eg, the electronic device 101 of FIG. 1 ), photographing a plurality of first images at a first angular interval, each of the plurality of first images Recognizing an object of a specified type in an operation of obtaining a plurality of second arrays based on an azimuth angle corresponding to an arrangement matching the first arrangement among the plurality of second arrangements of the electronic device (eg, the electronic device 101 of FIG. 1 ) Can be determined as a group of candidates for direction.

According to various embodiments, the acquiring of the plurality of second arrays includes: determining a virtual point of the 3D virtual map corresponding to the location information; acquiring a plurality of second images from the virtual point; and extracting the plurality of second arrays by using the plurality of second images.

According to various embodiments, the object is a building, and the operation of generating the first arrangement may include generating the first arrangement based on the number of buildings included in each of the plurality of first images. can

According to various embodiments, the method may further include determining an azimuth of one of the candidate groups by using at least one feature point among the plurality of first images.

According to various embodiments, the object is the sky, and the operation of generating the first arrangement includes generating the first arrangement based on a ratio of pixels occupied by the sky in each of the plurality of first images. may include

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the element from other elements in question, and may refer to elements in other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively". When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present document, one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101) may be implemented as software (eg, the program 140) including For example, a processor (eg, processor 120 ) of a device (eg, electronic device 101 ) may call at least one command among one or more commands stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store™) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly between smartphones (eg: smartphones) and online. In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. or one or more other operations may be added.

Claims (20)

In an electronic device,
camera module;
Memory; and
processor; including;
The processor is
Taking a plurality of first images at a first angular interval,
Recognizing a specified type of object in each of the plurality of first images,
combining the numbers or proportions associated with the recognized objects to create a first array,
Obtaining location information at which the plurality of first images were taken,
obtaining a plurality of second arrangements based on the location information;
and an azimuth corresponding to an arrangement matching the first arrangement among the plurality of second arrangements as a candidate group with respect to a direction of the electronic device. The method of claim 1, wherein the processor is
determining a virtual point on the 3D virtual map corresponding to the location information;
The electronic device extracts the plurality of second arrays by using the plurality of second images acquired at the virtual point. 3. The method of claim 2, wherein the processor
The electronic device extracts the plurality of second arrays by combining the number or ratio of virtual objects corresponding to the objects calculated from each of the plurality of second images. The method of claim 2, wherein the plurality of second images are
An electronic device corresponding to the first angular interval. The method of claim 2, wherein the plurality of second images are
An electronic device corresponding to a second angular interval smaller than the first angular interval. 6. The method of claim 5, wherein the processor
An electronic device for matching the first array by converting the plurality of second arrays to correspond to the first angular interval. 6. The method of claim 5, wherein the processor
The electronic device converts the plurality of second arrays into a matrix form to correspond to the first angular interval. The method of claim 1, wherein the processor is
An electronic device matching the plurality of second arrangements by cycle-shifting the first arrangement. The method of claim 1, wherein the processor is
transmit the location information to an external server;
The electronic device receives the plurality of second arrangements corresponding to the location information from the external server. The method of claim 1, wherein the processor is
The electronic device determines an azimuth of one of the candidate groups by using at least one feature point among the plurality of first images. According to claim 1, wherein the object is a building,
The processor is
An electronic device generating the first arrangement based on the number of buildings included in each of the plurality of first images. According to claim 1, wherein the object is a building,
The processor is
An electronic device to generate the first arrangement based on a distance from a first point to a second point of the building in each of the plurality of first images. According to claim 1, wherein the object is a building,
The processor is
An electronic device for generating the first arrangement based on an angle between an edge of a first point and an edge of a second point of the building in each of the plurality of first images. According to claim 1, wherein the object is the sky,
the processor is
The electronic device for generating the first arrangement based on a ratio of pixels occupied by the sky in each of the plurality of first images. The method of claim 1, wherein the processor is
An electronic device to determine the matching arrangement based on a first margin related to a similarity score between the first arrangement and the plurality of second arrangements, or a second margin related to an angle. The azimuth angle calculation method performed in the electronic device is
photographing a plurality of first images at first angular intervals;
recognizing a specified type of object in each of the plurality of first images;
generating a first arrangement by combining a number or a ratio associated with the recognized object;
acquiring location information at which the plurality of first images were captured;
acquiring a plurality of second arrays based on the location information; and
A method of determining an azimuth corresponding to an arrangement matching the first arrangement among the plurality of second arrangements as a candidate group for a direction of the electronic device. The method of claim 16 , wherein obtaining the plurality of second arrangements comprises:
determining a virtual point on the 3D virtual map corresponding to the location information;
acquiring a plurality of second images at the virtual point; and
and extracting the plurality of second arrays by using the plurality of second images. 17. The method of claim 16, wherein the object is a building,
The operation of creating the first array is
and generating the first arrangement based on the number of buildings included in each of the plurality of first images. 17. The method of claim 16,
and determining an azimuth of one of the candidate groups by using at least one feature point among the plurality of first images. 17. The method of claim 16, wherein the object is the sky,
The operation of creating the first array is
and generating the first array based on a ratio of pixels occupied by the sky in each of the plurality of first images.

Images