WO2023146092A1

WO2023146092A1 - Image-based inertial sensor correction method and electronic device for performing same

Info

Publication number: WO2023146092A1
Application number: PCT/KR2022/018601
Authority: WO
Inventors: 서영준
Original assignee: 삼성전자주식회사
Priority date: 2022-01-25
Filing date: 2022-11-23
Publication date: 2023-08-03

Abstract

Disclosed are an image-based inertial sensor correction method and an electronic device for performing same. The electronic device comprises: a camera for acquiring an image; an inertial sensor for sensing movements of the electronic device; a position sensor for sensing the position of the electronic device at the time the image is acquired; and a processor for controlling the camera, the inertial sensor, and the position sensor. The processor may: detect a first position value of a target object from a first image acquired at a first time; estimate the first position value of the target object on the basis of the position value measured by the position sensor at the first time; and determine posture correction data for correcting the posture of the electronic device determined by means of the inertial sensor, the data being determined on the basis of a first difference value between the detected first position value and the estimated first position value.

Description

Image-based inertial sensor calibration method and electronic device performing the same

The embodiments below relate to image-based inertial sensor calibration techniques.

An inertial sensor including at least one of an accelerometer, a gyroscope, and a geomagnetic field may be used together with a location sensor such as a global positioning system (GPS) to estimate position, posture, and direction information of the electronic device.

Sensors included in the electronic device may generate errors depending on the surrounding environment, and inertial sensors may generate accumulated errors over time when tracking the posture of the electronic device.

An image-based inertial sensor calibration method and an electronic device performing the method according to an embodiment may calibrate a posture determined based on an output of an inertial sensor and an inertial sensor using information included in an image.

An electronic device according to an embodiment includes a camera acquiring an image, an inertial sensor detecting motion of the electronic device, a position sensor detecting a position of the electronic device when acquiring the image, and the camera, the inertial sensor, and the A processor controlling the position sensor may be included. The processor detects a first position value of a target object in a first image obtained at a first time, and the first position value of the target object is detected based on the position value measured by the position sensor at the first time. 1 position value can be estimated. The processor may determine posture correction data for correcting the posture of the electronic device determined through the inertial sensor based on a first difference between the detected first position value and the estimated first position value. .

An electronic device according to an embodiment includes a camera acquiring an image, an inertial sensor detecting motion of the electronic device, a position sensor detecting a position of the electronic device when acquiring the image, and the camera, the inertial sensor, and the A processor controlling the position sensor may be included. The processor detects a first position value of a target object from a first image obtained at a first time, and the first position value of the target object based on the position value measured by the position sensor at the first time. can be estimated. The processor detects a second position value of a target object from a second image obtained at a second time, and the second position value of the target object is detected based on the position value measured by the position sensor at the second time. 2 position values can be estimated. The processor performs the processing based on a first difference value between the detected first position value and the estimated first position value and a second difference value between the detected second position value and the estimated second position value. Inertial sensor calibration data for calibrating the inertial sensor may be determined.

A method for calibrating an inertial sensor of an electronic device according to an embodiment includes an operation of detecting a first position value of a target object from a first image obtained at a first time using a camera of the electronic device, An operation of estimating a first position value of the target object based on a position value measured by a position sensor of the electronic device at 1 hour, and a difference between the detected first position value and the estimated first position value An operation of determining posture correction data for correcting the posture of the electronic device determined through the inertial sensor based on the first difference value.

1 is a block diagram of an electronic device in a network environment according to various embodiments.

2 is a diagram for explaining an outline of a method of correcting a posture of an electronic device according to an exemplary embodiment.

3 is a flowchart of a method of correcting a posture of an electronic device according to an exemplary embodiment.

4 is a diagram for explaining a position value of a target object detected from an image acquired through a camera of an electronic device according to an embodiment.

5 is a diagram for explaining an overview of a method of calibrating an inertial sensor of an electronic device according to an exemplary embodiment.

6 is a flowchart of a method of calibrating an inertial sensor of an electronic device according to an embodiment.

7 is a diagram for explaining position values of a target object detected from images obtained through a camera of an electronic device according to an exemplary embodiment.

8 is a diagram for explaining a data flow of an electronic device according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, a legacy communication module). It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 is a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external

electronic devices

102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

A location sensor such as an inertial measurement unit (IMU) including at least one of an accelerometer, a gyroscope, and a geomagnetic field, and a global positioning system (GPS) to estimate the position, posture, and orientation information of the electronic device 101. can be used together. For example, inertial sensors and position sensors may be included in sensor module 176 of FIG. 1 .

The inertial sensor may determine the attitude of the electronic device 101 using at least one of an accelerometer, a gyroscope, and a geomagnetic field. The attitude of the electronic device 101 may include information about the roll, pitch, and yaw of the electronic device 101 .

An error may occur in the inertial sensor included in the electronic device 101 depending on the surrounding environment. Accumulated errors may occur over time when the inertial sensor tracks the posture of the electronic device 101 . When the electronic device 101 estimates the attitude of the electronic device 101 using the inertial sensor of the electronic device 101, there may be an error in the estimated attitude due to the error of the inertial sensor, and more accurate results may be obtained. correction may be required.

In one embodiment, the electronic device 101 corrects the posture of the electronic device 101 using the image 205 acquired through the camera (eg, the camera module 180 of FIG. 1 ) of the electronic device 101 . can do. Referring to FIG. 2 , a process of correcting a posture of the electronic device 101 using an image 205 acquired through a camera is briefly illustrated. The electronic device 101 may detect the target object 220 in the image 205 and correct the posture of the electronic device 101 based on the detected location of the target object 220 .

In one embodiment, the electronic device 101 includes a camera for acquiring the image 205, an inertial sensor for detecting the motion of the electronic device 101, and a location for detecting the position of the electronic device 101 when the image 205 is acquired. May contain sensors.

The electronic device 101 may obtain an image 205 using a camera. The electronic device 101 may determine the posture of the electronic device 101 using a sensor of the electronic device 101 . In FIG. 2 , a direction 210 toward which the electronic device 101 is headed determined by using a sensor of the electronic device 101 on an image 205 obtained through a camera and an actual direction 215 toward which the electronic device 101 is headed this is shown The direction 210 toward which the electronic device 101 faces may be a direction corresponding to the posture of the electronic device 101 determined using a sensor, and is the actual direction 215 toward which the electronic device 101 faces. may be a direction corresponding to the actual posture of the electronic device 101 . Due to the error of the sensor, there may be an error in the attitude of the electronic device 101 determined using the sensor.

In one embodiment, the electronic device 101 may detect the target object 220 in the image 205 . The target object 220 may be, for example, a celestial body. The electronic device 101 detects the position value of the target object 220 from the image 205 and uses a position sensor such as GPS to acquire the position value of the target object 220 at the time when the corresponding image 205 is acquired. can be estimated. For example, the detected position value and the estimated position value of the target object 220 may be a position value on the horizontal coordinate system 225 including an altitude value and an azimuth value.

In an embodiment, the electronic device 101 may determine posture correction data for correcting the posture of the electronic device 101 based on a difference value between the detected position value and the estimated position value of the corresponding target object 220 . there is. The electronic device 101 may update the posture correction table based on the determined posture correction data. The posture correction table may be a comparison table capable of correcting a difference between the posture of the electronic device 101 determined through the sensor and the actual posture of the electronic device 101 . The electronic device 101 may correct the posture of the electronic device 101 using the posture correction table, and may determine a posture corresponding to the actual direction 215 toward which the electronic device 101 faces.

In an embodiment, the electronic device 101 corrects the posture of the electronic device 101 using the image 205 obtained through the camera of the electronic device 101, thereby correcting the posture of the electronic device 101 at a low cost. An accurate posture can be obtained by correcting, and the accuracy of the posture can be increased by repeatedly correcting using several images 205 acquired at different times. The electronic device 101 may automatically acquire the image 205 without user input to determine posture correction data, thereby providing the user with an accurate posture even if the user is not aware of the posture correction process.

Hereinafter, a posture correction method of the electronic device 101 will be described with reference to FIG. 3 .

Referring to FIG. 3 , an electronic device 101 including a camera acquiring an image, an inertial sensor detecting motion of the electronic device 101, and a position sensor detecting the position of the electronic device 101 when acquiring an image is electronic A flowchart of a method of correcting the attitude of the electronic device 101 determined by using the inertial sensor of the device 101 is shown.

In operation 305, the electronic device 101 may detect a target object from a first image obtained through a camera at a first time. For example, the target object may be a celestial body such as the sun, moon, or satellite.

In operation 310, the electronic device 101 may detect a first position value of the target object in the first image. The detected first position value of the target object may be determined based on the angle of view of the camera and the central point of the image. For example, the electronic device 101 may detect the first position value of the target object from the center point of the image by using information about the angle of view of the camera. The first location value may include an altitude value and a direction value.

For example, referring to FIG. 4 , a target object 405 detected in an image 400 obtained through a camera of the electronic device 101 is illustrated.

The electronic device 101 may detect the first position value of the target object 405 from the center of the image 400 using the angle of view of the camera. For example, in the example of FIG. 4 , the electronic device 101 uses the angle of view of the camera to a target object 405 having an elevation value 410 of 5.2 degrees and an azimuth value 415 of 20.3 degrees from the center of the image 400. It is possible to detect the first position value for

Referring back to FIG. 3 , in operation 315, the electronic device 101 determines the location of the electronic device 101 at the first time based on the location of the electronic device 101 measured at the first time using the position sensor of the electronic device 101. A first position value of the target object may be estimated. The estimated first position value may be a position value on a horizontal coordinate system having the position of the electronic device 101 as an origin. For example, when the electronic device 101 can know the time at which the first image was captured and the location at which the first image was captured, the electronic device 101 determines the actual location of the target object based on the corresponding time and location. A first position value corresponding to may be estimated.

In operation 320, the electronic device 101 may determine a first difference value between the first position value of the target object detected in operation 310 and the first position value estimated in operation 315. Based on the first difference value, the electronic device 101 may determine attitude correction data for correcting the attitude of the electronic device 101 determined using the inertial sensor.

The first difference value may be a value including an altitude value and a direction value. The electronic device 101 may convert the first difference value into a value on the Cartesian coordinate system.

In an embodiment, the electronic device 101 may convert the first difference value converted into a value on the Cartesian coordinate system into a value on the world coordinate system based on the attitude of the electronic device 101 determined through the inertial sensor at the first time. there is. The world coordinate system is a single geocentric coordinate system and may be an absolute coordinate system for displaying a location with the center of mass (geometric center) of the earth as the origin. The world coordinate system may be a Cartesian coordinate system. In an embodiment, the attitude of the electronic device 101 determined based on the output of the inertial sensor may be determined on a world coordinate system. The attitude of the electronic device 101 determined based on the output of the inertial sensor may be information indicating the degree of rotation of the electronic device 101 with respect to each axis of the world coordinate system.

Since the posture of the electronic device 101 indicates the degree of rotation of the electronic device 101 with respect to each axis of the world coordinate system, the electronic device 101 uses the posture and the first difference value converted to a value on the Cartesian coordinate system. can be converted to a value on the world coordinate system. The electronic device 101 may convert the first difference value converted into a value on the Cartesian coordinate system into a value on the world coordinate system using Equations 1, 2, and 3, for example.

In Equation 1, Equation 2, and Equation 3, x, y, and z are values of the x-axis, y-axis, and z-axis of the first difference value converted to a Cartesian coordinate system,

and

is the value of the x-axis, y-axis and z-axis on the world coordinate system,

and

Means the roll, pitch, and yaw of the electronic device 101 determined using the inertial sensor at the first time, respectively.

In an embodiment, the electronic device 101 may convert the first difference value converted into a value on the world coordinate system into a first reference posture, which is the posture of the electronic device 101 on the world coordinate system. In an embodiment, the electronic device 101 may determine the reference posture of the electronic device 101 on the world coordinate system using Equations 4, 5, and 6.

In Equation 4, Equation 5 and Equation 6,

and

are values of the x-axis, y-axis, and z-axis on the world coordinate system determined in Equations 1, 2, and 3, respectively,

and

is the first reference posture of the electronic device 101 on the world coordinate system.

The difference between the posture of the electronic device 101 determined at the first time using the inertial sensor and the first reference posture may mean an error included in the posture determined using the inertial sensor. The electronic device 101 may determine posture correction data for correcting the posture according to a difference between the posture of the electronic device 101 determined using the inertial sensor and the reference posture.

In operation 325, the electronic device 101 may update the posture correction table based on the determined posture correction data. In one embodiment, the electronic device 101 may correct the posture by applying the posture correction table to the posture determined through the inertial sensor of the electronic device 101 .

Operations

305, 310, 315, 320 and 325 may be periodically and/or iteratively performed using images acquired at different times.

Hereinafter, a method of calibrating an inertial sensor by an electronic device using a plurality of images will be described with reference to FIG. 5 .

Referring to FIG. 5 , a process in which the electronic device 101 corrects the inertial sensor 515 included in the electronic device 101 using images 505 obtained through a camera is briefly illustrated. The electronic device 101 detects the target object 520 from the images 505 and determines the amount of change in the posture of the electronic device 101 based on the position of the target object 520 detected from the images 505. And, the inertial sensor 515 may be calibrated based on the determined amount of change. For example, the inertial sensor 515 may include at least one of an accelerometer, a gyroscope, and a geomagnetic field.

In one embodiment, the electronic device 101 calibrates the inertial sensor 515 using the images 505 acquired through the camera of the electronic device 101, thereby calibrating the inertial sensor 515 at a low cost to obtain accurate accuracy. The posture can be obtained, and the accuracy of the posture can be increased by repeatedly calibrating the inertial sensor 515 using several images 505 acquired at different times. The electronic device 101 may automatically acquire images 505 without user input to determine the inertial sensor 515 calibration data, thereby providing the user with an accurate posture even if the user does not recognize the inertial sensor 515 calibration process. can provide

In one embodiment, the electronic device 101 may detect the target object 520 from the images 505 . The target object 520 may be, for example, a celestial body. The electronic device 101 detects the location values of the target object 520 from the images 505, and uses a location sensor such as GPS to acquire the corresponding images 505 of the target object 520 at the time. Position values can be estimated. For example, the detected location values and the estimated location values of the target object 520 may be location values on a horizontal coordinate system 525 including an altitude value and an azimuth value.

In one embodiment, the electronic device 101 provides inertial sensor 515 calibration data for calibrating the inertial sensor 515 based on difference values between detected position values and estimated position values of the corresponding target object 520 . can decide The electronic device 101 may update the inertial sensor 515 calibration table based on the inertial sensor 515 calibration data. The electronic device 101 may calibrate the inertial sensor 515 using the inertial sensor 515 calibration table and accurately determine the attitude of the electronic device 101 .

Hereinafter, a method of calibrating the inertial sensor of the electronic device 101 will be described with reference to FIG. 6 .

Referring to FIG. 6 , an electronic device 101 including a camera for obtaining an image, an inertial sensor for detecting motion of the electronic device 101, and a position sensor for detecting the position of the electronic device 101 when acquiring an image are electronic devices 101 A flow diagram of a method for calibrating an inertial sensor of device 101 is shown.

In operation 605, the electronic device 101 may detect a target object from a first image acquired through a camera at a first time, and may detect a target object from a second image acquired through a camera at a second time. there is. For example, the target object may be a celestial body such as the sun, moon, or satellite.

In operation 610, the electronic device 101 may detect a first position value of the target object in the first image and a second position value of the target object in the second image. The detected first position value and the detected second position value of the target object may be determined based on the angle of view of the camera and the central point of the image. For example, the electronic device 101 may detect the first position value and the second position value of the target object by using view angle information of the camera from the center point of the image. The first location value and the second location value may include an altitude value and a direction value.

For example, referring to FIG. 7 , a target object 710 detected from images acquired through a camera of the electronic device 101 is illustrated.

The electronic device 101 may detect the first position value and the second position value of the target object 710 from the center of the image using the angle of view of the camera. For example, in the first image of FIG. 7(A), the electronic device 101 uses the angle of view of the camera to set the target object 710 of an altitude value 715 of 5.2 degrees and an azimuth value 720 of 20.3 degrees from the center of the image. ) It is possible to detect the first position value for. In the second image of FIG. 7 (B) , the electronic device 101 provides an altitude value 725 of 7.5 degrees and a direction value 730 of 9.8 degrees from the center of the image using the angle of view of the camera for the target object 710 . 2 position values can be detected.

Referring back to FIG. 6 , in operation 615, the electronic device 101 determines the location of the electronic device 101 at the first time based on the location of the electronic device 101 measured at the first time using the position sensor of the electronic device 101. A first position value of the target object may be estimated. The electronic device 101 may estimate the second position value of the target object at the second time based on the measured position of the electronic device 101 at the second time using the position sensor of the electronic device 101 . The estimated first and second position values may be position values on a horizontal coordinate system having the position of the electronic device 101 as an origin. For example, when the electronic device 101 can know the time at which the first image was captured and the location at which the first image was captured, the electronic device 101 determines the actual location of the target object based on the corresponding time and location. A first position value corresponding to may be estimated. When the electronic device 101 can know the time at which the second image was captured and the location at which the second image was captured, the electronic device 101 determines a second image corresponding to the actual location of the target object based on the corresponding time and corresponding location. 2 position values can be estimated.

In operation 620, the electronic device 101 detects a first difference value between the first position value of the target object detected in operation 610 and the first position value estimated in operation 615, and in operation 610 A second difference value between the calculated second position value and the second position value estimated in operation 615 may be determined.

The first difference value and the second difference value may be values including an altitude value and a direction value. The electronic device 101 may convert the first difference value and the second difference value into values on the Cartesian coordinate system.

In an embodiment, the electronic device 101 may convert the first difference value converted into a value on the Cartesian coordinate system into a value on the world coordinate system based on the posture of the electronic device 101 at the first time. The electronic device 101 may convert the second difference value converted into a value on the Cartesian coordinate system into a value on the world coordinate system based on the posture of the electronic device 101 at the second time.

The world coordinate system is a single geocentric coordinate system and may be an absolute coordinate system for displaying a location with the center of mass (geometric center) of the earth as the origin. The world coordinate system may be a Cartesian coordinate system. In one embodiment, the attitude of the electronic device 101 determined through the inertial sensor may be determined on a world coordinate system. The posture of the electronic device 101 determined through the inertial sensor may be information indicating the degree of rotation of the electronic device 101 with respect to each axis of the world coordinate system.

Since the posture of the electronic device 101 represents the degree of rotation of the electronic device 101 with respect to each axis of the world coordinate system, the electronic device 101 determines the electronic device 101 through the inertial sensor at the first time and the second time ( Using the posture of 101), the first difference value and the second difference value converted to values on the Cartesian coordinate system may be converted to values on the world coordinate system. The electronic device 101 may convert the first difference value converted into a value on the Cartesian coordinate system into a value on the world coordinate system using Equations 1, 2, and 3, for example. Similarly, the electronic device 101 may convert the second difference value converted into a value on the Cartesian coordinate system into a value on the world coordinate system.

In an embodiment, the electronic device 101 may convert the first difference value and the second difference value converted into values on the world coordinate system into a reference posture, which is the posture of the electronic device 101 on the world coordinate system. In an embodiment, the electronic device 101 may determine the first reference posture of the electronic device 101 on the world coordinate system using Equations 4, 5, and 6. Similarly, the electronic device 101 may convert the second difference value converted into a value on the world coordinate system into a second reference posture, which is the posture of the electronic device 101 on the world coordinate system.

The electronic device 101 may determine a change rate of the posture of the electronic device 101 between the first time and the second time based on the amount of change between the first reference posture and the second reference posture and the difference between the first time and the second time. there is.

The electronic device 101 may determine inertial sensor correction data based on the output value of the inertial sensor between the first time and the second time and the determined attitude change speed. For example, the attitude change speed may include information about accelerations and angular velocities of the x-axis, y-axis, and z-axis of the electronic device 101 on the world coordinate system between the first time and the second time. The electronic device 101 calculates the angular velocity of the attitude change rate as the angular velocity of the x-axis, y-axis, and z-axis of the electronic device 101 on the world coordinate system output during the time between the first time and the second time through the gyroscope of the inertial sensor. By comparing with , the inertial sensor calibration data can be determined. For example, the electronic device 101 calculates the acceleration of the attitude change rate as the x-axis, y-axis, and x-axis of the electronic device 101 on the world coordinate system output during the time between the first time and the second time through an accelerometer of an inertial sensor. By comparing the z-axis acceleration, the inertial sensor calibration data can be determined.

In operation 625, the electronic device 101 may update the inertial sensor calibration table based on the determined inertial sensor calibration data. The inertial sensor calibration table may be a comparison table capable of correcting a difference between a raw value before calibration of the inertial sensor and a true value. In one embodiment, the electronic device 101 may calibrate the inertial sensor by applying the inertial sensor calibration table to the inertial sensor of the electronic device 101 .

Operations

605, 610, 615, 620 and 625 may be periodically and/or iteratively performed using images acquired at different times.

In one embodiment, the electronic device 101 may perform all of

operations

305, 310, 315, 320, and 325 of FIG. 3 and

operations

605, 610, 615, 620, and 625 of FIG. .

Referring to FIG. 8 , a process of updating an inertial sensor correction table 850 using values output from an inertial sensor 805, a camera 825, and a position sensor 830 included in the electronic device 101 is shown. has been For example, inertial sensor 805 and position sensor 830 may be included in sensor module 176 of FIG. 1 .

For example, the inertial sensor 805 includes an accelerometer 810, a gyroscope 815 and a magnetometer 820, where the accelerometer 810, the gyroscope 815 and the magnetometer 820 are calibrated prior to calibration.

Raw values

835, 840, and 845 may be output. The electronic device 101 applies the inertial sensor correction table 850 to the

low values

835, 840, and 845 of the accelerometer 810, the gyroscope 815, and the earth magnetometer 820, and corrects the corrected values. The posture 860 of the electronic device 101 may be determined using the position 860 . For example, the inertial sensor correction table 850 includes an accelerometer correction table 851 applied to the accelerometer 810, a gyroscope correction table 853 applied to the gyroscope 815, and an earth magnetometer 820 applied to the inertial sensor correction table 850. A geomagnetic field correction table 855 may be included.

The camera 825 may obtain images 865 and the position sensor 830 may output altitude, longitude, latitude, and time as the position 875 of the electronic device 101 . For example, the camera 825 may acquire a first image at a first time and a second image at a second time. The electronic device 101 detects a target object (eg, the target object 710 of FIG. 7 ) using the first image and the second image acquired by the camera 825, and the first position value 870 of the target object. ) and the second position value 870 can be detected. The electronic device 101 may estimate the first position value 880 and the second position value 880 of the target object at the first and second times using the output of the position sensor 830 .

The electronic device 101 determines a first difference value 885 between the detected first position value 870 and the estimated first position value 880, and determines the detected second position value 870 and the estimated first position value 880. A second difference value 885 between the two position values 880 may be determined. The electronic device 101 converts the first difference value 885 into a value 890 on the world coordinate system using the attitude 860 of the electronic device 101 determined using the inertial sensor 805 at the first time, and , the second difference value 885 may be converted into a value 890 on the world coordinate system using the posture 860 of the electronic device 101 determined using the inertial sensor 805 at the second time.

The electronic device 101 may determine a first reference posture and a second reference posture based on the first difference value 890 and the second difference value 890 converted into values on the world coordinate system. The electronic device 101 may determine a change rate of the posture of the electronic device 101 between the first time and the second time based on the amount of change between the first reference posture and the second reference posture and the difference between the first time and the second time. there is.

The electronic device 101 calculates the change in attitude 860 of the electronic device 101 measured using the inertial sensor 805 between the first time and the second time and the first difference value 890 converted into a value on the world coordinate system. ) and the attitude change rate determined based on the second difference value 890 to determine the inertial sensor correction data 895 .

The electronic device 101 may update the inertial sensor calibration table 850 based on the determined inertial sensor calibration data 895 . In an embodiment, the electronic device 101 may calibrate the inertial sensor 805 by applying the inertial sensor calibration table 850 to the inertial sensor 805 of the electronic device 101 .

The electronic device 101 according to an embodiment includes a camera 825 for obtaining an image, an inertial sensor 805 for detecting a motion of the electronic device 101, and a location for detecting the position of the electronic device 101 when acquiring an image. It includes a processor 120 that controls the sensor 830 and the camera 825, the inertial sensor 805, and the position sensor 830, the processor 120, the first time (time) obtained at the first time (time) The first position value of the target object is detected in 1 image, the first position value of the target object is estimated based on the position value measured by the position sensor 830 at the first time, and the detected first position value and Based on the first difference value between the estimated first position values, posture correction data for correcting the posture of the electronic device 101 determined through the inertial sensor 805 may be determined.

The processor 120 may detect the first position value based on information on the angle of view of the camera 825 and the center of the first image.

The processor 120 converts the first difference value on the horizontal coordinate system into a value on the world coordinate system based on the attitude of the electronic device 101 measured at the first time through the inertial sensor 805, and converts the value on the world coordinate system. A first reference posture may be determined based on the first difference value converted to , and posture correction data may be determined based on a difference between the first reference posture and the posture of the electronic device 101 measured at the first time.

The processor 120 converts the first difference value on the horizontal coordinate system into a value on the Cartesian coordinate system, and converts the first difference value converted into a value on the Cartesian coordinate system based on the posture of the electronic device 101 measured at the first time. It can be converted to a value in the world coordinate system.

The processor 120 may update the posture correction table based on the determined posture correction data.

The processor 120 may correct the posture of the electronic device 101 determined through the inertial sensor 805 based on the updated posture correction table.

The processor 120 detects a second position value of the target object from a second image obtained at a second time, and based on the position value measured by the position sensor 830 at the second time, the target object Estimating the second position value of the inertia based on the first difference value between the detected first position value and the estimated first position value and the second difference value between the detected second position value and the estimated second position value Inertial sensor 805 calibration data for calibrating sensor 805 may be determined.

The processor 120 converts the first difference value on the horizontal coordinate system into a value on the world coordinate system based on the attitude of the electronic device 101 at the first time, and based on the first difference value converted into the value on the world coordinate system. to determine the first reference posture, and based on the posture of the electronic device 101 at the second time, the second difference value on the horizontal coordinate system is converted into a value on the world coordinate system, and the second difference converted into a value on the world coordinate system A second reference posture is determined based on the value, a posture change rate between a first time and a second time is determined based on the first and second reference postures, and an inertial sensor between the first time and the second time ( Correction data of the inertial sensor 805 may be determined based on the output value of 805 and the attitude change rate.

The processor 120 may update the inertial sensor calibration table 850 based on the determined inertial sensor 805 calibration data.

The processor 120 may calibrate the inertial sensor 805 based on the updated inertial sensor calibration table 850 .

The electronic device 101 according to an embodiment includes a camera 825 for obtaining an image, an inertial sensor 805 for detecting a motion of the electronic device 101, and a location for detecting the position of the electronic device 101 when acquiring an image. It includes a processor 120 that controls the sensor 830 and the camera 825, the inertial sensor 805, and the position sensor 830, the processor 120, the first time (time) obtained at the first time (time) Detect the first position value of the target object in 1 image, estimate the first position value of the target object based on the position value measured by the position sensor 830 at the first time, and at the second time (time) The second position value of the target object is detected in the acquired second image, the second position value of the target object is estimated based on the position value measured by the position sensor 830 at a second time, and the detected first position value is determined. Inertial sensor 805 for calibrating the inertial sensor 805 based on a first difference value between the position value and the estimated first position value and a second difference value between the detected second position value and the estimated second position value Calibration data can be determined.

The processor 120 detects a first position value based on the angle of view information of the camera 825 and the center of the first image, and detects a second position value based on the angle of view information of the camera 825 and the center of the second image. can be detected.

The processor 120 converts the first difference value on the horizontal coordinate system into a value on the Cartesian coordinate system, and converts the first difference value converted into a value on the Cartesian coordinate system based on the posture of the electronic device 101 at the first time into a world coordinate system. Converts the second difference value on the horizontal coordinate system into a value on the Cartesian coordinate system, and converts the second difference value converted into a value on the Cartesian coordinate system based on the posture of the electronic device 101 at the second time in the world coordinate system. can be converted to a value of

The processor 120 may update the inertial sensor calibration table 850 based on the determined inertial sensor 805 calibration data and calibrate the inertial sensor 805 based on the updated inertial sensor calibration table 850. there is.

A method for calibrating an inertial sensor 805 according to an embodiment includes an operation of detecting a first position value of a target object from a first image acquired at a first time using a camera of an electronic device, and an electronic device at the first time. An operation of estimating a first position value of the target object based on the position value measured by the position sensor 830 of the device 101, and a first difference value between the detected first position value and the estimated first position value An operation of determining posture correction data for correcting the posture of the electronic device 101 determined through the inertial sensor 805 based on the .

The operation of determining the posture correction data includes an operation of converting the first difference value on the horizontal coordinate system into a value of the world coordinate system based on the posture of the electronic device 101 at the first time, and the operation of converting the first difference value into a value of the world coordinate system. An operation of determining a first reference posture based on the difference value, and an operation of determining posture correction data based on a difference between the first reference posture and the posture of the electronic device 101 at the first time.

A method for correcting an inertial sensor 805 according to an embodiment includes an operation of updating a posture correction table based on determined posture correction data, and an electronic device determined through the inertial sensor 805 based on the updated posture correction table ( 101) may further include an operation of correcting the posture.

A method for correcting an inertial sensor 805 according to an embodiment includes an operation of detecting a second position value of a target object from a second image obtained at a second time using a camera, and a position sensor 830 at the second time. An operation of estimating a second position value of the target object based on a position value measured by ), and a first difference value between the detected first position value and the estimated first position value and the detected second position value and estimation An operation of determining correction data of the inertial sensor 805 for calibrating the inertial sensor 805 based on the second difference between the second position values may be further included.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish that component from other corresponding components, and may refer to that component in other respects (eg, importance or order) is not limited. A (eg, first) component is said to be "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

Claims

In the electronic device 101,

a camera 825 to obtain an image;

an inertial sensor 805 for detecting motion of the electronic device 101;

a position sensor 830 for detecting the position of the electronic device 101 when the image is acquired; and

A processor 120 controlling the camera 825, the inertial sensor 805, and the position sensor 830,

The processor 120,

Detecting a first position value of a target object in a first image obtained at a first time,

Estimating a first position value of the target object based on a position value measured by the position sensor 830 at the first time;

Determine posture correction data for correcting the posture of the electronic device 101 determined through the inertial sensor 805 based on a first difference between the detected first position value and the estimated first position value To, the electronic device 101.
According to claim 1,

The processor 120,

The electronic device (101) detecting the first position value based on the view angle information of the camera (825) and the center of the first image.
According to claim 1 or 2,

The processor 120,

Based on the posture of the electronic device 101 measured at the first time through the inertial sensor 805, converting the first difference value on the horizontal coordinate system into a value on the world coordinate system,

Determine a first reference posture based on a first difference value converted to a value of the world coordinate system;

The electronic device 101 determining the posture correction data based on a difference between the first reference posture and the posture of the electronic device 101 measured at the first time.
According to claim 3,

The processor 120,

converting the first difference value on the horizontal coordinate system into a value on the Cartesian coordinate system;

The electronic device 101 converts a first difference value converted into a value of the Cartesian coordinate system into a value of the world coordinate system based on the posture of the electronic device 101 measured at the first time.
According to any one of claims 1 to 4,

The processor 120,

Based on the determined posture correction data, updating a posture correction table;

The electronic device 101 correcting the posture of the electronic device 101 determined through the inertial sensor 805 based on the updated posture correction table.
According to any one of claims 1 to 5,

The processor 120,

Detecting a second position value of a target object in a second image obtained at a second time,

Estimating a second position value of the target object based on the position value measured by the position sensor 830 at the second time;

The inertial sensor 805 is based on a first difference value between the detected first position value and the estimated first position value and a second difference value between the detected second position value and the estimated second position value. ), the electronic device 101 for determining inertial sensor correction data for correcting.
According to claim 6,

The processor 120,

Based on the posture of the electronic device 101 at the first time, converting the first difference value on a horizontal coordinate system into a value on a world coordinate system;

Determine a first reference posture based on a first difference value converted to a value of the world coordinate system;

Based on the posture of the electronic device 101 at the second time, converting the second difference value on the horizontal coordinate system into a value on the world coordinate system;

Determine a second reference posture based on a second difference value converted to a value of the world coordinate system;

determining a posture change rate between the first time and the second time based on the first reference posture and the second reference posture;

The electronic device 101 which determines the inertial sensor correction data based on the output value of the inertial sensor 805 and the attitude change rate between the first time and the second time.
According to claim 6 or 7,

The processor 120,

Based on the determined inertial sensor calibration data, an inertial sensor calibration table is updated;

The electronic device 101 for correcting the inertial sensor 805 based on the updated inertial sensor calibration table.
In the electronic device 101,

a camera 825 to obtain an image;

an inertial sensor 805 for detecting motion of the electronic device 101;

a position sensor 830 for detecting the position of the electronic device 101 when the image is acquired; and

A processor 120 controlling the camera 825, the inertial sensor 805, and the position sensor 830,

The processor 120,

Detecting a first position value of a target object in a first image obtained at a first time,

Estimating a first position value of the target object based on a position value measured by the position sensor 830 at the first time;

Detecting a second position value of a target object in a second image obtained at a second time,

Estimating a second position value of the target object based on the position value measured by the position sensor 830 at the second time;

The inertial sensor 805 is based on a first difference value between the detected first position value and the estimated first position value and a second difference value between the detected second position value and the estimated second position value. ), the electronic device 101 for determining inertial sensor correction data for correcting.
According to claim 9,

The processor 120,

Detecting the first position value based on information on the angle of view of the camera 825 and the center of the first image;

The electronic device (101) detecting the second position value based on the view angle information of the camera (825) and the center of the second image.
The method of claim 9 or 10,

The processor 120,

Based on the posture of the electronic device 101 at the first time, converting the first difference value on a horizontal coordinate system into a value on a world coordinate system;

Determine a first reference posture based on a first difference value converted to a value of the world coordinate system;

Based on the posture of the electronic device 101 at the second time, converting the second difference value on the horizontal coordinate system into a value on the world coordinate system;

Determine a second reference posture based on a second difference value converted to a value of the world coordinate system;

determining a posture change rate between the first time and the second time based on the first reference posture and the second reference posture;

The electronic device 101 which determines the inertial sensor correction data based on the output value of the inertial sensor 805 and the attitude change rate between the first time and the second time.
According to claim 11,

The processor 120,

converting the first difference value on the horizontal coordinate system into a value on the Cartesian coordinate system;

Converting a first difference value converted to a value of the Cartesian coordinate system into a value of the world coordinate system based on the attitude of the electronic device 101 at the first time;

converting the second difference value on the horizontal coordinate system into a value on the Cartesian coordinate system;

The electronic device 101 converts a second difference value converted into a value of the Cartesian coordinate system into a value of the world coordinate system based on the attitude of the electronic device 101 at the second time.
According to any one of claims 9 to 12,

The processor 120,

Based on the determined inertial sensor calibration data, an inertial sensor calibration table is updated;

The electronic device 101 for correcting the inertial sensor 805 based on the updated inertial sensor calibration table.
In the method for calibrating the inertial sensor 805 of the electronic device 101,

detecting a first position value of a target object from a first image obtained at a first time using the camera 825 of the electronic device 101;

estimating a first position value of the target object based on a position value measured by the position sensor 830 of the electronic device 101 at the first time; and

Determine posture correction data for correcting the posture of the electronic device 101 determined through the inertial sensor 805 based on a first difference between the detected first position value and the estimated first position value action to do

Including, how.
According to claim 14,

updating a posture correction table based on the determined posture correction data; and

Further comprising correcting the posture of the electronic device 101 determined through the inertial sensor 805 based on the updated posture correction table,

The operation of determining the posture correction data,

converting the first difference value on a horizontal coordinate system into a value on a world coordinate system based on the posture of the electronic device 101 at the first time;

determining a first reference posture based on a first difference value converted into a value of the world coordinate system; and

Determining the posture correction data based on the difference between the first reference posture and the posture of the electronic device 101 at the first time

Including, how.