KR20180079767A - An electronic device and control method therof - Google Patents

Abstract

Disclosed is an electronic apparatus and a control method thereof. The electronic apparatus according to various embodiments of the present invention comprises: a display unit; and a processor. The processor obtains camera information including image information photographed by an external photographing apparatus and information relating to a photographing posture of the external photographing apparatus. The processor also generates an image to be displayed on the electronic apparatus using the obtained image information. The processor may control the generated image to be corrected using the obtained camera information, and to display the corrected image on the display unit.

Description

ELECTRONIC DEVICE AND CONTROL METHOD THEREOF

The present disclosure relates to an electronic device and a control method thereof.

In recent years, with the rapid spread of portable electronic devices such as smart phones, the speed of spread of external attached electronic devices, which are connected to the portable electronic devices through wired communication or wireless communication to provide various functions to the user, is also increasing.

As the external accessory electronic device, a high-performance external photographing device has been actively introduced in recent years. For example, a photographing apparatus capable of photographing in 360 degrees can photograph an image in a plurality of different directions. The photographing apparatus can stitch the images photographed after photographing the images for a plurality of different directions to generate one omni-directional image. The photographing device can transmit the generated omnidirectional image to an electronic device that can use the omnidirectional image, for example, a personal computer or a smart phone. The electronic device connected to the photographing device may provide a virtual reality (VR) service using the received omnidirectional image. For example, the electronic device may be implemented as an HMD (Head Mounted Display) or combined with an HMD, and may display a part of an omnidirectional image according to a user's gaze direction.

In recent years, with the growing interest in virtual reality (VR), the development and dissemination of a variety of external electronic devices (e.g., Gears 360 ™, Gear VR ™, etc.) Is rapidly expanding.

An external electronic device (for example, a photographing device capable of photographing a 360-degree screen) connected to a portable electronic device, such as a smart phone, for providing an omnidirectional image for a virtual reality service, To provide a background image, it is used to capture an omnidirectional image. An electronic device, such as a smart phone or a PC, provided with a photographed image from an external photographing device for photographing an omnidirectional image processes the image by performing a post-process based on the photographed image. The processed image is then reproduced for a virtual reality service or the like.

On the other hand, there is a need to output the currently photographed images in real time, including the virtual reality service. However, the conventional omnidirectional image capturing and outputting methods have a lack of consideration for outputting photographed images from other electronic devices in real time Do. Particularly, there may be a shake in a photographed image due to various factors, such as a photographing device for photographing an omni-directional image may be installed on various mobile bodies such as a human body, a vehicle, an airplane, a drone or the like, . Therefore, it is required to provide a method of real-time processing and displaying an image in which the shaking occurs.

In response to this demand, according to the electronic apparatus according to various embodiments of this document, an image obtained by the external electronic apparatus is corrected in real time and displayed on the electronic apparatus, thereby providing a stable image without shaking to the user .

In particular, according to at least a part of various embodiments of the present document, when an image obtained by the external photographing apparatus is corrected in real time and displayed on the electronic device, in the case of an omnidirectional image, It provides some desired view area, that is, Field Of View (FOV), without shaking.

In addition, according to the electronic device according to various embodiments of the present document, by correcting and combining (stitching) the shooting axis of images obtained by a plurality of cameras in the external electronic device, It is possible to provide a wide-angle image such as an image.

An electronic device according to various embodiments of the present document includes a display unit and a processor, wherein the processor includes a camera including information relating to image information photographed by an external photographing apparatus and photographing posture of the external photographing apparatus, Acquiring the image information, generating an image to be displayed on the electronic device using the obtained image information, correcting the generated image using the obtained camera information, and displaying the corrected image on the display unit .

A control method of an electronic device according to various embodiments of the present document, the method comprising: acquiring camera information including image information photographed by an external photographing apparatus and information relating to a photographing posture of the external photographing apparatus; Generating an image to be displayed on the electronic device using the acquired image information, correcting the generated image using the obtained camera information, and displaying the corrected image .

According to various embodiments of the present document, an image obtained by the external electronic device is corrected in real time and displayed on the electronic device, thereby providing a stable image without shaking to the user. In particular, in at least some embodiments, in the case of an omnidirectional image, it may be provided without shaking the desired view area of the user, that is, the FOV (Field Of View). In addition, it is possible to provide an omni-directional image of higher quality by correcting and correcting the deviation of the image pickup axis of the images obtained by the plurality of cameras.

It should be apparent to those skilled in the art that the effects of this document are not limited to the effects described above, and that various effects are implied herein.

1 is an exemplary diagram for describing an electronic device in a network environment according to various embodiments of the present document.
2 is a block diagram illustrating an electronic device according to various embodiments of the present document.
3 is a block diagram of a program module according to various embodiments.
4A and 4B show a conceptual diagram for explaining an electronic apparatus and an external photographing apparatus according to various embodiments of the present document.
5 is a block diagram of an electronic device and an external photographing apparatus according to various embodiments of the present document.
6 is a diagram for explaining a data processing flow in an external photographing apparatus according to various embodiments of the present document.
Fig. 7 is an exemplary view for explaining an image generated by merge and stitching function / operation of an image performed by an electronic apparatus and an external photographing apparatus according to various embodiments of this document. Fig.
8 is an exemplary diagram for describing changes in pitch angle, roll angle, and yaw angle in an external imaging apparatus according to various embodiments of the present document.
Figs. 9A and 9B are exemplary diagrams for explaining a process of correcting an image using unique photographing characteristic information for a plurality of cameras transmitted from an external photographing apparatus, in an electronic device according to various embodiments of the present document .
10A and 10B are exemplary diagrams for explaining images that can be displayed in an electronic device in a case where an attitude of an external photographing apparatus according to various embodiments of the present document is changed.
Figs. 11A and 11B are exemplary diagrams illustrating the display of a user view area among omnidirectional images that can be displayed in an electronic device according to various embodiments of this document. Fig.
Figs. 12A, 12B and 12C illustrate the correction of the user's desired view area among the omnidirectional images that can be displayed in the electronic device according to various embodiments of the present document using the photographing posture information transmitted from the external photographing apparatus Fig.
13 is an exemplary diagram for explaining a communication procedure between an electronic apparatus and an external photographing apparatus according to various embodiments of the present document.
14 is an exemplary diagram for explaining an operational flow in an electronic apparatus and an external photographing apparatus according to various embodiments of the present document.
15A and 15B are exemplary diagrams for explaining the format of a file transmitted to an electronic device by an external photographing apparatus according to various embodiments of the present document.
16A and 16B are exemplary diagrams for describing a stream file format of video, audio and camera information transmitted to an electronic device by an external photographing device according to various embodiments of the present document.
17 is an exemplary diagram for explaining a control method of an electronic device according to various embodiments of this document.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. It should be understood, however, that this invention is not intended to be limited to the particular embodiments described herein but includes various modifications, equivalents, and / or alternatives of the embodiments of this document . In connection with the description of the drawings, like reference numerals may be used for similar components.

In this document, the expressions "have," "may," "include," or "include" may be used to denote the presence of a feature (eg, a numerical value, a function, Quot ;, and does not exclude the presence of additional features.

In this document, the expressions "A or B," "at least one of A and / or B," or "one or more of A and / or B," etc. may include all possible combinations of the listed items . For example, "A or B," "at least one of A and B," or "at least one of A or B" includes (1) at least one A, (2) Or (3) at least one A and at least one B all together.

The expressions "first," " second, "" first, " or "second ", etc. used in this document may describe various components, It is used to distinguish the components and does not limit the components. For example, the first user equipment and the second user equipment may represent different user equipment, regardless of order or importance. For example, without departing from the scope of the rights described in this document, the first component can be named as the second component, and similarly the second component can also be named as the first component.

(Or functionally or communicatively) coupled with / to "another component (eg, a second component), or a component (eg, a second component) Quot; connected to ", it is to be understood that any such element may be directly connected to the other element or may be connected through another element (e.g., a third element). On the other hand, when it is mentioned that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component) It can be understood that there is no other component (e.g., a third component) between the other components.

As used herein, the phrase " configured to " (or set) to be "adapted to, " To be designed to, "" adapted to, "" made to, "or" capable of ". The term " configured to (or set up) "may not necessarily mean" specifically designed to "in hardware. Instead, in some situations, the expression "configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be implemented by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) , And a generic-purpose processor (e.g., a CPU or an application processor) capable of performing the corresponding operations.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the other embodiments. The singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art. The general predefined terms used in this document may be interpreted in the same or similar sense as the contextual meanings of the related art and, unless expressly defined in this document, include ideally or excessively formal meanings . In some cases, even the terms defined in this document can not be construed as excluding the embodiments of this document.

An electronic device according to various embodiments of the present document may be, for example, a smartphone, a tablet personal computer, a mobile phone, a video phone, an e-book reader, Desktop personal computers, laptop personal computers, netbook computers, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs) A medical device, a camera, or a wearable device. According to various embodiments, the wearable device may be of the accessory type (e.g., a watch, a ring, a bracelet, a bracelet, a necklace, a pair of glasses, a contact lens or a head-mounted-device (HMD) (E. G., Electronic apparel), a body attachment type (e. G., A skin pad or tattoo), or a bioimplantable type (e.g., implantable circuit).

In some embodiments, the electronic device may be a home appliance. Home appliances include, for example, televisions, digital video disk players, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwaves, washing machines, air cleaners, set- home automation control panel, security control panel, TV box such as Samsung HomeSync ^TM , Apple TV ^TM or Google TV ^TM , game consoles such as Xbox ^TM and PlayStation ^TM , A key, a camcorder, or an electronic frame.

In an alternative embodiment, the electronic device may be any of a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a body temperature meter), magnetic resonance angiography (MRA) Navigation systems, global navigation satellite systems (GNSS), event data recorders (EDRs), flight data recorders (FDRs), infotainment (infotainment) systems, ) Automotive electronic equipment (eg marine navigation systems, gyro compass, etc.), avionics, security devices, head units for vehicles, industrial or home robots, automatic teller's machines (ATMs) Point of sale, or internet of things (eg, light bulbs, various sensors, electrical or gas meters, sprinkler devices, fire alarms, thermostats, street lights, Of the emitter (toaster), exercise equipment, hot water tank, a heater, boiler, etc.) may include at least one.

According to some embodiments, the electronic device is a piece of furniture or a part of a building / structure, an electronic board, an electronic signature receiving device, a projector, Water, electricity, gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic device may be a combination of one or more of the various devices described above. An electronic device according to some embodiments may be a flexible electronic device. Further, the electronic device according to the embodiment of the present document is not limited to the above-described devices, and may include a new electronic device according to technological advancement.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An electronic apparatus according to various embodiments will now be described with reference to the accompanying drawings. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

Referring to Figure 1, in various embodiments, an electronic device 101 in a network environment 100 is described. The electronic device 101 may include a bus 110, a processor 120, a memory 130, an input / output interface 150, a display 160, and a communication interface 170. In some embodiments, the electronic device 101 may omit at least one of the components or additionally include other components.

The bus 110 may include, for example, circuitry that interconnects the components 120-170 and communicates (e.g., control messages and / or data) between the components.

The processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 may perform computations or data processing related to, for example, control and / or communication of at least one other component of the electronic device 101.

The processor 120 acquires camera information including image information transmitted from another external electronic device (for example, the electronic device 102) and information related to the photographing posture at the time of photographing the image, And the camera information to control the image to be displayed on the electronic device and to display the corrected image on the display 160. [

Memory 130 may include volatile and / or non-volatile memory. Memory 130 may store instructions or data related to at least one other component of electronic device 101, for example. According to one embodiment, the memory 130 may store software and / or programs 140. The program 140 may include one or more of the following: a kernel 141, a middleware 143, an application programming interface (API) 145, and / or an application program . ≪ / RTI > At least a portion of the kernel 141, middleware 143, or API 145 may be referred to as an operating system (OS).

The kernel 141 may include system resources used to execute an operation or function implemented in other programs (e.g., middleware 143, API 145, or application program 147) (E.g., bus 110, processor 120, or memory 130). The kernel 141 also provides an interface to control or manage system resources by accessing individual components of the electronic device 101 in the middleware 143, API 145, or application program 147 .

The middleware 143 can perform an intermediary role such that the API 145 or the application program 147 can communicate with the kernel 141 to exchange data.

In addition, the middleware 143 may process one or more task requests received from the application program 147 according to the priority order. For example, middleware 143 may use system resources (e.g., bus 110, processor 120, or memory 130, etc.) of electronic device 101 in at least one of application programs 147 Priority can be given. For example, the middleware 143 may perform the scheduling or load balancing of the one or more task requests by processing the one or more task requests according to the priority assigned to the at least one task.

The API 145 is an interface for the application 147 to control the functions provided by the kernel 141 or the middleware 143, Control or the like, for example, instructions.

The input / output interface 150 may serve as an interface by which commands or data input from, for example, a user or other external device can be transferred to another component (s) of the electronic device 101. Output interface 150 may output commands or data received from other component (s) of the electronic device 101 to a user or other external device.

Display 160 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) A micro-electro-mechanical systems (MEMS) display, or an electronic paper display. Display 160 may display various content (e.g., text, image, video, icon, or symbol, etc.) to a user, for example. The display 160 may include a touch screen and may receive touch, gesture, proximity, or hovering input, for example, using an electronic pen or a portion of the user's body.

The communication interface 170 establishes communication between the electronic device 101 and an external device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106) . For example, communication interface 170 may be connected to network 162 via wireless or wired communication to communicate with an external device (e.g., second external electronic device 104 or server 106).

Wireless communications may include, for example, cellular communication protocols such as long-term evolution (LTE), LTE Advance (LTE), code division multiple access (CDMA), wideband CDMA (WCDMA) mobile telecommunications system, WiBro (Wireless Broadband), or Global System for Mobile Communications (GSM). The wireless communication may also include, for example, local communication 164. The local area communication 164 may include at least one of, for example, wireless fidelity (WiFi), Bluetooth, near field communication (NFC), or global navigation satellite system (GNSS). GNSS can be classified into two types according to the use area or bandwidth, for example, Global Positioning System (GPS), Global Navigation Satellite System (Glonass), Beidou Navigation Satellite System (Beidou) And may include at least one. Hereinafter, in this document, "GPS" can be interchangeably used with "GNSS ". The wired communication may include at least one of, for example, a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), or plain old telephone service (POTS). The network 162 may include at least one of a telecommunications network, e.g., a computer network (e.g., LAN or WAN), the Internet, or a telephone network.

Each of the first and second external electronic devices 102, 104 may be the same or a different kind of device as the electronic device 101. According to one embodiment, the server 106 may comprise a group of one or more servers. According to various embodiments, all or a portion of the operations performed in the electronic device 101 may be performed in one or more other electronic devices (e.g., electronic devices 102, 104, or server 106). According to the present invention, when electronic device 101 is to perform a function or service automatically or on demand, electronic device 101 may perform at least some functions associated therewith instead of, or in addition to, (E.g., electronic device 102, 104, or server 106) may request the other device (e.g., electronic device 102, 104, or server 106) Perform additional functions, and forward the results to the electronic device 101. The electronic device 101 may process the received results as is or additionally to provide the requested functionality or services. For example, Cloud computing, distributed computing, or client-server computing techniques can be used.

2 is a block diagram of an electronic device 201 according to various embodiments. The electronic device 201 may include all or part of the electronic device 101 shown in Fig. 1, for example. The electronic device 201 may include one or more processors (e.g., an application processor (AP)) 210, a communication module 220, a subscriber identification module 224, a memory 230, a sensor module 240, an input device 250 A display 260, an interface 270, an audio module 280, a camera module 291, a power management module 295, a battery 296, an indicator 297, and a motor 298 have.

The processor 210 may control a plurality of hardware or software components connected to the processor 210, for example, by driving an operating system or an application program, and may perform various data processing and calculations. The processor 210 may be implemented with, for example, a system on chip (SoC). According to one embodiment, the processor 210 may further include a graphics processing unit (GPU) and / or an image signal processor. Processor 210 may include at least some of the components shown in FIG. 2 (e.g., cellular module 221). Processor 210 may load or process instructions or data received from at least one of the other components (e.g., non-volatile memory) into volatile memory and store the various data in non-volatile memory have.

The communication module 220 may have the same or similar configuration as the communication interface 170 of FIG. The communication module 220 includes a cellular module 221, a WiFi module 223, a Bluetooth module 225, a GNSS module 227 (e.g., a GPS module, a Glonass module, a Beidou module, or a Galileo module) An NFC module 228, and a radio frequency (RF) module 229.

The cellular module 221 can provide voice calls, video calls, text services, or Internet services, for example, over a communication network. According to one embodiment, the cellular module 221 may utilize a subscriber identity module (e.g., a SIM card) 224 to perform the identification and authentication of the electronic device 201 within the communication network. According to one embodiment, the cellular module 221 may perform at least some of the functions that the processor 210 may provide. According to one embodiment, the cellular module 221 may include a communication processor (CP).

Each of the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228 may include a processor for processing data transmitted and received through the corresponding module, for example. At least some (e.g., two or more) of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228, according to some embodiments, (IC) or an IC package.

The RF module 229 can, for example, send and receive communication signals (e.g., RF signals). The RF module 229 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228 transmits / receives an RF signal through a separate RF module .

The subscriber identity module 224 may include, for example, a card containing a subscriber identity module and / or an embedded SIM and may include unique identification information (e.g., an integrated circuit card identifier (ICCID) Subscriber information (e.g., international mobile subscriber identity (IMSI)).

Memory 230 (e.g., memory 130) may include, for example, internal memory 232 or external memory 234. The built-in memory 232 may be implemented as, for example, a volatile memory (e.g., dynamic RAM, SRAM, or synchronous dynamic RAM), a non-volatile memory Programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash) A hard drive, or a solid state drive (SSD).

The external memory 234 may be a flash drive such as a compact flash (CF), a secure digital (SD), a micro secure digital (SD), a mini secure digital (SD) digital, a multi-media card (MMC), a memory stick, and the like. The external memory 234 may be functionally and / or physically connected to the electronic device 201 via various interfaces.

The sensor module 240 may, for example, measure a physical quantity or sense the operating state of the electronic device 201 to convert the measured or sensed information into an electrical signal. The sensor module 240 includes a gesture sensor 240A, a gyro sensor 240B, an air pressure sensor 240C, a magnetic sensor 240D, an acceleration sensor 240E, a grip sensor 240F, A temperature sensor 240G, a UV sensor 240G, a color sensor 240H (e.g., an RGB (red, green, blue) sensor), a living body sensor 240I, And a sensor 240M. Additionally or alternatively, the sensor module 240 may include, for example, an E-nose sensor, an electromyography sensor, an electroencephalogram sensor, an electrocardiogram sensor, , An infrared (IR) sensor, an iris sensor, and / or a fingerprint sensor. The sensor module 240 may further include a control circuit for controlling at least one or more sensors belonging to the sensor module 240. In some embodiments, the electronic device 201 further includes a processor configured to control the sensor module 240, either as part of the processor 210 or separately, so that while the processor 210 is in a sleep state, The sensor module 240 can be controlled.

The input device 250 may include a touch panel 252, a (digital) pen sensor 254, a key 256, or an ultrasonic input device 258). As the touch panel 252, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type can be used. Further, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile response to the user.

(Digital) pen sensor 254 may be part of, for example, a touch panel or may include a separate recognition sheet. Key 256 may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device 258 can sense the ultrasonic wave generated by the input tool through the microphone (e.g., the microphone 288) and confirm the data corresponding to the ultrasonic wave detected.

Display 260 (e.g., display 160) may include a panel 262, a hologram device 264, or a projector 266. Panel 262 may include the same or similar configuration as display 160 of FIG. The panel 262 may be embodied, for example, flexible, transparent, or wearable. The panel 262 may be composed of one module with the touch panel 252. [ The hologram device 264 can display a stereoscopic image in the air using interference of light. The projector 266 can display an image by projecting light onto a screen. The screen may be located, for example, inside or outside the electronic device 201. According to one embodiment, the display 260 may further comprise control circuitry for controlling the panel 262, the hologram device 264, or the projector 266.

The interface 270 may be implemented using a variety of interfaces including, for example, a high-definition multimedia interface (HDMI) 272, a universal serial bus (USB) 274, an optical interface 276, or a D- ) ≪ / RTI > The interface 270 may, for example, be included in the communication interface 170 shown in FIG. Additionally or alternatively, the interface 270 may be, for example, a mobile high-definition link (MHL) interface, a secure digital (SD) card / multi-media card (MMC) data association standard interface.

The audio module 280 can, for example, convert sound and electrical signals in both directions. At least some of the components of the audio module 280 may be included, for example, in the input / output interface 145 shown in FIG. The audio module 280 may process sound information input or output through, for example, a speaker 282, a receiver 284, an earphone 286, a microphone 288, or the like.

The camera module 291 may be, for example, a device capable of capturing still images and moving images, and may include one or more image sensors (e.g., a front sensor or a rear sensor), a lens, an image signal processor (ISP) , Or a flash (e.g., an LED or xenon lamp, etc.).

The power management module 295 can, for example, manage the power of the electronic device 201. [ According to one embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charger integrated circuit, or a battery or fuel gauge. The PMIC may have a wired and / or wireless charging scheme. The wireless charging scheme may include, for example, a magnetic resonance scheme, a magnetic induction scheme, or an electromagnetic wave scheme, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonant circuit, have. The battery gauge can measure, for example, the remaining amount of the battery 296, the voltage during charging, the current, or the temperature. The battery 296 may include, for example, a rechargeable battery and / or a solar battery.

The indicator 297 may indicate a particular state of the electronic device 201 or a portion thereof (e.g., processor 210), e.g., a boot state, a message state, or a state of charge. The motor 298 can convert electrical signals to mechanical vibration and can generate vibration, haptic effects, and the like. Although not shown, the electronic device 201 may include a processing unit (e.g., a GPU) for mobile TV support. The processing unit for supporting the mobile TV can process media data conforming to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or media flow ( ^TM ).

Each of the components described in this document may be composed of one or more components, and the name of the component may be changed according to the type of the electronic device. In various embodiments, the electronic device may comprise at least one of the components described herein, some components may be omitted, or may further include additional other components. In addition, some of the components of the electronic device according to various embodiments may be combined into one entity, so that the functions of the components before being combined can be performed in the same manner.

3 is a block diagram of a program module according to various embodiments. According to one embodiment, program module 310 (e.g., program 140) includes an operating system (OS) that controls resources associated with an electronic device (e.g., electronic device 101) (E.g., application programs 147) running on the system. The operating system may be, for example, android, iOS, windows, symbian, tizen, or bada.

The program module 310 may include a kernel 320, a middleware 330, an application programming interface (API) 360, and / or an application 370. At least a portion of the program module 310 may be preloaded on the electronic device or may be downloaded from an external electronic device such as the electronic device 102 104 or the server 106,

The kernel 320 (e.g., the kernel 141) may include, for example, a system resource manager 321 and / or a device driver 323. The system resource manager 321 can perform control, allocation, or recovery of system resources. According to one embodiment, the system resource manager 321 may include a process manager, a memory manager, or a file system manager. The device driver 323 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, or an inter-process communication .

The middleware 330 may provide various functions commonly required by the application 370 or may be provided through the API 360 in various ways to enable the application 370 to efficiently use limited system resources within the electronic device. Functions can be provided to the application 370. According to one embodiment, middleware 330 (e.g., middleware 143) includes a runtime library 335, an application manager 341, a window manager 342, a multimedia manager 343, a resource manager 344, a power manager 345, a database manager 346, a package manager 347, a connectivity manager 346, (Not shown) 348, a notification manager 349, a location manager 350, a graphic manager 351, or a security manager 352 can do.

The runtime library 335 may include, for example, a library module that the compiler uses to add new functionality via a programming language while the application 370 is executing. The runtime library 335 may perform input / output management, memory management, or functions for arithmetic functions.

The application manager 341 can manage the life cycle of at least one of the applications 370, for example. The window manager 342 can manage GUI resources used in the screen. The multimedia manager 343 can recognize the format required for reproducing various media files and can encode or decode the media file using a codec suitable for the format. The resource manager 344 can manage resources such as source code, memory or storage space of at least one of the applications 370.

The power manager 345 operates together with a basic input / output system (BIOS), for example, to manage a battery or a power source, and can provide power information and the like necessary for the operation of the electronic device. The database manager 346 may create, retrieve, or modify a database for use in at least one of the applications 370. The package manager 347 can manage installation or update of an application distributed in the form of a package file.

The connection manager 348 may manage wireless connections, such as, for example, WiFi or Bluetooth. The notification manager 349 may display or notify events such as arrival messages, appointments, proximity notifications, etc. in a way that is not disturbed to the user. The location manager 350 may manage the location information of the electronic device. The graphic manager 351 may manage the graphic effect to be provided to the user or a user interface related thereto. The security manager 352 can provide all security functions necessary for system security or user authentication. According to one embodiment, when an electronic device (e.g., electronic device 101) includes a telephone function, middleware 330 further includes a telephony manager for managing the voice or video call capabilities of the electronic device can do.

Middleware 330 may include a middleware module that forms a combination of various functions of the above-described components. The middleware 330 may provide a module specialized for each type of operating system in order to provide differentiated functions. In addition, the middleware 330 may dynamically delete some existing components or add new ones.

The API 360 (e.g., API 145) may be provided in a different configuration depending on the operating system, for example, as a set of API programming functions. For example, for Android or iOS, you can provide one API set per platform, and for tizen, you can provide more than two API sets per platform.

An application 370 (e.g., an application program 147) may include, for example, a home 371, a dialer 372, an SMS / MMS 373, an instant message 374, a browser 375, The camera 376, the alarm 377, the contact 378, the voice dial 379, the email 380, the calendar 381, the media player 382, the album 383 or the clock 384, or one or more applications capable of performing functions such as health care (e.g., measuring exercise or blood glucose), or providing environmental information (e.g., providing atmospheric pressure, humidity, or temperature information, etc.).

According to one embodiment, an application 370 is an application that supports the exchange of information between an electronic device (e.g., electronic device 101) and an external electronic device (e.g., electronic devices 102 and 104) For convenience, an "information exchange application"). The information exchange application may include, for example, a notification relay application for communicating specific information to an external electronic device, or a device management application for managing an external electronic device.

For example, the notification delivery application may send notification information generated by other applications (e.g., SMS / MMS applications, email applications, health care applications, or environmental information applications) of the electronic device to external electronic devices , 104), respectively. Further, the notification delivery application can receive notification information from, for example, an external electronic device and provide it to the user.

The device management application may be configured to perform at least one function (e.g., turn-on or turn-off) of an external electronic device (e.g., an electronic device 102 or 104) (E.g., on / off-off, or adjusting the brightness (or resolution) of the display), managing applications (e.g., , Or updated).

According to one embodiment, the application 370 may include an application (e.g., a healthcare application of a mobile medical device, etc.) designated according to an attribute of an external electronic device (e.g., electronic device 102, 104). According to one embodiment, application 370 may include an application received from an external electronic device (e.g., server 106 or electronic device 102, 104) May include a preloaded application or a third party application downloadable from a server. The names of the components of the program module 310 according to the illustrated embodiment may include the type of operating system Therefore, it can be changed.

According to various embodiments, at least some of the program modules 310 may be implemented in software, firmware, hardware, or a combination of at least two of them. At least some of the program modules 310 may be implemented (e.g., executed) by, for example, a processor (e.g., processor 210). At least some of the program modules 310 may include, for example, modules, programs, routines, sets of instructions or processes, etc. to perform one or more functions.

4A and 4B show a conceptual diagram for explaining an electronic apparatus and an external photographing apparatus according to various embodiments of the present document.

As shown in FIG. 4A, the electronic device 101 may be connected to an associated external photographing device, for example, a photographing device 400. [ At this time, the electronic device 101 may be wirelessly connected. The electronic device 101 and the photographing device 400 may include a communication circuit that supports, for example, a WiFi method including a WiFi direct method, and may be wirelessly connected thereto. In addition, it may be configured to perform wireless communication through various short-range communication methods such as a Bluetooth method, a BLE (Bluetooth Low Energy) method, an NFC method, and a Zig-bee method.

4B, the electronic device 101 may be wired to the photographing apparatus 400 via a cable 402. In this case, The cable 402 may be made, for example, based on a USB standard, and may include connectors conforming to various USB standards. The electronic device 101 and the photographing device 400 may also include a physical port in accordance with various USB standards so that the electronic device 101 and the photographing device 400, Can be connected.

The photographing apparatus 400 can perform photographing with respect to an external landscape, and thus can acquire an image. The photographing apparatus 400 can photograph an image in omnidirectional 360 degrees based on the fixed point of time. Here, an omnidirectional image is an image including both a view that the photographer rotates once in its turn and a view that the viewer looks up and tilts.

The photographing apparatus 400 may include a plurality of cameras for capturing an image in all directions, for example, a plurality of directions, thereby obtaining a plurality of images for a plurality of directions. Each of the plurality of cameras included in the photographing apparatus 400 may include, for example, a fish-eye lens, and may have an angle of view exceeding 180 degrees, for example. Each of the plurality of cameras may be arranged so that a part of each of the view angles overlap with each other, so that the images captured by the plurality of cameras may include overlapping portions. The photographing apparatus 400 may transmit a plurality of images to the electronic apparatus 101, and the electronic apparatus 101 may stitch the plurality of images into one image. Alternatively, the plurality of images photographed by the photographing apparatus 400 may be stitched as one image (i.e., an omnidirectional image) and transmitted to the electronic device 101. [

For example, the photographing apparatus 400 may include two cameras having a fisheye lens having an angle of view of more than 180 degrees, one on the front side and one on the rear side, Images can be acquired. By using these two images, the electronic device 100 generates one omni-directional image. It should be understood by those skilled in the art that the number, the angle of view, and the installation position of the cameras provided in the photographing apparatus 400 described above are merely exemplary and that various modifications may be made.

Meanwhile, the photographing apparatus 400 can photograph the location of the photographing apparatus, which is received through the photographed image and related metadata, such as a sensor (e.g., GPS, WiFi fingerprint, gyro sensor, acceleration sensor, geomagnetic sensor, altitude sensor, , Posture information, etc., can be associated with at least one of images captured as camera property information (e.g., camera calibration parameter, shooting state information). For example, the photographing apparatus 400 may acquire at least camera position information (for example, pitch, roll, yaw angle) of the posture of the photographing apparatus 400, Quot; camera information ").

In addition, the photographing apparatus 400 may map the captured omnidirectional image to a two-dimensional plane image, encode the mapped two-dimensional plane image, store the encoded two-dimensional plane image in its own memory, or transmit it to the electronic device 101.

5 is a block diagram of an electronic device and an external photographing apparatus according to various embodiments of the present document.

Referring to FIG. 5, an external photographing apparatus 501 such as a camera apparatus serves as an input apparatus for receiving an omnidirectional image, for example. The external photographing apparatus 501 is capable of photographing 360 degrees in all directions and wirelessly transmitting a photographed image or a preview screen image in real time. The electronic device 502 such as a smart phone serves as an output device for displaying an omnidirectional image provided from the external photographing apparatus 501.

The external photographing apparatus 501 includes a video input unit 511 and a video encoder 512 for inputting and processing video signals; A sensor unit 531 and an angle information converting unit 532 for obtaining camera information; A multiplexing unit 544 for synthesizing image data and camera data and a communication unit 545 for transmitting the output of the multiplexing unit 544 to the electronic device 502. [ In addition, it may further include an audio input unit 521 and an audio encoder 522 for inputting and processing audio signals.

The video input unit 511 includes two cameras (a fish-eye lens and an image sensor), that is, a first camera 511-1 and a second camera 511-1 installed on the front and rear sides, respectively, And a video merging unit 511-3 for merging the images photographed by the first and second cameras 511-1 and 511-2. The video encoder 512 performs encoding on the merged image provided from the image merging unit 511-3 of the video input unit 511. [ For example, the video encoder 512 may perform encoding based on codec standards such as H.264, MPEG-4, and HEVC. The audio input unit 521 may include a microphone or the like, and receives an external audio signal. The audio encoder 522 performs encoding on the received audio signal provided from the audio input unit 521. [ Audio encoder) 522 converts the received audio signal into a compressed signal or PCM data using an encoding compression technique such as AAC or AC3.

The sensor unit 531 is implemented by a gyro sensor, an acceleration sensor, and the like, and detects an angular velocity and an acceleration of the external photographing apparatus 501. The angle information converting unit 532 converts the raw data sensed by the sensor unit 531 into angle information and outputs information on the photographing attitude (for example, pitch, roll, yaw) (e.g., a yaw angle), the position and movement information of the external photographing apparatus 510, and the like. The sensor unit 531 may further include a GPS module, a movement detection sensor, and the like so that the sensor unit 531 detects the position and movement of the external photographing apparatus 501 and transmits the information to the electronic apparatus.

The multiplexer 544 synthesizes the video / audio data encoded by the video encoder 512 and the audio encoder 522 and converts the synthesized video / audio data into, for example, a format for wireless transmission or storage. In particular, the multiplexing unit 544 performs a function of associating the camera information provided by the angle information converting unit 532 with metadata related to the video / audio signal. That is, the metadata may be stored in a memory (not shown) together with the encoded video / audio data, or may be transmitted in streaming or file format. According to various embodiments, the metadata may be generated in a separate document format and associated with the encoded audio / video data and stored, or may be transmitted in association with a request for the audio / video data. In this way, the multiplexer 544 must support a plurality of video and audio data, and then support camera information for stitching a plurality of images input from the electronic device 502 into one image. An example of a file format for this purpose will be described in detail with reference to FIG. 15 and the like which will be described later.

The wireless communication unit 545 includes, for example, a WiFi module 545-1 and wirelessly transmits a data file (or streaming) provided by the multiplexing unit 544 in a WiFi manner.

The electronic device 502 includes a wireless communication unit 555 for receiving data files or streaming including omnidirectional image information wirelessly transmitted from the external imaging device 501 and data received via the wireless communication unit 555 A demultiplexer 556 for demultiplexing the video data and the camera data from the file, a video decoder 557 for decoding the video data provided by the demultiplexer 556, And a display unit 560 for displaying an image on the screen according to the display mode. In addition, the decoded video data may further include a stitching unit 558 for stitching the decoded video data when a plurality of images of the cameras are merged, and further includes a picture stabilization unit 559 for compensating the shaking of the picture can do. In addition, it may further include an audio decoder 567 and an audio output unit 569 (e.g., a speaker) for decoding and outputting audio data.

The wireless communication unit 555 of the electronic device 502 is provided with a WiFi module 555-1 or the like and transmits data wirelessly transmitted according to the WiFi method from the wireless communication unit 545 of the external photographing apparatus 501 And provides the demultiplexing unit 556 with the demultiplexed data.

In various embodiments of the present document, the demultiplexing unit 556 can demultiplex the video data, the audio data, and the camera information in the data file received by the wireless communication unit 545, and output the demultiplexed data.

Video decoder. The video data provided by the demultiplexer 556 can be decoded based on codec standards such as H.264, MPEG-4, and HEVC.

The stitching unit 558 stitches the video data provided by the demultiplexing unit 556 when a plurality of (for example, two) camera images are merged to form one image, for example, one And generates an omnidirectional image. At this time, according to at least a part of various embodiments of this document, the stitching unit 558 corrects the deviation of the shooting axis of each of the images in accordance with characteristic information unique to each camera corresponding to the plurality of images obtained from the camera information So that the shooting axes of the plurality of images are accurately aligned with each other. As described above, when a plurality of images are stitched to each other through the correction of a shooting axis of a plurality of images, the problem that the images do not exactly match each other at the interface between a plurality of images is solved. The details of correction of the shift correction of the shooting axis between images will be described later.

The image stabilizing unit 559 compensates for an image provided by the stitching unit 558, for example, an omnidirectional image, so that the image is stably displayed on the screen. At this time, the display unit 559 confirms the variation of the photographing posture corresponding to the image to be displayed at present based on the information about the photographing posture obtained from the camera information, according to at least a part of various embodiments of the present document, And outputs it to the display unit 560. The display unit 560 receives the positional change of the image according to the change of the photographing posture. The display unit 560 displays the image output from the image stabilization unit 559 on the screen. Accordingly, the image of the displayed screen is stably displayed without shaking. The details of the shake correction of the image will be described later.

6 is a diagram for explaining a flow of signal (data) processing in an external photographing apparatus according to various embodiments of the present document. Referring to Fig. 6, the data processing in the external photographing apparatus according to various embodiments of this document will be described in more detail.

Referring to FIG. 6, the entire signal flow in the external photographing apparatus can be largely divided into an audio signal part, a video signal part, and a file system part. The video and file system parts can be separated into a main video signal part and a sub video signal part. The following explains the block and each signal in each signal part.

The audio input unit 600 receives audio data and converts the input analog audio signal into a digital audio signal a0. The audio encoder 601 converts the input digital audio signal into a compressed signal or PCM data a1 using an encoding compression technique such as AAC or AC3.

The video input unit 610 converts the input analog video signal received through the camera sensor into a digital signal and the video merge unit 611 merges the input video images into a video signal vO . The merged video signal v0 becomes the input of the video size converter 612 and the video encoder 6132, respectively. The video size converter 612 converts the input video signal v0 into a video signal v1 of a size set by the user. The image size conversion method is performed by using Bi-linear, Bi-cubic, and Non-linear filters. That is, the user can select one of various image sizes and set them in advance. The magnitude-converted signal v1 becomes an input to the video encoder 613. [ The input data of the video encoder 613 is the original video signal v0 and the converted video signal v1 and the output data is the compressed video data v2 and the converted compressed video data v3 ).

The storage A / V multiplexer 620 is a general file system multiplexer and stores the inputted audio data a1 and video data v2 and camera information into a predetermined file format standard and stores them in the storage medium 630 . Typical file format specifications include 'MP4', 'mov', and 'avi'.

The stream A / V multiplexer 640 transmits data to the network connection unit 650 for real-time transmission. The stream A / V multiplexer 640 has an appropriate file format for real time transmission. The file structure may include a plurality of video information and a plurality of audio information, and in particular, a plurality of camera-related information. The information related to this camera has the following contents. Camera-calibration information, geomagnetic sensor information, GSP information, and camera motion information at all times. The view point related information can be set as a view point, for example, as the basic view point, when the user does not designate a view point separately, for example, at the front of the photographing apparatus.

Fig. 7 is an exemplary view for explaining an image generated by merge and stitching function / operation of an image performed by an electronic apparatus and an external photographing apparatus according to various embodiments of this document. Fig. The external photographing apparatus 501 and the electronic apparatus 502 shown in FIG. 7 may have the structure shown in FIG.

Referring to FIG. 7, an external photographing apparatus according to various embodiments of the present document may combine, for example, images obtained by two cameras provided in the external photographing apparatus, as shown in FIG. 7 . In FIG. 7, for the convenience of explanation, the case where two cameras are provided in the external photographing apparatus has been exemplarily described, but the embodiment of the present document is not limited thereto. An external photographing apparatus according to various embodiments of the present document may transmit image information on the merged image to the electronic device 501. [

Referring to FIG. 7, an electronic device 502 according to various embodiments of the present document generates an image to be displayed on a display unit of the electronic device using image information received from the external photographing apparatus 501 . 7 illustrates an example in which images stitched on the merged images provided by the external electronic device 501 by the electronic device 502 (and also, for example, the inter-image photographing axis correction is performed) As shown in Fig. Through the stitching function / operation, the electronic device 502 can provide a 360-degree omni-directional image to the user. An image displayed on the electronic device 502 after the stitching is performed may be obtained by a preview image obtained by the external photographing device 501 and / or a preview image obtained by the external photographing device 501 ≪ RTI ID = 0.0 >

8 is an exemplary diagram for describing changes in pitch angle, roll angle, and yaw angle in an external imaging apparatus according to various embodiments of the present document.

According to various embodiments of the present document, a change in the photographing posture of the external photographing apparatus may occur, for example, the external photographing apparatus may be installed on various mobile bodies or the installation environment may be changed. A change in at least one of a pitch angle, a roll angle and a yaw angle may be caused by the external photographing apparatus to correspond to the change of the photographing posture. The external photographing apparatus transmits information on the pitch, roll, and yaw angle (i.e., camera information) to the electronic apparatus, so that the electronic apparatus can recognize the photographing posture of the external photographing apparatus. The pitch angle corresponds to a tilt value (e.g., a rotation angle of the x-axis) of the camera (or the external photographing apparatus) Corresponds to a rotation value (e.g., a rotation angle with respect to the y-axis) having the photographing axis as a rotation axis, and the yaw angle corresponds to a rotation / pan-pan value of the camera (or the external photographing apparatus) (E.g., the angle of rotation about the z-axis).

In addition, according to various embodiments of the present document, when the external photographing apparatus includes a plurality of cameras, when information is initially connected to the electronic apparatus, information about photographing characteristics unique to the plurality of cameras is stored And transmitted to an electronic device.

More specifically, an external photographing apparatus according to various embodiments of the present document may include at least two cameras, for example, to provide a 360-degree omnidirectional image to a user. At this time, the external photographing apparatus can previously store photographing characteristic information unique to each of the plurality of cameras, and when the external photographing apparatus and the electronic apparatus are initially connected, for example, To the electronic device.

The unique photographing characteristic information for the plurality of cameras may also be referred to as a default camera information or a reference camera information. This is because when the external photographing apparatus is manufactured, Pitch angle information, roll angle information, and yaw angle information. For example, the pitch angle, the roll angle, and the yaw angle may be set to 0 degrees as the unique photographing characteristic information of the first camera. However, this is an example, and at least one of the angles may not be set to 0 due to an error in the manufacturing process or the like. Also, among the plurality of cameras, the pitch, the roll, and the yaw angle may be different from each other due to installation tolerance between the first camera and the second camera. In this document, the inherent photographing characteristic information about a plurality of cameras installed in the external photographing apparatus is transmitted to the electronic apparatus, so that the electronic apparatus can correct the deviation of the photographing axis of the image photographed by the plurality of cameras.

Figs. 9A and 9B are exemplary diagrams for explaining a process of correcting an image using unique photographing characteristic information for a plurality of cameras transmitted from an external photographing apparatus, in an electronic device according to various embodiments of the present document .

An electronic device according to various embodiments of the present document can calibrate each of a plurality of images transmitted from the external photographing apparatus using unique photographing characteristic information for a plurality of cameras transmitted from the external photographing apparatus. In FIG. 9A, an image obtained by a front camera (a camera located in a direction facing a specific object), for example, a front camera (a camera located in a direction facing a specific object) An angle and a yaw angle) of the image. Similarly, in FIG. 9B, an image obtained by a rear camera (a camera located in a direction facing the user) of the external photographing apparatus is referred to as the second camera unique photographing characteristic information (i.e., Angle, and yaw angle) of the image. In the example of FIGS. 9A and 9B, it is shown that the images taken by the first camera and the second camera are accurately corrected in the horizontal direction slope.

9A and 9B according to various embodiments of the present document, it is possible to perform stitching on the corrected image. Accordingly, a plurality of images photographed by a plurality of cameras can be accurately aligned, so that the interface between a plurality of images can be smoothly connected at the time of stitching. The electronic device may display the stitched image on the electronic device.

10A and 10B are exemplary diagrams for explaining images that can be displayed in an electronic device in a case where an attitude of an external photographing apparatus according to various embodiments of the present document is changed.

10A shows an example of an omnidirectional image stitched in an electronic device when, for example, the position of the external photographing apparatus is in a correct position. 10A, when the posture of the external photographing apparatus is reversed, an omnidirectional image stitched in the electronic apparatus is in a state in which the object of the image is inverted as compared with that of FIG. 10A. When an omnidirectional image is displayed on the screen in the state shown in Fig. 10B, an inverted image is displayed on the screen.

On the other hand, the entire area of the omnidirectional image as shown in Figs. 10A and 10B may be displayed on the screen of the electronic device, but the object recognition rate, the viewing angle and visibility of the actual person, , Only a part of the view area corresponding to the user interest area (FOV) among the omnidirectional images is displayed.

Figs. 11A and 11B are exemplary diagrams illustrating the display of a user view area among omnidirectional images that can be displayed in an electronic device (e.g., a smartphone) according to various embodiments of the present document. 11A and 11B may be screens corresponding to the forward direction images of FIGS. 10A and 10B, respectively.

As shown in FIGS. 11A and 11B, an image displayed on the screen of the electronic device displays a partial view area (FOV) corresponding to a view point designated by the user among the omnidirectional images. In FIGS. 11A and 11B, for example, the view area is set as the center part of the omnidirectional image. In FIG. 11B, the image of the view area is displayed in an inverted state as compared with FIG. 11B.

12A, 12B, and 12C illustrate the correction of a user's desired view area of an omnidirectional image that can be displayed in an electronic device according to various embodiments of the present document using photographing posture information transmitted from an external photographing apparatus Fig. 12A, 12B, and 12C show an omnidirectional image and an example of a rectangular region of the center of the omnidirectional image is a view region.

The external photographing apparatus transmits the image information in real time while information about the photographing posture (i.e., the pitch angle, the roll angle, the yaw angle) of the external photographing apparatus is periodically included in the camera information, Lt; / RTI > The electronic device can know the shake of the external photographing apparatus (and thus the shaking of the photographing screen) by using information about the photographing posture transmitted from the external photographing apparatus. In addition, the electronic device corrects the position of the currently displayed view area in accordance with the information on the photographing posture, resulting in a display image without blur.

That is, in various embodiments of the present document, the electronic device uses information (pitch angle, roll angle, yaw angle information) about the photographing posture in a state in which the rectangular area of Fig. , The omnidirectional image is rotated in the clockwise / counterclockwise direction or moved up and down and left and right so that the image corresponding to the position of the view region is displayed on the screen without change (that is, without shaking) . Alternatively, in various other embodiments of this document, as shown in FIG. 12C, the view region may be rotated in the clockwise / counterclockwise direction in the omnidirectional image or may be moved in the up / down / left / And as a result, it is possible to eliminate blurring in the image displayed on the screen.

For example, when the orientation of the external photographing apparatus is reversed and rotated, the electronic apparatus displays the photographing position information such that the image is displayed in the same direction as the image photographed without the external photographing apparatus being rotated, for example, Directional image is rotated according to the roll angle so that the view area is displayed on the screen. Similarly, when the attitude of the external photographing apparatus tilts upward or downward or panes left and right, the electronic device moves the omnidirectional image vertically or horizontally according to the pitch angle or yaw angle in the photographing posture information, As shown in Fig. In addition, in various embodiments of the present document, it is also possible to implement such that the external photographing apparatus transmits its own movement information to the electronic apparatus, and the electronic apparatus further reflects such movement information, It can also be implemented to change it.

13 is an exemplary diagram for explaining a communication procedure between an electronic device (e.g., a smart phone) and an external photographing apparatus according to various embodiments of the present document.

Referring to FIG. 13, in operation 1301, an electronic device and an external photographing apparatus perform an interconnection operation according to a preset method among various communication methods. For example, Universal Plug and Play (UPnP). Such a connection operation can perform operations such as mutual device discovery, mutual authentication, establishment of a communication connection, and mutually supportable service identification.

In operation 1311, the electronic device requests to start shooting video transmission to an external imaging device. In response to the request, the external imaging device responds with an electronic device. Upon receiving the response to the external imaging device, the electronic device requests basic information relating to image, audio, and camera (i.e., unique characteristic information of each of the plurality of cameras) in 1313 operation. In operation 1314, the external imaging device transmits the relevant information in response to the request of the electronic device.

Thereafter, in the operation 1315, the external photographing apparatus proceeds to transmit images and audio data photographed through a plurality of cameras for realizing the forward direction image in real time. In particular, the camera information including the information about the photographing position is transmitted periodically. Accordingly, the electronic device can perform the shooting-axis correction and the shooting-posture correcting operation on a plurality of images. On the other hand, at this 1315 operation, as described below in accordance with at least some of the various embodiments of this document, for example, the transmission of the video centered at the base view point (i.e., the image corresponding to the view area) .

In 1316 operation, according to at least some of the various embodiments of this document. When an event of a viewpoint adjustment by a user occurs in the electronic device, it is requested to the external photographing device. Thus, in operation 1317, the external imaging device can update the camera information and transmit the requested viewpoint-related image data, along with the associated information.

In the 1316 operation, the viewpoint adjustment event may be generated corresponding to an operation on the touch screen appropriately set in advance, for example, in order to operate the movement in the up, down, left, and right directions of the screen in the electronic device having the touch screen display. Alternatively, the viewpoint adjustment event may be generated corresponding to a change in the direction of the user's gaze, for example, in an electronic device implemented as an HMD or combined with an HMD.

Meanwhile, in various embodiments of the present document, the external photographing apparatus transmits all image data of images photographed through a plurality of cameras to an electronic device for generating an omnidirectional image, And adjust the view area by itself in accordance with the input of the viewpoint adjustment event. However, as shown in FIG. 13, when the external image pickup device transmits image data corresponding to the view area to the electronic device, the transmission efficiency can be increased.

14 is an exemplary diagram for explaining an operational flow in an electronic apparatus and an external photographing apparatus according to various embodiments of the present document. The operation shown in FIG. 14 may correspond to a more detailed operation flow of operation 1315 for transmitting related data including image data in real time, for example, in the communication procedure shown in FIG.

Referring to Fig. 14, an external photographing apparatus (for example, reference numeral 501 in Fig. 5) performs image input and processing procedures, audio input and processing procedures, and camera information input and processing procedures simultaneously. That is, the operation 1401, the operation 1411, and the operation 1421 are performed in parallel, and information on the image input, the audio input, and the photographing posture (i.e., the sensor information) is received. In operation 1402, if the input image is an image input by a plurality of cameras (i.e., lenses) and the image is input by a plurality of cameras, a plurality of images are merged in operation 1403, And encodes the data for the merged images. If it is determined in operation 1402 that the input image is not an image input by a plurality of cameras, the image processing apparatus proceeds directly to operation 1404 to encode the input image. On the other hand, in the operation 1421, In operation 1411, encoding is performed on the audio data input. In operation 1422, the sensor information (raw data) input in operation 1421 is converted into angle information.

In operation 1430, the encoded image data and audio data and the converted angle information are multiplexed. In operation 1440, the multiplexed data file is wirelessly transmitted to the electronic device.

Through the above operation, it can be seen that the external photographing device sums (multiplexes) the video / audio / sensor information inputted in real time and wirelessly transmits it to the electronic device in real time.

An electronic device (e.g., reference numeral 502 in FIG. 5) receives the data file transmitted in real time from the external photographing device, and performs image processing procedures, audio processing procedures, and camera information processing procedures at the same time. That is, the 1471 operation and the 1481 operation are performed in parallel to decode video data and audio data, respectively. In operation 1472, if the decoded image data is a plurality of image data by a plurality of cameras (i.e., lenses), a plurality of images are stitched together with the image sensing axis correction in the 1473 operation .

In operation 1490, the decoded audio is output while displaying the image of the view area of the stitched image or the stitched image on the screen. At this time, the shake correction of the image displayed on the screen is performed.

15A and 15B are exemplary diagrams for explaining the format of a file transmitted to an electronic device by an external photographing apparatus according to various embodiments of the present document. The file format shown in Figs. 15A and 15B may be, for example, a format of a data file transmitted at the initial connection from an external photographing apparatus to an electronic apparatus.

15A and 15B, the transmission file can be basically divided into a file format field portion and a data field portion. In the file format field portion, a video file corresponding to each of a plurality of cameras (or video) (I.e., 'Video Steam Information 1, 2, ..., n') and format information for an audio channel (i.e., 'Audio Stream Information'). The format information for these video channels and the format information for the audio channel may be substantially the same as the format of the format information for general video / audio transmission. At this time, in the file format field portion, the camera data format information (i.e., 'Camera Information 1, 2, ..., n') for each of a plurality of cameras is recorded in addition to the audio / video format information according to various embodiments of this document ) Is further included.

As shown in FIG. 15B, the field portion for the camera data format information includes a header portion (e.g., a 'camera data header mark') field portion indicating a camera data format information, a header size field portion, It can be subdivided into real information field parts for formatting.

The actual information field portion related to the camera data format includes pitch angle, roll angle, and yaw angle information corresponding to unique photographing characteristic information for each of the plurality of cameras, pitch angle, roll angle, There may be angle information. Alternatively, there may be information obtained by converting the pitch angle, roll angle, and yaw angle information into clock / counterclockwise rotation, up / down tilt, and pan (horizontal) Value). In addition, although not shown in FIG. 15B, movement information of the external photographing apparatus may also be included. In particular, in various embodiments of the present document, the unique imaging characteristics information for each of the plurality of cameras may be transmitted, for example, in clockwise / counterclockwise rotation, up / down slope, and horizontal rotation values, Pitch angle, roll angle, and yaw angle information.

In addition, each of the pieces of information included in the camera data format information may be defined in advance for the transmission type. In addition, the transmission period for the pitch angle, roll angle, and yaw angle information corresponding to the photographing posture can be defined in advance.

As described above, a file format field including camera data format information can be implemented, and each audio / video data and camera data associated with the file format field are transmitted in a data field portion. In the real-time data stream transmission operation, in the data field, the pitch angle, the roll angle, and the yaw angle information corresponding to the unique photographing characteristic information for each of the plurality of cameras, for example, Clockwise / counterclockwise rotation, up-and-down tilt, and horizontal rotation values corresponding to pitch angle, roll angle, and yaw angle information corresponding to information on the real-time shooting posture, Or GPS information, movement information of the external photographing apparatus, and the like are transmitted.

16A and 16B are exemplary diagrams for describing a stream file format of video, audio and camera information transmitted to an electronic device by an external photographing device according to various embodiments of the present document. 16A and 16B, in particular, a transmission format of audio / video data and camera data corresponding to each of a plurality of cameras, which can be included in the data field portion, is shown in more detail in comparison with Figs. 15A and 15B.

As shown in FIGS. 16A and 16B, each of the video data, audio data, and camera data includes a tag field (i.e., 'Video Data Tag', 'Audio Data Tag', and 'Camera Data Tag' , A field describing a data size, a time stamp field, and a data field.

16A shows an embodiment in which data is transferred using, for example, an external photographing apparatus using DMA (Direct Memory Access). 16B, when data is processed in accordance with a communication standard (for example, when dummy information is included in the data) in a case where data is transmitted through a specific communication standard (for example, a protocol for high- And the size of the data to be transmitted is aligned).

In this document, the process of stitching an image obtained from the external electronic device has been described as being performed by the electronic device. However, according to various embodiments of this document, the stitching function / or operation may be performed by the external electronic device. In this case, information about the stitched image may be provided to the electronic device.

According to various embodiments of the present document, the external electronic device may include at least some of at least two cameras, a memory, a processor, a communication interface, and a display.

17 is an exemplary diagram for explaining a control method of an electronic device according to various embodiments of this document.

17, a method of controlling an electronic device according to various embodiments of the present document may include an act 1700 of obtaining image information obtained by an external electronic device and camera information of the external photographing device have. An operation 1710 of generating an image to be displayed on the electronic device using the obtained image information, an operation 1720 of correcting the generated image using the obtained camera information, (1730) to the electronic device.

In the description of the various embodiments of the present document described above, an operation in which stitching is performed on an image obtained in the external photographing apparatus has been described as being performed by the electronic apparatus. However, according to various embodiments of this document, in addition, the stitching function / or operation may be performed by the external photographing apparatus. Of course, in this case, the photographing axis correcting operation for a plurality of cameras can be performed in the external photographing apparatus when the stitching function / operation is performed. The photographing axis is corrected in the external photographing apparatus and the stitched image can be provided to the electronic apparatus. In this case, the electronic apparatus performs the photographing posture correcting operation to perform the image providing operation without blurring.

In addition, in the description of the various embodiments of the present document, the case where the external photographing apparatus and the electronic apparatus are directly or wirelessly connected is described as an example. However, according to various embodiments of this document, the external photographing device and the electronic device may not be directly connected. For example, an electronic device may receive and use an image provided from an external photographing device at a remote location via a wireless Internet network.

In addition, according to various embodiments of the present document, the electronic device may perform the reproducing operation based on the image information already stored in the recording device, instead of receiving the image information transmitted in real time from the external photographing device, , It is also possible to perform the photographing axis correction and / or the posture correcting operation through the camera information that can be included in the previously stored image information.

In addition, according to various embodiments of the present document, the external photographing apparatus may perform an image photographing operation through a single wide-angle camera instead of a plurality of cameras. In such an external photographing apparatus, May be a wide-angle image photographed by a camera of the user. In this case, the photographing apparatus may perform only the photographing posture correction (i.e., the shake correction) while displaying the image of the view area on the screen through the wide angle image provided from the external photographing apparatus.

Also, according to various embodiments of the present document, an image photographed by the external photographing apparatus may not be multiplexed with audio data. In addition, various modifications and variations of the present document may be possible. In addition, the description of the electronic device according to various embodiments of the above-described document can be equally applied to the control method of the electronic device according to various embodiments of the present document.

As used in this document, the term "module" may refer to a unit comprising, for example, one or a combination of two or more of hardware, software or firmware. A "module" may be interchangeably used with terms such as, for example, unit, logic, logical block, component, or circuit. A "module" may be a minimum unit or a portion of an integrally constructed component. A "module" may be a minimum unit or a portion thereof that performs one or more functions. "Modules" may be implemented either mechanically or electronically. For example, a "module" may be an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs) or programmable-logic devices And may include at least one.

At least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to various embodiments may include, for example, computer-readable storage media in the form of program modules, As shown in FIG. When the instruction is executed by a processor (e.g., processor 120), the one or more processors may perform a function corresponding to the instruction. The computer readable storage medium may be, for example, memory 130. [

The computer readable recording medium may be a hard disk, a floppy disk, a magnetic media (e.g., a magnetic tape), an optical media (e.g., a compact disc read only memory (CD-ROM) digital versatile discs, magneto-optical media such as floptical disks, hardware devices such as read only memory (ROM), random access memory (RAM) Etc. The program instructions may also include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter, etc. The above- May be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

Modules or program modules according to various embodiments may include at least one or more of the elements described above, some of which may be omitted, or may further include additional other elements. Operations performed by modules, program modules, or other components in accordance with various embodiments may be performed in a sequential, parallel, iterative, or heuristic manner. Also, some operations may be performed in a different order, omitted, or other operations may be added. And the embodiments disclosed in this document are presented for the purpose of explanation and understanding of the disclosed technology and do not limit the scope of the technology described in this document. Accordingly, the scope of this document should be interpreted to include all modifications based on the technical idea of this document or various other embodiments.

Claims (20)

In an electronic device,
A display unit; And
The processor comprising:
Acquiring camera information including image information photographed by an external photographing apparatus and information relating to a photographing posture of the external photographing apparatus,
Generating an image to be displayed on the electronic device using the acquired image information,
Correcting the generated image using the obtained camera information,
And to display the corrected image on the display unit. The method according to claim 1,
Wherein the information related to the photographing posture is information on a pitch angle, a roll angle, and a yaw angle sensed at the time of photographing the image in the external photographing apparatus. The method according to claim 1,
And displays a part of the view area corresponding to a user interest area (FOV) when displaying the corrected image on the display unit. The method of claim 3,
And transmits information on the viewpoint adjustment event to the external photographing apparatus when a preset viewpoint adjustment event occurs while displaying the corrected image on the display unit. The method according to claim 3 or 4,
Wherein the correcting of the generated image using the obtained camera information is performed by rotating, moving up and down, and moving left and right so as to cancel the positional variation of the image according to the photographing posture, Electronic device. 5. The image capturing apparatus according to any one of claims 1 to 4, wherein the camera information includes information on movement of the external photographing apparatus,
Wherein the electronic apparatus further corrects the generated image by further using information about movement of the external photographing apparatus when correcting the generated image using the obtained camera information. . The method according to claim 1,
Wherein the image information is a merged image of a plurality of images obtained by a plurality of cameras. 8. The method of claim 7,
Wherein the electronic device generates an image to be displayed by stitching the plurality of images into one wide angle image. 8. The method according to claim 7, wherein the camera information includes unique photographing characteristic information for each of the plurality of cameras,
Wherein the electronic device corrects the shift of the shooting axis of the plurality of images when stitching the plurality of images into one wide angle image. The method of claim 7, wherein the unique photographing characteristic information for each of the plurality of cameras is information on a pitch angle, a roll angle, and a yaw angle with respect to an installation state of each of the plurality of cameras The electronic device comprising: A method of controlling an electronic device,
Acquiring camera information including image information photographed by an external photographing apparatus and information relating to a photographing posture of the external photographing apparatus;
Generating an image to be displayed on the electronic device using the acquired image information,
Correcting the generated image using the obtained camera information,
And displaying the corrected image. ≪ Desc / Clms Page number 20 > 12. The method of claim 11,
Wherein the information related to the photographing posture is information about a pitch angle, a roll angle, and a yaw angle sensed at the time of photographing the image in the external photographing apparatus, . 12. The method of claim 11,
And displaying a part of the view area corresponding to a user interest area (FOV) when displaying the corrected image. 14. The method of claim 13,
Wherein when the preset view point adjustment event occurs while displaying the corrected image on the display unit, information on the view point adjustment event is transmitted to the external photographing apparatus . The method according to claim 13 or 14,
Wherein the correcting of the generated image using the obtained camera information is performed by rotating, moving up and down, and moving left and right so as to cancel the positional variation of the image according to the photographing posture, A method of controlling an electronic device. 15. The apparatus according to any one of claims 11 to 14, wherein the camera information includes information on movement of the external photographing apparatus,
Wherein the control unit further corrects the generated image by further using information about movement of the external photographing apparatus when the generated image is corrected using the obtained camera information. 12. The method of claim 11,
Wherein the image information is an image obtained by merging images obtained by a plurality of cameras. 18. The method of claim 17,
Wherein the image to be displayed is generated by stitching the plurality of images into one wide angle image. The method of claim 17, wherein the camera information includes unique photographing characteristic information for each of the plurality of cameras,
Wherein the electronic device corrects a deviation of the shooting axis of the plurality of images when stitching the plurality of images into one wide angle image. 18. The method of claim 17,
Wherein the unique photographing characteristic information for each of the plurality of cameras is information on a pitch angle, a roll angle, and a yaw angle with respect to an installation state of each of the plurality of cameras, / RTI >

Images