KR20180036359A - Method for displaying an image and an electronic device thereof - Google Patents

Abstract

Provided are a method for displaying an image and an electronic device thereof. According to an embodiment of the present invention, an operating method of the electronic device comprises the following steps of: generating a plurality of image frames captured by a plurality of cameras, respectively; converting the image frames based on at least one virtual camera located at a different position from the cameras; generating at least one matching image from the converted image frames; and displaying part of the at least one matching image. According to the present invention, by applying an omni-directional stereoscopic (ODS) projection as described above, it is possible to eliminate stitching errors due to parallax in a stitched panoramic image, and to allow a steric area and a flat area to continuously change in a panoramic image.

Description

[0001] METHOD FOR DISPLAYING AN IMAGE AND AN ELECTRONIC DEVICE THEREOF [0002]

The present invention relates to an electronic device, and more particularly, to a method and apparatus for displaying an image in an electronic device.

The spread and use of electronic devices (for example, mobile terminals) is rapidly increasing owing to the remarkable development of information communication technology and semiconductor technology. BACKGROUND OF THE INVENTION As electronic devices become widespread, electronic devices provide a variety of content to users.

Recently, various electronic devices in a form that can be directly worn on the body are being developed. Such wearable electronic devices include, for example, a head-mounted device (HMD), a smart glass, a smart watch, a smart band, a contact lens- , Ring devices, shoe devices, garment devices, glove devices, and the like. Such a wearable electronic device is directly worn on the body or clothes, so that portability and user's accessibility can be dramatically improved.

Among various examples of the wearable electronic device, the HMD which can be mounted on the head of the user can provide a 360-degree panoramic image at a position close to the user's eyes, thereby providing a realistic image to the user.

Various embodiments of the present invention provide a method and an electronic device for displaying a panoramic image.

Various embodiments of the present invention provide a method and an electronic device for stitching image frames to produce a 360 degree rotated image.

Various embodiments of the present invention provide a method and an electronic device for converting an image frame captured by a plurality of cameras as if captured by a virtual camera.

Various embodiments of the present invention provide a method and an electronic device for warping rays in accordance with an omni-directional stereoscopic (ODS) projection.

The various embodiments of the present invention provide a method and apparatus for capturing a warped ray in terms of a virtual camera.

An operation method of an electronic device according to an embodiment of the present invention includes: generating a plurality of image frames captured by each of the plurality of cameras; Generating at least one registration image from the transformed plurality of image frames, and displaying the portion of the at least one registered image.

An electronic device according to another embodiment of the present invention includes an image generation unit having a plurality of cameras and generating a plurality of image frames captured by each of the plurality of cameras; A controller for converting the plurality of image frames based on at least one virtual camera located and generating at least one registration image from the converted plurality of image frames, a display for displaying a part of the at least one registered image, .

By applying the omni-directional stereoscopic projection as described above, it is possible to eliminate the stitching error due to parallax in the stitched panoramic image, and the steric area and the flat area can continuously change in the panoramic image have.

Figure 1 illustrates a network environment including an electronic device according to various embodiments of the present invention.
Figure 2 shows a block diagram of an electronic device according to various embodiments of the present invention.
3 is a block diagram of a program module in accordance with various embodiments of the present invention.
4A illustrates a camera module according to various embodiments of the present invention.
4B shows a first frontal structure of a camera module according to various embodiments of the present invention.
4C shows a second frontal structure of a camera module according to various embodiments of the present invention.
Figure 4d shows a third frontal structure of the camera module according to various embodiments of the present invention.
4E illustrates a fourth frontal view of a camera module according to various embodiments of the present invention.
Figure 4f illustrates a fifth frontal structure of a camera module according to various embodiments of the present invention.
Figure 4G shows a sixth frontal structure of a camera module according to various embodiments of the present invention.
Figure 4h illustrates a side view of a camera module according to various embodiments of the present invention.
Figure 5 illustrates schematically modules of an electronic device according to various embodiments of the present invention.
Figure 6 illustrates a functional block structure of an electronic device according to various embodiments of the present invention.
Figure 7 illustrates the rails of a camera module according to various embodiments of the present invention.
Figure 8 shows an image captured by a camera module with a plurality of cameras in accordance with various embodiments of the present invention.
9 is a diagram illustrating a parallax problem occurring after stitching of image frames in accordance with various embodiments of the present invention.
Figure 10 illustrates warping of lasers according to various embodiments of the present invention.
Figure 11 illustrates laps after warping in accordance with various embodiments of the present invention.
Figure 12 shows the location of a view point in accordance with various embodiments of the present invention.
Figure 13 illustrates a method of warping a ray based on depth values in accordance with various embodiments of the present invention.
Figure 14 illustrates a method of capturing an image using a virtual camera in accordance with various embodiments of the present invention.
FIG. 15 is a flow diagram illustrating a registration image generated using virtual cameras in accordance with various embodiments of the present invention.
16 is a flow chart for transforming an image based on warping in accordance with various embodiments of the present invention.
17 is a diagram for calculating the depth value of an image pixel in accordance with various embodiments of the present invention.
18 is a flow chart for performing camera calibration in accordance with various embodiments of the present invention.
Figure 19 is a flowchart of stitching image frames in accordance with various embodiments of the present invention.
Figure 20 shows an image matched in accordance with various embodiments of the present invention.
Figure 21 illustrates the relationship between elevation angles of cameras and distances of view of cameras according to various embodiments of the present invention.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. It is to be understood that the embodiments and terminologies used herein are not intended to limit the invention to the particular embodiments described, but to include various modifications, equivalents, and / or alternatives of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar components. The singular expressions may include plural expressions unless the context clearly dictates otherwise. In this document, the expressions "A or B" or "at least one of A and / or B" and the like may include all possible combinations of the items listed together. Expressions such as " first, "" second," " first, "or" second, " But is not limited to those components. When it is mentioned that some (e.g., first) component is "(functionally or communicatively) connected" or "connected" to another (second) component, May be connected directly to the component, or may be connected through another component (e.g., a third component).

In this document, the phrase "configured to A to B" may be used to designate an A to B capability, for example, hardware or software, Quot ;, "modified from A to B," " made from A to B, " In some circumstances, the expression "a device configured to perform an A to B" may mean that the device may be "A to B" with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be implemented by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) , And a general purpose processor (e.g., a CPU or an application processor) capable of performing the corresponding operations.

Electronic devices in accordance with various embodiments of the present document may be used in various applications such as, for example, smart phones, tablet PCs, mobile phones, videophones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, PDAs, a portable multimedia player, an MP3 player, a medical device, a camera, or a wearable device. Wearable devices may be of the type of accessories (eg, watches, rings, bracelets, braces, necklaces, glasses, contact lenses or head-mounted-devices (HMD) (E.g., a skin-pads or tattoo), or a bio-implantable circuit. In some embodiments, the electronic device may be, for example, a television, a digital video disk (Such as Samsung HomeSyncTM, Apple TVTM, or Google TVTM), game consoles (such as air conditioners, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air purifiers, set top boxes, home automation control panels, E.g., Xbox (TM), PlayStation (TM)), electronic dictionary, electronic key, camcorder, or electronic frame.

In an alternative embodiment, the electronic device may be any of a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a body temperature meter), magnetic resonance angiography (MRA) A navigation system, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), an automobile infotainment device, a marine electronic equipment (For example, marine navigation systems, gyro compasses, etc.), avionics, security devices, head units for vehicles, industrial or domestic robots, drones, ATMs at financial institutions, of at least one of the following types of devices: a light bulb, a fire detector, a fire alarm, a thermostat, a streetlight, a toaster, a fitness device, a hot water tank, a heater, a boiler, . According to some embodiments, the electronic device may be a piece of furniture, a building / structure or part of an automobile, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., Gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic device is flexible or may be a combination of two or more of the various devices described above. The electronic device according to the embodiment of the present document is not limited to the above-described devices. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

Hereinafter, terms used in this document are defined as follows.

'Field of view (FOV)' means a range of angles that one camera can observe. The angular range may be determined as an angular range in the horizontal direction and an angular range in the vertical direction with respect to the optical axis of the camera.

'Image frame' means images that are the basis for creating panoramic images. In other words, a plurality of image frames may be matched to produce a single panoramic image. Each image frame can be generated by different cameras or by capturing at a certain time interval while one camera is moving. Adjacent image frames to generate a panoramic image among a plurality of image frames include overlapping objects. In some cases, an image frame may be referred to variously as an 'image', a 'base image', a 'raw image', and the like.

&Quot; Image stitching " or " stitching " refers to an overall operation for creating a panoramic image from a plurality of image frames. For example, image stitching may be performed by matching feature points extracted from a plurality of image frames to align the image frames so that the common areas overlap, determine boundaries in the overlapping areas of the aligned image frames, Exposure to the image, color compensation, blending, and so on. In some cases, stitching can be referred to variously as 'combination', 'synthesis', 'combination', 'match', and the like.

Referring to Figure 1, in various embodiments, an electronic device 101 in a network environment 100 is described. The electronic device 101 may include a bus 110, a processor 120, a memory 130, an input / output interface 150, a display 160, and a communication interface 170. In some embodiments, the electronic device 101 may omit at least one of the components or additionally include other components. The bus 110 may include circuitry to connect the components 110-170 to one another and to communicate communications (e.g., control messages or data) between the components. Processor 120 may include one or more of a central processing unit, an application processor, or a communications processor (CP). The processor 120 may perform computations or data processing related to, for example, control and / or communication of at least one other component of the electronic device 101.

According to various embodiments, the processor 120 may convert image frames according to an omni-directional stereoscopic (ODS) projection. In addition, the processor 120 may stitch the transformed image frames to produce a panoramic image.

Memory 130 may include volatile and / or non-volatile memory. Memory 130 may store instructions or data related to at least one other component of electronic device 101, for example. According to one embodiment, the memory 130 may store software and / or programs 140. The program 140 may include, for example, a kernel 141, a middleware 143, an application programming interface (API) 145, and / or an application program . At least some of the kernel 141, middleware 143, or API 145 may be referred to as an operating system. The kernel 141 may include system resources used to execute an operation or function implemented in other programs (e.g., middleware 143, API 145, or application program 147) (E.g., bus 110, processor 120, or memory 130). The kernel 141 also provides an interface to control or manage system resources by accessing individual components of the electronic device 101 in the middleware 143, API 145, or application program 147 .

According to various embodiments, the memory 130 may store image frames and may store panorama images generated from image frames. In addition, the memory 130 may store parameters required by the camera to capture an image.

The middleware 143 can perform an intermediary role such that the API 145 or the application program 147 can communicate with the kernel 141 to exchange data. In addition, the middleware 143 may process one or more task requests received from the application program 147 according to the priority order. For example, middleware 143 may use system resources (e.g., bus 110, processor 120, or memory 130, etc.) of electronic device 101 in at least one of application programs 147 Prioritize, and process the one or more task requests. The API 145 is an interface for the application 147 to control the functions provided by the kernel 141 or the middleware 143. The API 145 is an interface for controlling the functions provided by the application 141. For example, An interface or a function (e.g., a command). Output interface 150 may be configured to communicate commands or data entered from a user or other external device to another component (s) of the electronic device 101, or to another component (s) of the electronic device 101 ) To the user or other external device.

The display 160 may include a display such as, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical system (MEMS) display, or an electronic paper display . Display 160 may display various content (e.g., text, images, video, icons, and / or symbols, etc.) to a user, for example. Display 160 may include a touch screen and may receive a touch, gesture, proximity, or hovering input using, for example, an electronic pen or a portion of the user's body.

According to various embodiments, the display 160 may display a panoramic image generated by stitching a plurality of image frames.

The communication interface 170 establishes communication between the electronic device 101 and an external device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106) . For example, communication interface 170 may be connected to network 162 via wireless or wired communication to communicate with an external device (e.g., second external electronic device 104 or server 106).

The wireless communication may include, for example, LTE, LTE-A (LTE Advance), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) System for Mobile Communications), and the like. According to one embodiment, the wireless communication may be wireless communication, such as wireless fidelity (WiFi), Bluetooth, Bluetooth low power (BLE), Zigbee, NFC, Magnetic Secure Transmission, Frequency (RF), or body area network (BAN). According to one embodiment, the wireless communication may comprise a GNSS. GNSS may be, for example, Global Positioning System (GPS), Global Navigation Satellite System (Glonass), Beidou Navigation Satellite System (Beidou) or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" can be used interchangeably with "GNSS ". The wired communication may include, for example, at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), a power line communication or a plain old telephone service have. Network 162 may include at least one of a telecommunications network, e.g., a computer network (e.g., LAN or WAN), the Internet, or a telephone network.

Each of the first and second external electronic devices 102, 104 may be the same or a different kind of device as the electronic device 101. According to various embodiments, all or a portion of the operations performed in the electronic device 101 may be performed in one or more other electronic devices (e.g., electronic devices 102, 104, or server 106). According to the present invention, when electronic device 101 is to perform a function or service automatically or on demand, electronic device 101 may perform at least some functions associated therewith instead of, or in addition to, (E.g., electronic device 102, 104, or server 106) may request the other device (e.g., electronic device 102, 104, or server 106) Perform additional functions, and forward the results to the electronic device 101. The electronic device 101 may process the received results as is or additionally to provide the requested functionality or services. For example, Cloud computing, distributed computing, or client-server computing technology may be utilized.

2 is a block diagram of an electronic device 201 according to various embodiments. The electronic device 201 may include all or part of the electronic device 101 shown in Fig. 1, for example. The electronic device 201 may include one or more processors (e.g., AP) 210, a communications module 220, a subscriber identification module 224, a memory 230, a sensor module 240, an input device 250, An interface 270, an audio module 280, a camera module 291, a power management module 295, a battery 296, an indicator 297, and a motor 298. The processor 290, The processor 210 may be, for example, an operating system or an application program to control a plurality of hardware or software components coupled to the processor 210, and to perform various data processing and operations. ) May be implemented, for example, as a system on chip (SoC). According to one embodiment, the processor 210 may further include a graphics processing unit (GPU) and / or an image signal processor. The cellular module 210 may include at least some of the components shown in Figure 2 (e.g., the cellular module 221) The processor 210 may load and process instructions or data received from at least one of the other components (e.g., non-volatile memory) into the volatile memory and store the resulting data in a non-volatile memory.

According to various embodiments, the processor 210 may convert image frames according to an omni-directional stereoscopic (ODS) projection. In addition, the processor 120 may stitch the transformed image frames to produce a panoramic image.

May have the same or similar configuration as communication module 220 (e.g., communication interface 170). The communication module 220 may include, for example, a cellular module 221, a WiFi module 223, a Bluetooth module 225, a GNSS module 227, an NFC module 228 and an RF module 229 have. The cellular module 221 can provide voice calls, video calls, text services, or Internet services, for example, over a communication network. According to one embodiment, the cellular module 221 may utilize a subscriber identity module (e.g., a SIM card) 224 to perform the identification and authentication of the electronic device 201 within the communication network. According to one embodiment, the cellular module 221 may perform at least some of the functions that the processor 210 may provide. According to one embodiment, the cellular module 221 may comprise a communications processor (CP). At least some (e.g., two or more) of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228, according to some embodiments, (IC) or an IC package. The RF module 229 can, for example, send and receive communication signals (e.g., RF signals). The RF module 229 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228 transmits / receives an RF signal through a separate RF module . The subscriber identification module 224 may include, for example, a card or an embedded SIM containing a subscriber identity module, and may include unique identification information (e.g., ICCID) or subscriber information (e.g., IMSI (international mobile subscriber identity).

Memory 230 (e.g., memory 130) may include, for example, internal memory 232 or external memory 234. Volatile memory (e.g., a DRAM, an SRAM, or an SDRAM), a non-volatile memory (e.g., an OTPROM, a PROM, an EPROM, an EEPROM, a mask ROM, a flash ROM , A flash memory, a hard drive, or a solid state drive (SSD). The external memory 234 may be a flash drive, for example, a compact flash (CF) ), Micro-SD, Mini-SD, extreme digital (xD), multi-media card (MMC), or memory stick, etc. External memory 234 may communicate with electronic device 201, Or may be physically connected.

According to various embodiments, the memory 230 may store image frames and may store a panorama image generated from the image frames. In addition, the memory 230 may store parameters required by the camera to capture an image.

The sensor module 240 may, for example, measure a physical quantity or sense the operating state of the electronic device 201 to convert the measured or sensed information into an electrical signal. The sensor module 240 includes a gesture sensor 240A, a gyro sensor 240B, an air pressure sensor 240C, a magnetic sensor 240D, an acceleration sensor 240E, a grip sensor 240F, A temperature sensor 240G, a UV sensor 240G, a color sensor 240H (e.g., an RGB (red, green, blue) sensor), a living body sensor 240I, And a sensor 240M. Additionally or alternatively, the sensor module 240 may be configured to perform various functions such as, for example, an e-nose sensor, an electromyography (EMG) sensor, an electroencephalograph (EEG) sensor, an electrocardiogram An infrared (IR) sensor, an iris sensor, and / or a fingerprint sensor. The sensor module 240 may further include a control circuit for controlling at least one or more sensors belonging to the sensor module 240. In some embodiments, the electronic device 201 further includes a processor configured to control the sensor module 240, either as part of the processor 210 or separately, so that while the processor 210 is in a sleep state, The sensor module 240 can be controlled.

The input device 250 may include, for example, a touch panel 252, a (digital) pen sensor 254, a key 256, or an ultrasonic input device 258. As the touch panel 252, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type can be used. Further, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile response to the user. (Digital) pen sensor 254 may be part of, for example, a touch panel or may include a separate recognition sheet. Key 256 may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device 258 can sense the ultrasonic wave generated by the input tool through the microphone (e.g., the microphone 288) and confirm the data corresponding to the ultrasonic wave detected.

Display 260 (e.g., display 160) may include panel 262, hologram device 264, projector 266, and / or control circuitry for controlling them. The panel 262 may be embodied, for example, flexibly, transparently, or wearably. The panel 262 may comprise a touch panel 252 and one or more modules. According to one embodiment, the panel 262 may include a pressure sensor (or force sensor) capable of measuring the intensity of the pressure on the user's touch. The pressure sensor may be integrated with the touch panel 252 or may be implemented by one or more sensors separate from the touch panel 252. The hologram device 264 can display a stereoscopic image in the air using interference of light. The projector 266 can display an image by projecting light onto a screen. The screen may be located, for example, inside or outside the electronic device 201.

According to various embodiments, the display 260 may display a panoramic image generated by stitching a plurality of image frames.

The interface 270 may include, for example, an HDMI 272, a USB 274, an optical interface 276, or a D-sub (D-subminiature) 278. The interface 270 may, for example, be included in the communication interface 170 shown in FIG. Additionally or alternatively, the interface 270 may include, for example, a mobile high-definition link (MHL) interface, an SD card / multi-media card (MMC) interface, or an infrared data association have.

The audio module 280 can, for example, convert sound and electrical signals in both directions. At least some of the components of the audio module 280 may be included, for example, in the input / output interface 145 shown in FIG. The audio module 280 may process sound information input or output through, for example, a speaker 282, a receiver 284, an earphone 286, a microphone 288, or the like. The camera module 291 is, for example, a device capable of capturing still images and moving images, and according to one embodiment, one or more image sensors (e.g., a front sensor or a rear sensor), a lens, an image signal processor (ISP) , Or flash (e.g., an LED or xenon lamp, etc.).

According to various embodiments, camera module 2910 may capture image frames that are the basis for generating a panoramic image.

The power management module 295 can, for example, manage the power of the electronic device 201. [ According to one embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charging IC, or a battery or fuel gauge. The PMIC may have a wired and / or wireless charging scheme. The wireless charging scheme may include, for example, a magnetic resonance scheme, a magnetic induction scheme, or an electromagnetic wave scheme, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonant circuit, have. The battery gauge can measure, for example, the remaining amount of the battery 296, the voltage during charging, the current, or the temperature. The battery 296 may include, for example, a rechargeable battery and / or a solar cell.

The indicator 297 may indicate a particular state of the electronic device 201 or a portion thereof (e.g., processor 210), e.g., a boot state, a message state, or a state of charge. The motor 298 can convert the electrical signal to mechanical vibration, and can generate vibration, haptic effects, and the like. The electronic device 201 is a mobile TV support device capable of processing media data conforming to specifications such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or media flow (TM) GPU). Each of the components described in this document may be composed of one or more components, and the name of the component may be changed according to the type of the electronic device. In various embodiments, an electronic device (e. G., Electronic device 201) may have some components omitted, further include additional components, or some of the components may be combined into one entity, The functions of the preceding components can be performed in the same manner.

3 is a block diagram of a program module according to various embodiments. According to one embodiment, program module 310 (e.g., program 140) includes an operating system that controls resources associated with an electronic device (e.g., electronic device 101) and / E.g., an application program 147). The operating system may include, for example, AndroidTM, iOSTM, WindowsTM, SymbianTM, TizenTM, or BadaTM. 3, program module 310 includes a kernel 320 (e.g., kernel 141), middleware 330 (e.g., middleware 143), API 360 (e.g., API 145) ), And / or an application 370 (e.g., an application program 147). At least a portion of the program module 310 may be preloaded on an electronic device, 102 and 104, a server 106, and the like).

The kernel 320 may include, for example, a system resource manager 321 and / or a device driver 323. The system resource manager 321 can perform control, allocation, or recovery of system resources. According to one embodiment, the system resource manager 321 may include a process manager, a memory manager, or a file system manager. The device driver 323 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, or an inter-process communication . The middleware 330 may provide various functions through the API 360, for example, to provide functions that are commonly needed by the application 370 or allow the application 370 to use limited system resources within the electronic device. Application 370 as shown in FIG. According to one embodiment, the middleware 330 includes a runtime library 335, an application manager 341, a window manager 342, a multimedia manager 343, a resource manager 344, a power manager 345, a database manager The location manager 350, the graphic manager 351, or the security manager 352. In this case, the service manager 341 may be a service manager, a service manager, a service manager, a package manager 346, a package manager 347, a connectivity manager 348, a notification manager 349,

The runtime library 335 may include, for example, a library module that the compiler uses to add new functionality via a programming language while the application 370 is executing. The runtime library 335 may perform input / output management, memory management, or arithmetic function processing. The application manager 341 can manage the life cycle of the application 370, for example. The window manager 342 can manage GUI resources used in the screen. The multimedia manager 343 can recognize the format required for reproducing the media files and can perform encoding or decoding of the media file using a codec according to the format. The resource manager 344 can manage the source code of the application 370 or the space of the memory. The power manager 345 may, for example, manage the capacity or power of the battery and provide the power information necessary for operation of the electronic device. According to one embodiment, the power manager 345 may interoperate with a basic input / output system (BIOS). The database manager 346 may create, retrieve, or modify the database to be used in the application 370, for example. The package manager 347 can manage installation or update of an application distributed in the form of a package file.

The connectivity manager 348 may, for example, manage the wireless connection. The notification manager 349 may provide the user with an event such as, for example, an arrival message, an appointment, a proximity notification, and the like. The location manager 350 can manage the location information of the electronic device, for example. The graphic manager 351 may, for example, manage the graphical effects to be presented to the user or a user interface associated therewith. Security manager 352 may provide, for example, system security or user authentication. According to one embodiment, the middleware 330 may include a telephony manager for managing the voice or video call function of the electronic device, or a middleware module capable of forming a combination of the functions of the above-described components . According to one embodiment, the middleware 330 may provide a module specialized for each type of operating system. Middleware 330 may dynamically delete some existing components or add new ones. The API 360 may be provided in a different configuration depending on the operating system, for example, as a set of API programming functions. For example, for Android or iOS, you can provide a single API set for each platform, and for Tizen, you can provide two or more API sets for each platform.

The application 370 may include a home 371, a dialer 372, an SMS / MMS 373, an instant message 374, a browser 375, a camera 376, an alarm 377, Contact 378, voice dial 379, email 380, calendar 381, media player 382, album 383, watch 384, healthcare (e.g., measuring exercise or blood glucose) , Or environmental information (e.g., air pressure, humidity, or temperature information) application. According to one embodiment, the application 370 may include an information exchange application capable of supporting the exchange of information between the electronic device and the external electronic device. The information exchange application may include, for example, a notification relay application for communicating specific information to an external electronic device, or a device management application for managing an external electronic device. For example, the notification delivery application can transmit notification information generated in another application of the electronic device to the external electronic device, or receive notification information from the external electronic device and provide the notification information to the user. The device management application may, for example, control the turn-on / turn-off or brightness (or resolution) of an external electronic device in communication with the electronic device (e.g., the external electronic device itself Control), or install, delete, or update an application running on an external electronic device. According to one embodiment, the application 370 may include an application (e.g., a healthcare application of a mobile medical device) designated according to the attributes of the external electronic device. According to one embodiment, the application 370 may include an application received from an external electronic device. At least some of the program modules 310 may be implemented (e.g., executed) in software, firmware, hardware (e.g., processor 210), or a combination of at least two of the same, Program, routine, instruction set or process.

As used herein, the term "module " includes units comprised of hardware, software, or firmware and may be used interchangeably with terms such as, for example, logic, logic blocks, components, or circuits. A "module" may be an integrally constructed component or a minimum unit or part thereof that performs one or more functions. "Module" may be implemented either mechanically or electronically, for example, by application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs) And may include programmable logic devices. At least some of the devices (e.g., modules or functions thereof) or methods (e.g., operations) according to various embodiments may be stored in a computer readable storage medium (e.g., memory 130) . ≪ / RTI > When the instruction is executed by a processor (e.g., processor 120), the processor may perform a function corresponding to the instruction. The computer-readable recording medium may be a hard disk, a floppy disk, a magnetic medium such as a magnetic tape, an optical recording medium such as a CD-ROM, a DVD, a magnetic-optical medium such as a floppy disk, The instructions may include code that is generated by the compiler or code that may be executed by the interpreter. Modules or program modules according to various embodiments may include at least one or more of the components described above Operations that are performed by modules, program modules, or other components, in accordance with various embodiments, may be performed in a sequential, parallel, iterative, or heuristic manner, or at least in part Some operations may be executed in a different order, omitted, or other operations may be added.

A panoramic image may be generated in accordance with various embodiments of the present invention. What is needed is an apparatus for generating at least one image frame to generate a panoramic image. The apparatus for generating at least one image frame comprises at least one camera, and preferably comprises two or more cameras. One example of an apparatus for generating at least one image frame will now be described with reference to Figures 4A to 4H.

4A illustrates a camera module 410 in accordance with various embodiments of the present invention.

4A, the camera module 410 includes a plurality of cameras 430-1, 430-2, 430-3, and 430-4 disposed on the side and a camera 420 disposed on the upper surface thereof . Although not shown, the camera module 410 may further include additional cameras other than the plurality of cameras 430-1, 430-2, 430-3, and 430-4. In addition, the arrangement and number of cameras shown in FIG. 4A are exemplary, and the camera module may include other arrangements and a different number of cameras. Each camera recognizes a ray incident within its FOV as an image sensor and can generate an image frame. The detailed structure of the camera module 410 is as shown in Figs. 4B to 4G and 4H, and the camera module 410 may be approximated as a cylinder having a radius of R and a height of 2h. According to various embodiments of the present invention, the height of the camera module 410 may be regarded as h rather than actual height 2h for convenience.

4B shows a first frontal structure of a camera module according to various embodiments of the present invention. 4B illustrates the side cameras of the camera module 410 including the cameras 430-1, 430-2, 430-3, and 430-4. Illustratively, according to FIG. 4B, the camera module 410 includes 16 side cameras. The sixteen side cameras include eight left cameras L ₁ , L ₂ , ..., L ₈ and eight right cameras R ₁ , R ₂ , ..., R ₈ can do. Each of the eight left cameras L ₁ through L ₈ corresponds to each of the eight right cameras R ₁ through R ₈ to produce a stereoscopic image. That is, the left panoramic image of the stereoscopic panoramic image is generated from the image frames captured by the left cameras L ₁ to L ₈ , and the right panoramic image is generated from the image frames captured by the right cameras R ₁ to R ₈ do. The pair of cameras marked with the same number in each of the left and right cameras represents the corresponding cameras. Hereinafter, the pair of corresponding cameras in this disclosure is defined as a 'stereo pair'. Accordingly, the camera module 410 according to FIG. 4B has eight stereo pair cameras. For example, the camera L- ₁ 430-4 constitutes a stereoscopic pair with the camera R ₂ 430-1, and the cameras constituting the stereoscopic pair are disposed on the same plane such that the optical axes thereof are parallel.

As shown, camera L ₁ 430-4 and camera R ₁ 430-1 can capture light rays reflected from the same object. At this time, since the camera L ₁ (430-4) and the camera R ₁ (430-1) are located at different places, the center of the camera L ₁ (430-4) and the camera R ₁ And the dispairity of the image is generated. The perspective of the object is identified from the image frames generated based on the angle of view difference between the camera L ₁ 430-4 and the camera R ₁ 430-1 such that the perspective of the object is identified by the person's binocular . In other words, the cameras forming the stereopair can generate image frames in which the perspective of the object can be identified, and can generate a stereoscopic image based on the perspective of the object. As an example, an image captured by the camera L ₁ 430-4 is displayed to be identified through the user's left eye, and a corresponding image captured by the camera R ₁ 430-1 is identified through the user's right eye When displayed, the user can enjoy the stereoscopic image.

The cameras L ₁ to L ₈ are arranged in a circle with respect to the center of the camera module 410 as shown in FIG. 4B. Accordingly, the cameras L ₁ through L ₈ can capture light rays reflected from all directions with respect to the camera module 410, and the image frames generated from the cameras L ₁ through L ₈ are stitched to produce a 360-degree panoramic image Can be generated. Cameras R ₁ through R ₈ can also capture light rays that are reflected in all directions with respect to the camera module 410, The image frames generated from R ₁ through R ₈ may be stitched to produce a 360 degree panoramic image. Hereinafter, a 360-degree panorama generated from the left cameras L ₁ to L ₈ is defined as a 'left panorama', and a 360-degree panorama generated from the right cameras R ₁ to R ₈ is defined as a right panorama.

In the cameras L ₁ to L ₈ and R ₁ to R ₈ , the cameras constituting the stereopair are arranged at a constant distance in the same plane. For example, a camera 420 is disposed on the upper surface of each of the cameras L ₁ to L ₈ and R ₁ to R ₈ and the camera module 410.

Side cameras (L ₁ to L ₈ , R ₁ to R ₈ ) disposed in the camera module 410 can capture light rays in all directions incident in the horizontal direction. However, for the vertical direction, there is an area that side cameras can not capture due to the limitation of the FOV. Thus, the 360 degree panoramic image generated from the image frames captured from the side cameras does not cover the area beyond the FOV of the side camera. As such, the camera 420 disposed on the top surface is arranged to cover an area in the vertical direction that the side cameras can not capture. According to Fig. 4H, the camera 420 is disposed at a distance h from the camera center. Also, the center-to-center distance R in each side camera may be 9.1 cm. Each of the cameras L ₁ to L ₈ and R ₁ to R ₈ is rotated such that the optical axis is at a constant angle? In the direction of the center of the camera from the center of the camera module (hereinafter referred to as a radial line direction) . Specifically, each of the left cameras L ₁ to L ₈ has its optical axis rotated clockwise by an angle? In the direction of the center line, and each right camera R ₁ through R ₈ has its optical axis rotated by a in the counterclockwise direction in the direction of the center line . As an example,? = 33.75 ?. However, the values of the parameters R and alpha of the camera module 410 described above are exemplary, and the camera module 410 may have other values of parameters.

The configuration of the camera module 410 shown in FIG. 4B is exemplary, and the camera module 410 may have a configuration different from that shown in FIG. 4B. 4B, the camera module 410 includes eight left cameras L ₁ , L ₂ , ..., L ₈ and eight right cameras R ₁ , R ₂ , ..., R ₈ , But this is exemplary and the camera module 410 may include various numbers of side cameras. The configuration of the camera module 410 including the various numbers of side cameras is illustratively shown in Figures 4C-4G.

The configuration of the camera module 410 according to various embodiments may include four side cameras as shown in FIG. 4C. 4C shows a second frontal structure of a camera module according to various embodiments of the present invention. According to FIG. 4C, the four side cameras may include two left cameras L ₁ and L ₂ and two right cameras R _{1 and} R ₂ . The horizontal FOV of each camera may be greater than 180 degrees. The pair of cameras marked with the same number in the left cameras and the right cameras can constitute stereoscopic pairs. Accordingly, when the camera module 410 has the configuration as shown in FIG. 4C, the camera module 410 can generate a 360-degree stereoscopic panoramic image using two stereoscopic pair cameras.

The configuration of the camera module 410 according to various embodiments may include six side cameras as shown in FIG. 4D. Figure 4d shows a third frontal structure of the camera module according to various embodiments of the present invention. According to Fig. 4D, the six side cameras may include three left cameras L ₁ , L ₂ and L ₃ and three right cameras R ₁ R ₂ and R ₃ . The horizontal FOV of each camera may be greater than 120 °. The pair of cameras marked with the same number in the left cameras and the right cameras can constitute stereoscopic pairs. Accordingly, when the camera module 410 has the configuration as shown in FIG. 4D, the camera module 410 can generate a 360-degree stereoscopic panoramic image using three stereoscopic pair cameras.

The configuration of the camera module 410 according to various embodiments may include eight side cameras as shown in FIG. 4E. 4E illustrates a fourth frontal view of a camera module according to various embodiments of the present invention. 4E, the eight side cameras include four left cameras L ₁ , L ₂ , L ₃ and L ₄ and three right cameras R ₁ R ₂ , R ₃ and R ₄ can do. The horizontal FOV of each camera may be greater than 90 °. The pair of cameras marked with the same number in the left cameras and the right cameras can constitute stereoscopic pairs. Accordingly, when the camera module 410 has the configuration as shown in FIG. 4E, the camera module 410 can generate a 360-degree stereoscopic panoramic image using four stereoscopic pair cameras.

The configuration of the camera module 410 according to various embodiments may include twelve side cameras as shown in FIG. 4f. Figure 4f illustrates a fifth frontal structure of a camera module according to various embodiments of the present invention. According to FIG. 4F, the twelve side cameras include six left cameras L ₁ , L ₂ , L ₃ , L ₄ , L _{5 and} L ₆ and six right cameras R ₁ R ₂ and R ₃ , R ₄ , R _5, R ₆ ). The horizontal FOV of each camera may be greater than 60 degrees. The pair of cameras marked with the same number in the left cameras and the right cameras can constitute stereoscopic pairs. Accordingly, when the camera module 410 has the configuration as shown in FIG. 4F, the camera module 410 can generate a 360-degree stereoscopic panoramic image using six stereoscopic pair cameras.

The configuration of the camera module 410 according to various embodiments may include ten side cameras as shown in FIG. 4G. Figure 4G shows a sixth frontal structure of a camera module according to various embodiments of the present invention. According to FIG. 4G, the 10 side cameras include five left cameras L ₁ , L ₂ , L ₃ , L ₄ and L ₅ and five right cameras R ₁ R ₂ , R ₃ , R ₄ , R ₅ ). The horizontal FOV of each camera may be greater than 72 °. The pair of cameras marked with the same number in the left cameras and the right cameras can constitute stereoscopic pairs. Accordingly, when the camera module 410 has the configuration as shown in FIG. 4G, the camera module 410 can generate a 360-degree stereoscopic panoramic image using five stereoscopic pair cameras.

The configuration of the camera module 410 shown in Figs. 4B to 4G is exemplary, and the configuration of the camera module 410 is not limited thereto. The camera module 410 may include various numbers of side cameras to produce a 360 degree stereoscopic panoramic image, and the arrangement of the cameras may also be different from that shown in Figures 4C-4G.

The camera module 410 may include a camera on the top side as well as side cameras. According to various embodiments of the present invention, one camera may be disposed on the top surface of the camera module 410 as shown in FIG. 4H. Figure 4h illustrates a side view of a camera module according to various embodiments of the present invention. According to FIG. 4H, the top camera 420 is disposed at a distance of h from the center. For example, h = 1.9 cm. However, this is exemplary and the parameter h may be another value. As for the structure of the camera module 410 shown in each of Figs. 4B to 4G, the top camera 420 may be disposed on the top surface of the camera module 410 as shown in Fig. 4H. The upper surface camera 420 may be disposed on the upper surface of the camera module 410 as shown in FIG. 4H even when the camera module 410 has a structure different from that shown in FIGS. 4B to 4G .

Since one camera 420 is disposed on the upper surface of the camera module 410 according to FIG. 4H, the perspective of the object due to the angle of view difference by at least two cameras can not be identified. In other words, the vertical image frame provided by the camera 420 corresponds to a 2D image, not a stereoscopic image. However, one camera 420 is disposed on the upper surface of the camera module 410, and an additional camera may be disposed on the upper surface of the camera module 410. In this case, the vertical image frame provided by the camera 420 and the additional camera may be stereoscopic.

FIG. 5 illustrates schematically modules of an electronic device 201 in accordance with various embodiments of the present invention. Fig. 5 shows an electronic device 201 configured with a combination of camera module 410, computing device 510, and display device 520. Fig.

The camera module 410 may have a structure as described in Figs. 4A to 4H, and may perform operations. In other words, the camera module 410 may capture image frames using a plurality of cameras disposed on the side and cameras disposed on the top side to generate a panoramic image.

The computing device 510 may perform operations on data received from the camera module 410 or data received from the display device 520. For example, the computing device 520 may stitch the image frames received from the camera module 410 to produce a panoramic image, and may transmit the generated panoramic image to the display device 520. The computing device 510 may also receive information about the user's input sensed through an input interface (e.g., a touchpad, button) of the display device 520 and may receive information from the display device 520 It is possible to receive the sensing information.

The display device 520 may provide the user with a visual image. For example, the display device may be a wearable device mounted on a user's head. In this case, the display device 520 may include a strap to be secured to the user's head. As another example, the display device 520 may be a device separate from the wearable device. In this case, the display device 520 may include a fastening unit 414 (e.g., a clip, a hole, a cover, etc.) for coupling with the wearable device and the display device 520 may be coupled to the wearable device It is possible to provide an image to the user in a state of being In addition, the display device 520 can sense the user's input through the input interface of the display device 520 and display an image corresponding to the sensed motion. In order to provide an image corresponding to the movement of the user, the display device 520 may include at least one sensor (not shown). For example, the at least one sensor may include an accelerometer, a GPS receiver, or at least one sensor for detecting other motion to obtain information related to the user's motion. Through at least one sensor, motion of the user's head can be detected.

For data exchange, camera module 410, computing device 510, and display device 520 may perform communications. For example, at least two of camera module 410, computing device 510, and display device 520 may communicate through a wired line (e.g., a cable) or may communicate over a wireless channel have. To this end, each of the camera module 410, the calculation device 510 and the display device 520 may further include a communication module for wireless communication.

In the embodiment described with reference to FIG. 5, the display device 520 may display the panoramic image received from the computing device 520. However, the display device 520 may receive image frames to generate a panoramic image, and may display the panoramic image that it has generated. To this end, the display 520 may include an independent processor for operation.

5, the electronic device 201 is a combination of a camera module 410, a computing device 510, and a display device 520. In the embodiment shown in FIG. However, according to another embodiment, electronic device 201 may include all of the components necessary for at least one of camera module 410, computing device 510 to display and compute images. The electronic device 201 may be a combination of the camera module 410, the computing device 510 and the display device 520 or may be a combination of the camera module 410, the computing device 510 and the display device 520. [ Device 520. < / RTI >

Figure 6 illustrates a functional block structure of an electronic device according to various embodiments of the present invention.

6, the electronic device 201 includes an image generation unit 610, a control unit 620, a display 630, and a storage unit 640.

The image generating unit 610 may be a device that can capture still images and moving images and may include one or more image sensors, a lens, an image signal processor (ISP), or a flash (e.g., LED or xenon lamp) . The image generation unit 610 may generate image frames for generating a panoramic image. To this end, the image generating unit 610 may capture an image frame using at least one camera. According to various embodiments, the image generator 610 may include a camera module 410. The image generation unit 610 can capture light beams incident on the camera module 410 in all directions using a plurality of cameras of the camera module 410 and can generate image frames for constructing a 360- Can be generated.

The control unit 620 may control the image generating unit 610, the display 630, and the storage unit 640 functionally combined with the control unit 620. In some embodiments, the controller 620 may include at least one microprocessor or microcontroller for control.

The control unit 620 may execute another process or a program existing in the storage unit 640. [ The control unit 620 may move or recall data to the storage unit 640 as required in the execution process. In some embodiments, the control unit 620 is configured to execute the application in response to the received signal based on the OS.

The controller 620 according to various embodiments may perform operations to generate a panoramic image from a plurality of image frames. For this, the control unit 620 may receive the image frames generated from the image generating unit 610. Here, the image generated based on the plurality of image frames may be a 360 degree rotated image. Also, the controller 620 can adaptively change the panorama image by performing an operation on the generated panorama image. For example, the control unit 620 may rearrange a plurality of image frames. The control unit 620 may control the display 630 to display the generated panoramic image.

The display 630 displays a screen including images, graphics, text, and the like. For example, the display 630 may be composed of a liquid crystal, a light emitting diode display, or other material. The display 630 may display a screen corresponding to the data received through the control unit 620. [ Display 630 may also include a touch screen for sensing user input. In accordance with various embodiments, the display 510 may include a display device 520 and may display a panoramic image generated from a plurality of image frames.

The storage unit 640 is coupled to the controller 620. A portion of the storage 640 may include random access memory (RAM) and another portion of the storage 640 may be a flash memory or other read- only memory, ROM). According to various embodiments, the storage unit 640 may store image frames for generating a panoramic image. In addition, the storage unit 640 may store the panoramic image generated from the image frames.

FIG. 7 illustrates lanes of a camera module 410 in accordance with various embodiments of the present invention. 7, the camera L ₁ 430-4 and the camera L ₂ 430 of the plurality of cameras L ₁ to L ₈ , R ₁ to R ₈ included in the camera module 410 -2). However, the cameras not shown in FIG. 7 can also perform the operations for the camera L ₁ 430-4 and the camera L ₂ 430-2 described below in the same manner.

The camera L- ₁ 430-4 and the camera L- ₂ 430-2 capture the ladle 720 and the ladle 710, respectively, to generate image frames for constructing a panoramic image. In Fig. 7, the arrows constituting each of the lasers 710 and 720 represent lasers incident in the direction corresponding to the arrows. The arrows in the middle of the arrows constituting each of the lasers 710 and 720 correspond to the rays incident on the respective cameras L ₁ 430-4 and L ₂ 430-2 in the direction of the optical axis , &Quot; middle ray "). The image frames captured by each of the cameras L- ₁ 430-4 and L- ₂ 430-2 constitute adjacent image frames among the stitched image frames in the panoramic image. Ray that is incident through the lens of the camera L- ₁ (430-4) is a projection (projection) on the image sensor of the camera L ₁ (430-4), the camera L- ₁ (430-4) is an image sensor by the projection It is possible to generate an image frame based on the recognized data.

According to Fig. 7, the arrows representing each ladle 720 all originate at the same point where the lens of camera L ₁ 430-4 is located. In other words, all the lasers 720 captured by the camera L ₁ 430-4 to generate the image frame are all collected by the lens of the camera L ₁ 430-4. The lasers 710 that camera L-- ₂ 430-2 captures to create an image frame are also all converged to camera camera L ₂ 430-2 which represents ladle 710 in FIG. Arrows all appear to start at the same point.

Figure 8 shows an image 800 captured by a camera module 410 with a plurality of cameras in accordance with various embodiments of the present invention. For example, the image 800 may be an image in which a plurality of image frames captured by the camera module 410 are stitched and a left panoramic image is spread out in a plane. A rectangle corresponding to each of the numbers 1 to n in the area 830 may correspond to an image frame captured by each of the left cameras L ₁ to L ₈ of the camera module 410, May correspond to an image frame captured by the top camera 420 of the camera module 410. [ Alternatively, the rectangles corresponding to the numbers 1 to n in the area 830 may correspond to the image frames captured by the right cameras R ₁ to R ₈ of the camera module 410. In other words, the left cameras (L ₁ to L ₈ ) and the right cameras (R ₁ to R ₈ ) of the camera module 410 correspond to the left panorama image such as the image 800 shown in FIG. 8 and the right panorama image Lt; RTI ID = 0.0 > frames. ≪ / RTI >

In the area 830, for the corresponding image frame of each number, there exists an image frame generated by another camera constituting the stereoscopic pair. For example, if the image frame corresponding to the number 1 in the image 800 of the image frames generated camera L ₁ (430-4), on the right panorama image, the camera L ₁ (430-4) and a stereoscopic camera pairs R ₁ 430-1 generates an image frame. The same applies to image frames corresponding to other numbers, and thus the area 830 represents an area where the user can feel a stereoscopic effect by the image frames generated by the cameras constituting each stereoscopic pair.

On the other hand, since the area 840 corresponds to the image frame generated by the top camera 420 and the stereoscopic pair does not exist with respect to the top camera 420, the user can not feel the stereoscopic effect on the area 840. Area 850 represents the lower portion of the FOV outside the side cameras that the camera module 410 can cover and a blind spot if the bottom surface of the camera module 410 does not include at least one camera. Lt; / RTI >

As shown in FIG. 8, the image 800 includes a region 830 where the user can feel a three-dimensional feeling, and regions 840 and 850 that are not. According to various embodiments, the region 830 where the user may feel a three-dimensional feeling may be referred to as a 3D (three-dimensional) region, and the region 840 where the user can not sense a three-dimensional feeling may be referred to as a 2D region. The 3D region 830 and the 2D region 840 are divided through the boundaries 810 and 820. When the user moves his or her gaze from the region 830 to the region 840 or the region 850, .

According to various embodiments, 3D and 2D may not be discontinuous concepts. For example, as the user moves his or her eyes from the area 830 to the area 840, the three-dimensional sensation felt by the user does not change abruptly with respect to the boundaries 810 and 820, 810, and 820 without changing the size of the image. In addition, as described above, since there is no limit to the number of side cameras, the number of image frames corresponding to each camera in the area 830 as shown in FIG. 8 may vary as the value of n changes. Here, n may be an integer greater than or equal to two.

9 is a diagram illustrating a parallax problem occurring after stitching of image frames in accordance with various embodiments of the present invention. 9 illustrates a situation where each of the cameras 910-1 and 910-2 located at different locations captures both a stationary object 920 located at a short distance and a stationary object 930 located at a long distance.

When the camera 910-1 captures the near object 920 and the far object 930, each of the objects 920 and 930 is projected onto the projection plane 940-1 of the image sensor , And generates an image frame 950-1. Depending on the projection result, the image corresponding to the near object 920 in the image frame 950-1 is displayed on the right side relative to the image corresponding to the far object 930. [ Similarly, when the camera 910-2 captures the near object 920 and the far object 930, each of the objects 920 and 930 is projected onto the projection plane 940-2 of the image sensor, And generates an image frame 950-2. Depending on the projection result, the image corresponding to the near object 920 in the image frame 950-2 is displayed on the left side relative to the image corresponding to the far object 930. [

In the stitching process, feature points are extracted from the image frame 950-1 and the image frame 950-2, and the image frames 950-1 and 950-2 are aligned based on corresponding ones of the feature points. According to various embodiments, it is assumed in FIG. 9 that feature points are extracted centering on pixels corresponding to the near object 920 in the image frames 950-1 and 950-2. In this case, the image frames 950-1 and 950-2 are aligned so that the pixels corresponding to the near object 920 overlap as shown in FIG. The aligned image frames 950-1 and 950-2 are combined into one image based on the padding 960. [ In the combined image, the near object 920 is appropriately displayed, but the image corresponding to the far object 930 corresponds to the near object (the object corresponding to the image frame 950-1 and the image frame 950-2) 920) are displayed on both the left and right sides of the image. As described above, a double image may occur in the stitched image due to the parallax occurring when capturing the same objects in cameras at different positions. Similarly, when the feature points are extracted around the pixels corresponding to the far object 930 and the image frames 950-1 and 950-2 are aligned, the image corresponding to the near object 920 in the stitched image (Truncated near object), or it may not be visible at all.

In the case of stitching the image frames generated from the camera L ₁ 430-4 and the image frames generated from the camera L ₂ 430-2, distortion may exist due to parallax between the cameras. Also, for stereoscopic consistency, the left and right camera belonging to the same stereopair should correspond to the generated image frames. For example, when the same object is captured by cameras (L ₁ , R _1, and L ₂ , R ₂ ) belonging to two stereopair pairs, the captured images are captured by cameras belonging to the same stereopair in the left panorama and the right panorama The object must correspond to the image frames. Otherwise, the depth perception of the object of the user looking at the object may deteriorate, and the 3D quality may deteriorate.

As described above, a stitching error such as a short-distance object truncation or a double image may occur in a panoramic image generated by stitching image frames, and there may arise a problem that a 3D region and a 2D region are clearly and discretely distinguished by boundaries. These problems are due to the fact that the lasers captured to generate the image frames as shown in Fig. 7 only converge to a certain number of specific positions where each of the plurality of cameras is present, and as the number of cameras for capturing lasers increases, Are less likely to occur. However, there is a limit to increasing the number of cameras, and thus various embodiments of the present invention allow a limited number of cameras to capture captured lasers as if they were captured by cameras located at various locations Method and an electronic apparatus thereof.

Figure 10 illustrates warping of lasers according to various embodiments of the present invention. For convenience of explanation, only the camera L ₁ 430-4 among the plurality of cameras L ₁ to L ₈ , R ₁ to R ₈ included in the camera module 410 is shown in FIG. However, the cameras not shown in FIG. 7 can also perform the same operations on the camera L ₁ 430-4 described below.

As shown in FIG. 10, the camera L ₁ 430-4 captures the ladle 1010 to generate an image frame for constructing a panoramic image. In Fig. 10, the arrows constituting each ladle 1010 indicate the rays incident in the direction corresponding to the arrows. The ray that is incident through the lens of the camera L ₁ (430-4) (1010) is projected onto the image sensor of the camera L ₁ (430-4), camera L ₁ (430-4) is the image sensor by the projection An image frame can be generated based on the recognized data.

In other words, the ladle 1010 camera L ₁ (430-4) to the capture to create an image frame, and are all gathered by the lens of the camera L ₁ (430-4), the image frame that is actually generated Ray (1010 ). However, in order to transform the image frame captured by the camera L ₁ 430-4 captured by the virtual cameras located elsewhere, various embodiments of the present invention may be used for warping of the ray, . The converted image frame can be referred to as a virtual image frame since it can be regarded as a virtual camera captured. As shown in FIG. 10, when the ladle 1010 is warped, the remaining ladies except the center ray of the ladle 1010 are moved in parallel to the corresponding virtual camera positions. In the warped ladle 1020, the starting point of the arrow indicating each ray indicates the position of the virtual camera corresponding to each ray, and the direction indicated by the arrow indicates the direction of the ray incident on the virtual camera. That is, one camera L ₁ 430-4 actually captures the ladle 1010 to generate an image frame, which is captured by each corresponding virtual camera as if the warped ladder 1020 were captured As shown in FIG.

)가 정해지면, 워핑되는 레이는 중심선 방향과 이루는 각도가 α가 되도록 평행이동 되므로, 각도

에 따라 워핑되는 레이의 위치가 결정될 수 있다. 레이의 워핑에 의해, 최종적으로 생성되는 파노라마 이미지는 가상의 카메라들이 레이를 캡쳐하여 생성된 가상의 이미지 프레임들을 포함할 수 있다.When the lasers 1010 are moved in parallel, each ray is warped such that the angle with the centerline direction is?. The position at which each ray is moved in parallel can be determined by the angle alpha with the centerline direction. In other words, the angle of the warped ray of the ladle 1010 with the optical axis of the camera L ₁ (430-4)

Is determined, the warped ray is moved in parallel so that the angle with the center line direction becomes?

The position of the ray to be warped can be determined. By the warping of the ray, the finally produced panoramic image may include virtual image frames generated by virtual cameras capturing the ray.

FIG. 10 shows the louvers captured by the camera L ₁ 430-4, but the lasers captured by other cameras included in the camera module 410 may also be warped. FIG. 11 illustrates lanes after a plurality of cameras included in the camera module 410 have been warped captured in accordance with various embodiments of the present invention. The picture shown on the left shows that the louvers captured by the plurality of left cameras L ₁ to L ₈ are warped and the picture shown on the right is captured by the plurality of right cameras R ₁ to R ₈ Lt; / RTI > The electronic device 201 transforms the image frames as if the plurality of left cameras L ₁ to L ₈ and the virtual cameras captured corresponding lasers respectively as shown on the left side of Fig. 11 And generate a left panorama based on the transformed image frames. The electronic device 201 also converts the image frames as if the plurality of right cameras R ₁ to R ₈ respectively captured corresponding lasers as shown on the right side of FIG. 11, The right panorama can be generated.

In the embodiments described above, a plurality of cameras (L ₁ to L _8, R ₁ to R ₈₎ and by a plurality of virtual cameras, the real plurality of cameras in a camera module (410) (L ₁ to L ₈ , R ₁ to R ₈ ) as well as the positions of the plurality of virtual cameras can be captured to generate image frames. Hereinafter, the virtual cameras generated according to the warping of the lasers will be used to capture the lasers incident on all directions of the camera module 410 to detect the actual cameras L ₁ to L ₈ , R ₁ to R ₈ , Is defined as 'omni-directional stereo projection' (ODS).

)를 이루는 레이는 워핑되어, 해당되는 위치의 가상의 카메라가 워핑된 레이 방향의 물체를 바라보고 캡쳐하는 것으로 볼 수 있다. 카메라가 물체를 바라보는 위치를 '관점(view)'이라 정의할 수 있으며, 본 발명의 다양한 실시 예들에서 관점은 실제 카메라들(L₁ 내지 L₈, R₁ 내지 R₈)의 위치로 고정된 것이 아니라, 레이가 실제 카메라의 광축과 이루는 각도 또는 레이의 방향에 따라 달라질 수 있다.In the lasers captured by each of the plurality of cameras (L ₁ to L ₈ , R ₁ to R ₈ ), each of the rays except the center ray can be seen as being captured by the virtual camera at the corresponding position. In other words, the optical axis of the camera and a certain angle

) Is warped, and it can be seen that a virtual camera at the corresponding position looks at an object in the warped ray direction and captures it. The position in which the camera looks at an object can be defined as a " view ", and in various embodiments of the present invention, the viewpoint is fixed to the position of the actual cameras (L ₁ to L ₈ , R ₁ to R ₈ ) But may vary depending on the angle of the ray with respect to the optical axis of the actual camera or the direction of the ray.

)뿐만 아니라, 수직 방향의 고도 각(

)의 변화에 의해서도 달라질 수 있으며, 이는 도 12에서 보다 상세히 설명된다.The viewpoint is a horizontal angle (

), As well as vertical altitude angles (

), Which is described in more detail in FIG.

Figure 12 shows the location of an aspect in accordance with various embodiments of the present invention.

)에 따라 달라질 수 있다. 도 12에 도시된 화원(viewing circle)들은 각각 동일한 레이의 고도 각

에서 가능한 관점들의 집합을 나타낸다. 레이의 고도 각(1210)

가 90˚(천정 방향)에서 -90˚로 변함에 따라, 화원의 반지름 R'은 R'=R*cos(

)를 만족하도록 변경되고, 관점의 높이 h'는 h'= h*sin(

)를 만족하도록 변경된다. 따라서, 천정(

=90˚)에서 R'= 0, h'= h 이므로, ODS 투영을 위한 관점은 정확히 탑 카메라(420)의 위치와 동일하게 될 수 있다.12, the viewpoints 1240 of the cameras constituting the stereoscopic pair include an altitude angle 1210 (

). ≪ / RTI > The viewing circles shown in Fig. 12 each have an elevation angle of the same ray

The set of possible perspectives. The elevation angle of the ray (1210)

(Ceiling direction) to -90 degrees, the radius R 'of the flower source is R' = R * cos (

), And the height h 'of the viewpoint is changed to satisfy h' = h * sin (

). Therefore,

= 90 DEG), the viewpoint for the ODS projection can be exactly the same as the position of the top camera 420, since R '= 0 and h' = h.

As the radius of the flower source changes, the stereoscopic baseline of the cameras that make up the stereo pair can also change. The distance between viewpoints is related to the degree of 3D perception, since the angle of view for an object decreases as the distance between viewpoints decreases, making it difficult for the perspective to be relatively identified. The relationship between the specific viewpoint distance and the degree of 3D recognition is described in more detail in Fig.

)와 수직 방향의 고도 각(

)에 기반하여 결정될 수 있다. 일 예로서, 수평 방향의 각도

의 범위는 [-180˚, 180˚]이고, 수직 방향의 고도 각

의 범위는 [-90˚, 90˚]일 수 있다. 그러나, 상술한 바와 같이, 관점은 단일 위치에 고정된 것이 아니라, 관점이 바라보는 방향의 수평 방향의 각도(

) 및 수직 방향의 고도 각(

)에 따라 변할 수 있다. 구체적으로, 관점은 하기의 <수학식 1>에 의해 결정될 수 있다.The ODS projection can be regarded as an equirectangular spherical projection and the viewing direction can be regarded as an angle in the horizontal direction

) And vertical altitude angle (

). &Lt; / RTI > As an example,

Is in the range of [-180 °, 180 °], and the vertical altitude angle

May be in the range of [-90 DEG, 90 DEG]. However, as described above, the viewpoint is not fixed to a single position, but rather the angle in the horizontal direction in the direction in which the viewpoint is viewed

) And vertical altitude angle (

). &Lt; / RTI &gt; Specifically, the viewpoint can be determined by Equation (1) below.

는 관점이 바라보는 방향의 수평 방향의 각도로서, 카메라가 캡쳐하는 레이가 카메라의 광축과 수평 방향으로 이루는 각도를 의미한다.

는 관점이 바라보는 방향의 수직 방향의 고도 각으로서, 카메라가 캡쳐하는 레이가 카메라의 광축과 수직 방향으로 이루는 각도를 의미한다. α는 카메라의 광축이 중심선 방향과 이루는 각도로, α 앞에 표기된 부호는 각각 좌측 카메라와 우측 카메라에 대응한다. 일 예로, +α는 좌측 카메라의 광축이 중심선 방향과 이루는 시계 방향의 각도를, -α는 우측 카메라의 광축이 중심선 방향과 이루는 반시계 방향의 각도를 의미할 수 있다. f는 고도각

에 곱해지는 계수로, 수직 방향의 3D 효과를 제한하기 위한 값을 의미한다. f값에 따른 3D 효과의 제한은 도 21에서 보다 상세히 설명된다.Where vp is the coordinates of the viewpoint, R is the radius of the camera module 410, h is the height of the camera module 410, and the R and h values can be predetermined.

Is an angle in the horizontal direction in the direction in which the viewpoint is viewed and an angle formed by the ray captured by the camera in the horizontal direction with the optical axis of the camera.

Is an elevation angle in the vertical direction in the direction in which the viewpoint is viewed and means an angle formed by the ray captured by the camera in a direction perpendicular to the optical axis of the camera. α is the angle formed by the optical axis of the camera with the centerline direction, and the signs before α correspond to the left camera and the right camera, respectively. For example, + α may be an angle in the clockwise direction in which the optical axis of the left camera is aligned with the centerline direction, and -α may be a counterclockwise angle in which the optical axis of the right camera is in the centerline direction. f is the altitude angle

And means a value for limiting the 3D effect in the vertical direction. The limitation of the 3D effect according to the f value is explained in more detail in Fig.

As the ODS projection is applied, the lasers captured by each camera constituting the camera module 410 are warped, and the electronic device 201 is moved from the perspective of the warping to the actual camera based on the ray captured by the virtual camera The captured image frame should be converted. The position (or coordinate value) of the actual object with respect to the virtual camera must be determined in order to identify which ray corresponds to which real object the ray captured by the virtual camera. This will be described in more detail in Fig. 13 below.

In order to determine the position of the actual object corresponding to the ray captured by the virtual camera, the distance of the actual object to the virtual camera, i.e., the depth value, must first be determined. The depth value of the actual object with respect to the virtual camera can be easily calculated based on the depth value of the actual object with respect to the actual camera. The process of determining the depth value is shown in FIG. 13 below.

Figure 13 illustrates a method of warping a ray based on depth values in accordance with various embodiments of the present invention.

인 레이가 워핑되고, 워핑된 레이를 가상 카메라 L₀가 가상 카메라 L₀의 관점(1340)에서 캡쳐하는 것을 도시한다. 도 13에 따르면, 가상 카메라 L₀가 캡쳐하는 워핑된 레이에 해당하는 물체는 물체 P(1370)이다. 그러나, 가상 카메라 L₀는 실제로 카메라가 아니므로, 워핑된 레이에 해당하는 물체 P(1370)을 캡쳐할 수 없다. 대신, 실제의 카메라 L₁이 물체 P(1370)을 캡쳐하는 것이며, 따라서 전자 장치(201)는 실제의 카메라 L₁이 위치(1310)에서 실제 레이에 기반하여 캡쳐한 물체 P(1370)을 위치(1320)에서 가상의 카메라 L₀가 캡쳐한 것처럼 변환하여야 한다(가상의 이미지 프레임 생성). 실제의 카메라 L₁의 위치(1310)에 대한 물체 P(1370)의 깊이 값은 가상의 카메라 L₀의 위치(1320)에 대한 물체 P(1370)의 깊이 값과 상이하다. 그러므로, 전자 장치(201)는 실제의 카메라 L₁이 위치(1310)에서 물체 P(1370)에 대응하는 레이를 캡쳐하여 생성된 이미지 픽셀을, 가상의 카메라 L₀의 위치(1320)에 대한 물체 P(1370)의 깊이 값에 기반하여 변환하여야 한다. 변환된 이미지 픽셀은 가상의 카메라 L₀가 워핑된 레이를 캡쳐하여 생성된 이미지 픽셀로 볼 수 있다. 전자 장치(201)는 카메라 L₁이 위치(1310)에서 워핑 전의 레이를 캡쳐하여 생성된 이미지 픽셀을, 가상 카메라 L₀이 위치(1320)에서 워핑 후의 레이를 캡쳐하여 생성된 이미지 픽셀로 변환할 수 있다. 이를 통해, 카메라 L₁이 실제로 광축과 이루는 수평 방향의 각도가

인 레이를 캡쳐하여 생성된 이미지 픽셀은, 해당 레이가 워핑되어 가상의 카메라에 의해 캡쳐될 때 생성되는 이미지 픽셀로 변환될 수 있다.Referring to FIG. 13, the camera L ₁ is an actual camera, and actually captures lanes to generate an image frame. The lasers are warped according to the horizontal and vertical elevation angles of the camera L ₁ with respect to the optical axis, and in a modified view, the virtual camera captures the warped rails to create a virtual image frame, Some are converted. However, the center ray incident in the direction of the optical axis of the camera L ₁ is not warped, and the center ray is captured in view of the actual position 1310 of the camera L ₁ . In Figure 13 illustratively forms an angle in the horizontal direction with the optical axis of the camera ₁ L

Ray is warped, there is shown that the virtual camera to the warped ray L ₀ L ₀ of the virtual camera from the capture point of view (1340). According to FIG. 13, an object corresponding to a warped ray captured by the virtual camera L ₀ is an object P (1370). However, since the virtual camera L ₀ is not actually a camera, it can not capture the object P 1370 corresponding to the warped ray. Instead, the actual camera L ₁ captures an object P 1370, so that the electronic device 201 can locate the object P 1370 captured by the actual camera L ₁ based on the actual ray at location 1310 The virtual camera L ₀ should be converted as if captured by the virtual camera L ₀ (virtual image frame generation). The depth value of the object P 1370 with respect to the actual camera position L ₁ 1310 is different from the depth value of the object P 1370 with respect to the position 1320 of the virtual camera L ₀ . The electronic device 201 therefore obtains the image pixel generated by capturing the ray corresponding to the object P 1370 at the actual camera L ₁ at position 1310 to the object 1320 for the location 1320 of the virtual camera L ₀ P &lt; / RTI &gt; (1370). The converted image pixels can be viewed as image pixels generated by capturing a virtual camera L ₀ warped ray. The image pixels are generated by capturing the ray prior to warping in the electronic device 201 includes a camera L ₁ is located 1310, the virtual camera L ₀ The position 1320 to be converted in to an image pixel is generated by capturing the ray after warped . Thus, the horizontal angle of the camera L ₁ with respect to the optical axis is

The image pixels generated by capturing the inlay can be converted into image pixels that are generated when the corresponding ray is warped and captured by the virtual camera.

In order to convert the image pixels, first, it is identified which position P (1370) captured by the virtual camera L ₀ exists in the image frame generated by the camera L ₁ . In other words, the camera coordinates of object P 1370 should be identified. The position (or coordinates) of the object P 1370 with respect to the viewpoint 1320 of the virtual camera L ₀ is represented by the following equation (13), when the depth value for the object P 1370 in the viewpoint 1320 of the virtual camera L ₀ is d: Can be calculated by Equation (2).

는 관점이 바라보는 방향의 수평 방향의 각도로서, 카메라가 캡쳐하는 레이가 카메라의 광축과 수평 방향으로 이루는 각도를 의미한다.

는 관점이 바라보는 방향의 수직 방향의 고도 각으로서, 카메라가 캡쳐하는 레이가 카메라의 광축과 수직 방향으로 이루는 각도를 의미한다.Where P _w is the position of the object P 1370 with respect to the determined view 1320 and d is the depth value of the object P 1370 with respect to the view 1320.

Is an angle in the horizontal direction in the direction in which the viewpoint is viewed and an angle formed by the ray captured by the camera in the horizontal direction with the optical axis of the camera.

Is an elevation angle in the vertical direction in the direction in which the viewpoint is viewed and means an angle formed by the ray captured by the camera in a direction perpendicular to the optical axis of the camera.

Given a camera matrix, the electronic device 201 can convert the coordinates P _w of the object P 1370 calculated by Equation (2) into the camera coordinate system. In other words, the electronic device 201 may calculate the camera coordinates (u, v) obtained when camera projection is performed on the object P (1370). The camera coordinates may indicate at least one image pixel generated when an object P 1370 is captured according to a ray in which a virtual camera L ₀ is warped in an image frame generated by an actual camera L ₁ . The image pixel is converted based on the depth value (d) of the object P 1370 for the virtual camera L ₀ , and the image pixel generated by capturing the ray before the actual camera L ₁ is warped is converted into the converted image pixel It is replaced.

(1330)은 카메라 L₁(1310)의 광축과 카메라 L₁(1310)에서 물체 P(1370) 방향이 이루는 각도를 의미한다.

(1350)는 카메라 L₁(1310) 및 가상 카메라 L₀(1320)이 물체 P(1370)에 대해 이루는 화각 차를 의미한다. 고도 각

이 0˚일 때, 화각 차

(1350)는 각도

(1330)와 워핑된 레이가 카메라 L₁(1310)의 광축과 이루는 각도(

)의 차이로 정의될 수 있다. 물체 P(1370)가 멀리 떨어져 있을수록, 화각 차

(1350)는 줄어들게 되며, 물체 P(1370)가 가까이 있을수록, 화각 차

(1350)는 커지게 된다.13,

1330 refers to the optical axis and the angle of the camera ₁ L (1310) the object P (1370) forming the camera direction is L ₁ (1310).

1350 denotes an angle of view angle formed by the camera L ₁ 1310 and the virtual camera L ₀ 1320 with respect to the object P 1370. Altitude angle

Is 0 DEG, the angle of view angle

(1350)

(1330) and the angle formed by the warped ray with the optical axis of the camera L ₁ (1310)

). &Lt; / RTI &gt; As the object P (1370) is farther away,

(1350) is decreased, and as the object P (1370) is closer to the object P

(1350) becomes larger.

또한 이미지 픽셀의 변환을 위해 고려될 수 있다.Although only one ray is warped in FIG. 13, the rays captured by all of the plurality of cameras provided in the camera module 410 can be warped, and the above-described operations can be applied. Further, the angle of the ray in the vertical direction with respect to the optical axis

It can also be considered for conversion of image pixels.

Figure 14 illustrates a method of capturing an image using a virtual camera in accordance with various embodiments of the present invention.

The ray 1410 and the ray 1420 may be center lasers captured by the actual cameras at positions corresponding to the start points of the arrows. The center ray is not warped and can be captured by real cameras. Ladders 1430-1, 1430-2, and 1430-3 correspond to warped lasers, which can be seen as being captured by virtual cameras at locations corresponding to the starting point of the arrow. All of the lasers are captured by the actual cameras provided by the camera module 410 and the warped lasers 1430-1, 1430-2, 1430-3 are captured by the actual cameras, And is converted based on the viewpoints and depth values of the cameras. An image frame for the object 1450 is generated based on the lasers 1410, 1420, 1430-1, 1430-2, 1430-3 captured by the actual cameras and the virtual cameras, and the actual cameras and virtual cameras Because the object 1450 has been captured from various perspectives, the parallax problem for the object 1450 in the image frame can be minimized.

The depth planes consist of positions at the same distance from the center of the camera module 410. According to Fig. 14, the object 1450 intersects with different depth planes according to the curvature of the object 1450. That is, the depth value of an object may be different depending on the point of the object, and the depth value may be different depending on the viewpoint of the point even at the same point. The point of the object corresponding to the ray captured by the virtual camera has a depth value different from that of the virtual camera when captured by the actual camera and the electronic device 201 captures the ray to capture a virtual image frame The depth value for the virtual camera must be considered in order to convert the image frame.

FIG. 15 is a flow diagram illustrating a registration image generated using virtual cameras in accordance with various embodiments of the present invention.

Referring to FIG. 15, in operation 1510, the control unit 620 determines a plurality of image frames generated by the plurality of cameras. For example, the image generating unit 610 of the electronic device 201 may include a plurality of cameras, and each of the plurality of cameras may capture images of the lasers gathered by the camera lens to generate image frames. The plurality of cameras corresponds to an actual physical camera. The control unit 610 may receive the image frames generated by the image generating unit 610 and may determine a plurality of image frames generated by the plurality of cameras. As another example, the electronic device 201 may receive a plurality of image frames generated from an external device that is separate from the electronic device 201. [ In other words, the external device, such as the image generating unit 610, may have a plurality of cameras, and each of the plurality of cameras may capture images of the lasers gathered by the camera lens to generate image frames. The controller 610 may receive the image frames generated by the external device and determine a plurality of image frames generated by the plurality of cameras.

In operation 1520, the control unit 620 converts a plurality of image frames based on the plurality of cameras and at least one virtual camera located elsewhere. The captured images of the respective cameras constituting the image generator 610 are warped to the positions of the corresponding virtual cameras, and the controller 620 captures warped rails from the virtual camera's viewpoints, Image frames as if they were generated. That is, each ray corresponds to a partial area of the image frame, and each of the partial areas of the image frame may be deformed through warping. Here, some regions include at least one pixel.

In operation 1530, the control unit 620 generates at least one matched image from the plurality of transformed image frames. For example, the control unit 620 may convert the image frames captured by the left cameras of the image generating unit 610, and may stitch the converted image frames to generate a left panorama, And stitch the converted image frames to create a right panorama.

At operation 1540, the display 630 may display a portion of at least one registered image. For example, the left panorama and the right panorama generated by the controller 620 in operation 1530 may be a 360-degree rotated image, and the display 630 may display a FOV Can be displayed. Display 630 may also allow both a portion of the left panorama and a portion of the right panorama to be displayed so that each portion is rendered through both eyes of the user.

16 is a flow chart for transforming an image based on warping in accordance with various embodiments of the present invention. Operations in the flow chart of FIG. 16 can be performed in operation 1520 to convert a plurality of image frames based on at least one virtual camera, where the control unit 620 is located at a different location from the plurality of cameras.

16, at operation 1610, the controller 620 identifies at least one first image pixel generated based on a ray in a direction other than the direction of the optical axis. A plurality of cameras constituting the image generating unit 610 can generate image frames by capturing lasers reflected from an object, respectively, and the control unit 620 controls the lasers, not the direction of the optical axis, Thereby identifying the generated image pixels.

In operation 1620, the controller 620 determines at least one second image pixel for the direction of the ray in at least one virtual camera at a location corresponding to the ray. When ODS projection is applied, the lasers can be warped and the control unit 620 can convert the image frame as if the virtual camera captured the ray at the warped position. To this end, the control unit 620 can determine an image pixel to be generated if an object in the ray direction is captured by the virtual camera in terms of the virtual camera. When the ray to be warped is determined, the angle in the horizontal direction and the angle in the vertical direction that the ray forms with the optical axis are determined, so that the viewpoint of capturing the warped ray by Equation (1) can be determined. In other words, when the ray is determined, the position corresponding to the ray can be determined, and the control unit 620 can identify the second image pixel with respect to the ray direction in at least one virtual camera at a position corresponding to the ray. The second image pixel may be an image pixel to be generated if, for example, the virtual camera L ₀ captures an object P 1370 corresponding to the warped ray at location 1320 as described in FIG. 13, May be a pixel corresponding to the camera coordinates (u, v) calculated in Equation (2).

In operation 1630, the controller 620 calculates the depth value of the determined at least one second image pixel. By way of example, FIG. 17 illustrates a method of computing depth values for image pixels in accordance with various embodiments of the present invention. In Fig. 17, what is represented by the dashed lines represents the depth planes 1730, 1740, 1750, and 1760. The depth planes consist of positions at the same distance from the center of the camera module 410. The camera L ₁ is an actual camera, capturing lasers at the position 1710 with the lens of the camera L ₁ to generate an image frame. For ODS projection, the lasers can be warped and the virtual camera L ₀ captures a corresponding warped one of the lasers at location 1720. The depth value of the object with respect to the virtual camera L ₀ must be considered in order to achieve the same effect that the virtual camera L ₀ captures when the ray is warped.

As an example, not only the camera L ₁ but also other actual cameras (at least one of L ₂ to L ₃ (not shown)) captures an image frame generated by capturing the rays into respective depth planes 1730, 1740, 1750 , 1760, the depth value of the object with respect to the virtual camera L ₀ can be calculated. The re-projected image frames overlap each other and determine how well the image pixels corresponding to the warped ray captured by the virtual camera L ₀ in the re-projected image frame overlap in each depth plane so that the depth value of the object Can be calculated. For example, the degree of overlap of the image pixels may be determined based on the pixel value difference of the image pixels in the re-projected image frame for each depth plane. The smaller the pixel value difference is, the more the image pixels overlap well in the corresponding depth plane, and the controller 620 determines the depth value of the most overlapping depth plane as the depth value of at least one second image pixel corresponding to the warped ray .

The calculation of the depth value of an image pixel as described in Fig. 17 is merely exemplary. In various embodiments of the present invention, the depth value of an image pixel can be calculated by a method different from the depth value calculation method described in FIG. 17, including an algorithm for calculating a depth value of a general image pixel. In addition, if the depth value of an object captured by an actual camera is already known, the viewpoint of the virtual camera capturing the object can be determined by Equation (1) It can easily be calculated.

In operation 1640, the control unit 620 converts at least one first image pixel based on the depth value of the at least one second image pixel for the at least one virtual camera. The first image pixel should be replaced with the second image pixel generated based on the warped ray and the second image pixel should reflect the depth value for the virtual camera other than the actual camera. That is, the depth value of the second image pixel is converted so that the depth value for the virtual camera is applied, and the first image pixel is replaced with the converted second image pixel.

Prior to performing the ODS projection by capturing image frames, a camera calibration must be performed first. Camera calibration means adjusting the cameras so that the stereoscopic object or image captured by the camera is projected to the corresponding position in the image sensor. In other words, once the camera calibration is applied, it can be determined that the coordinates (u, v) in the image sensor and all the coordinates (x, y, z)

) 및 수직 방향의 고도 각(

)에 의해 방향이 결정될 수 있다. 이에 따라, 각 레이에 대응하는 이미지 센서의 좌표 (u,v)는 하기의 <수학식 3>에 의해 결정될 수 있다.When the camera captures an object, each point of the object can correspond to a ray captured by the camera. Each ray has a horizontal angle with the optical axis of the camera (

) And vertical altitude angle (

The direction can be determined by the angle? Accordingly, the coordinates (u, v) of the image sensor corresponding to each ray can be determined by Equation (3) below.

는 레이가 카메라의 광축과 이루는 수평 방향의 각도,

는 어안 반경(fisheye radius)을 의미한다. 어안 반경

는 하기의 <수학식 4>와 같이 5차 다항식

로 모델링 될 수 있다.Here, (u, v) is the coordinates of the image sensor, A is an affine transformation matrix for compensating the tilt of the camera lens, (x _c , y _c ) is the optical center of the camera ),

Is the angle of the ray in the horizontal direction with the optical axis of the camera,

Means the fisheye radius. Fisheye radius

As shown in Equation (4) below,

Lt; / RTI &gt;

는 어안 반경,

는 레이가 카메라의 광축과 이루는 수직 방향의 고도 각,

는 5차 다항식의 계수(coefficient)를 의미한다.From here,

The fisheye radius,

Is an elevation angle in the vertical direction that the ray forms with the optical axis of the camera,

Denotes the coefficient of the fifth-order polynomial.

Assuming that the camera is positioned as shown in FIGS. 4B-4H, the three rotation angles (roll, pitch, yaw) of the camera can be optimally adjusted for camera calibration. The procedure for optimizing the three rotation angles is as follows.

First, the corresponding points for the cameras that make up each stereo pair are determined a long distance from the cameras. Next, for each of the left cameras L ₁ to L ₈ and each of the right cameras R ₁ to R ₈ , the roll and pitch are set to 0 ° and set to yaw (i-1) * 45 °. Here, i is the number of each camera (L _i Or R _i . Assuming that the corresponding points are very far from the camera, the distance after the corresponding points are projected onto the ODS is calculated. Finally, the roll, pitch, and yaw angle of each camera can be optimized to minimize the distance between corresponding points in the ODS projection.

After the above-described calibration procedure for the camera is performed, the controller 620 may store parameters (e.g., roll, pitch, and yaw angle values) by camera calibration in the storage unit 640. The image frame generation and processing performed by the electronic device 201 thereafter may be performed based on the stored parameters.

18 is a flow chart for performing camera calibration in accordance with various embodiments of the present invention. The following operations may be performed prior to operation 1510.

Referring to Fig. 18, in operation 1810, the control unit 620 performs calibration for each of the plurality of cameras. For example, calibration refers to determining the roll, pitch, and yaw angle for each camera.

In operation 1820, the control unit 620 stores at least one parameter for the calibration. The control unit 620 may store the parameters (e.g., roll, pitch and yaw angle values) by calibration in the storage unit 640 and the image frame creation and processing performed by the electronic device 201 thereafter Can be performed based on stored parameters.

Figure 19 is a flowchart of stitching image frames in accordance with various embodiments of the present invention. After the image frames are converted by the ODS projection, the converted image frames are stitched to produce the final panorama image. The operation of stitching the converted image frames is as follows.

Referring to FIG. 19, at operation 1910, the control unit 620 determines at least one shim for a plurality of image frames. Before determining the shims for image stitching, the feature points should be detected in the image frames and the image frames should be aligned based on the corresponding points of the detected feature points, but the ODS projection according to various embodiments of the present invention Once applied, image frames can be seen as already aligned in the process of warping the lasers. Thus, according to various embodiments of the present invention, image alignment operations based on feature point detection for image stitching can be omitted. In one example, the controller 620 may determine shims in the area of the superimposed image frames, and may determine shims to minimize differences in parameter values for pixel values, hues, and brightness in pixels of each image frame adjacent to the shim. As another example, the control unit 620 can more simply determine a vertical linear line in each transformed image frame. In other words, since the converted image frames can be regarded as well-aligned in the process of warping the lanes, the controller 620 can determine the vertical line in the center of the two cameras in the converted image frames have.

In operation 1920, the control unit 620 performs blending on a plurality of image frames based on at least one shim. For example, the control unit 620 can compensate the exposure and color parameters to be uniform in the corresponding region so that the regions separated by the shim are seen to be smoothly connected.

According to the embodiment described above, a panoramic image, i.e. a matched image, can be generated. An example of a panoramic image generated according to various embodiments of the present invention is shown in FIG. 20 below. FIG. 20 illustrates a matched image 2000 in accordance with various embodiments of the present invention.

The image frames generated by the left cameras (L ₁ to L ₈ ) and the right cameras (R ₁ to R ₈ ) are converted by the ODS projection, and the converted image frames are stitched to generate a matched image. The matching image 2000 may be a left panorama or a right panorama. The shim 1 2010 and the shim 2 2020 are determined in the stitching process of the image frames and the region separated by the shim 1 2010 and the shim 2 2020 in the matching image 2000 is the matching image 2000 It corresponds to the image frame to be generated. The shim 1 2010 and shim 2 2020 generated in the matching image 2000 are illustrative and the matching image 2000 may include shims as many as the number of image frames, 0.0 &gt; 2020 &lt; / RTI &gt; As described above, since the image frames converted by the ODS projection can be regarded as well-aligned in the process of warping the lobes, the shim 1 2010 and the shim 2 2020, as shown in Fig. 20, . &Lt; / RTI &gt;

When ODS projection is applied, the lasers that each camera captures to produce a panoramic image are not all projected from the perspective of the actual camera, and each ray is warped and transformed as if captured from the perspective of the virtual camera. Therefore, the parallax problem caused by the cameras at different positions capturing the same object according to the ODS projection can be reduced in the matching image. According to the matching image 2000 shown in Fig. 20, the matching image 2000 is constructed by stitching image frames including a long distance and a short distance object, No top or short cuts appear.

Figure 21 illustrates the relationship between elevation angles of cameras and distances of view of cameras according to various embodiments of the present invention. In Fig. 21, the horizontal axis represents the distance between the viewpoints of the cameras constituting the stereo pair, and the vertical axis represents the altitude of the ray captured by the camera.

)에 따라 달라질 수 있다. 관점간 거리가 짧아질수록 물체에 대한 화각이 줄어들어 원근이 상대적으로 식별되기 어려우므로, 관점간 거리는 3D 효과의 정도와 관련이 있다. 다시 말해서, 관점간 거리가 큰 경우 사용자에 의한 3D 인식이 양호할 수 있고, 관점간 거리가 짧을 경우 사용자에 의해 파노라마 이미지가 2D에 가깝게 인식될 수 있다. 또한, 관점간 거리가 0일 경우(zero baseline), 파노라마 이미지는 완전한 2D(pure 2D)로 사용자에게 인식될 수 있다. 12, when the ODS projection is applied, the distance between the viewpoints of the cameras constituting the stereopair is determined by the height angle (

). &Lt; / RTI &gt; The distance between viewpoints is related to the degree of 3D effect, as the distance between viewpoints is shorter and the angle of view for the object is reduced, making it difficult to identify the perspective relative. In other words, if the distance between the viewpoints is large, 3D recognition by the user can be good, and if the distance between the viewpoints is short, the panorama image can be perceived close to 2D by the user. In addition, when the distance between viewpoints is zero (zero baseline), the panoramic image can be recognized by the user in complete 2D (pure 2D).

)의 관계를 도시한다. v는 관점간 거리가 0일 때의 고도 각으로, 파노라마 이미지가 사용자에 의해 완전한 2D로 인식되는 고도 각을 의미한다. 예를 들어, v는 <수학식 1>에서 계수 f의 값에 따라 45˚ 내지 90˚ 범위에서 변하는 각도일 수 있다. 도 21에 따르면, 고도 각

가 증가할수록 관점간 거리가 포물선을 따라 부드럽게 감소하므로, 고도 각에 따라 3D 효과의 정도가 적어질 수 있다. 다시 말해서, ODS 투영에 의해 사용자의 시선 이동에 따른 3D에서 2D로의 급격한 변화가 방지될 수 있다. 도 21에 따르면, 영역(2110) 이내의 고도 각 범위에서는 사용자에 의한 3D 인식이 양호할 수 있다. 또한, 고도각이

v 범위를 벗어난 영역(2120)의 경우, 파노라마 이미지는 사용자에게 2D로 인식될 수 있다. 영역(2110) 및 영역(2120) 이외의 영역은 트렌지션 영역(transition area)(2130)으로, 고도 각이 증가할수록 사용자가 느끼는 3D 효과가 점진적으로 감소하는 영역에 해당한다. 다시 말해서, 트랜지션 영역(2130)은 고도 각이 증가함에 따라 3D에서 2D로의 급격한 변화를 방지하는 버퍼 영역일 수 있다. The parabolic curve shown in Fig. 21 shows the relationship between the viewpoint distance and the altitude angle of the ray captured by the camera

). &Lt; / RTI > v is the elevation angle when the distance between viewpoints is 0, and the elevation angle at which the panoramic image is recognized by the user as complete 2D. For example, v may be an angle varying in the range of 45 DEG to 90 DEG according to the value of the coefficient f in Equation (1). According to Fig. 21,

, The distance between viewpoints decreases smoothly along the parabola, so that the degree of the 3D effect may be reduced depending on the elevation angle. In other words, the sudden change from 3D to 2D due to the user's gaze movement can be prevented by the ODS projection. According to Fig. 21, 3D recognition by the user can be good in the elevation angle range within the area 2110. [ In addition,

v For the out-of-range region 2120, the panoramic image may be recognized as 2D by the user. A region other than the region 2110 and the region 2120 is a transition region 2130 and corresponds to a region where the 3D effect experienced by the user gradually decreases as the elevation angle increases. In other words, the transition region 2130 may be a buffer region that prevents sudden changes from 3D to 2D as the elevation angle increases.

의 변화에 따라 화면 뒤집어짐 문제(veiw flip problem)이 생길 수 있다. 그러나, 본 발명의 다양한 실시 예들에 따르면, 3D에서 2D로의 변화는 급격하게 발생하지 않고, 서서히 발생하므로, 화면 뒤집어짐 문제가 줄어들 수 있다. If the content is rendered with the same inter-view distance for all view angles, the altitude angle

The screen flip problem (veiw flip problem) may occur. However, according to various embodiments of the present invention, the 3D to 2D transition does not occur rapidly but occurs slowly, so that the problem of screen overturn can be reduced.

As used herein, the term "module " includes units comprised of hardware, software, or firmware and may be used interchangeably with terms such as, for example, logic, logic blocks, components, or circuits. A "module" may be an integrally constructed component or a minimum unit or part thereof that performs one or more functions. "Module" may be implemented either mechanically or electronically, for example, by application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs) And may include programmable logic devices.

At least some of the devices (e.g., modules or functions thereof) or methods (e.g., operations) according to various embodiments may be stored in a computer readable storage medium (e.g., memory 130) . &Lt; / RTI &gt; When an instruction is executed by a processor (e.g., processor 120), the processor may perform a function corresponding to the instruction. The computer-readable recording medium may be a hard disk, a floppy disk, a magnetic medium such as a magnetic tape, an optical recording medium such as a CD-ROM, a DVD, a magnetic-optical medium such as a floppy disk, The instructions may include code that is generated by the compiler or code that may be executed by the interpreter. Modules or program modules according to various embodiments may include at least one or more of the components described above Some of which may be omitted, or may further include other components.

And the embodiments disclosed in this document are presented for the purpose of explanation and understanding of the disclosed contents and do not limit the scope of various embodiments of the present invention. Accordingly, the scope of the various embodiments of the present invention should be construed as including all changes or various other embodiments based on the technical idea of various embodiments of the present invention.

Claims (20)

A method of operating an electronic device,
Determining a plurality of image frames generated by the plurality of cameras,
Converting the plurality of image frames based on at least one virtual camera located at a different location from the plurality of cameras;
Generating at least one matched image from the transformed plurality of image frames;
And displaying at least a portion of the at least one registered image.
The method according to claim 1,
The plurality of cameras including a first group of cameras and a second group of cameras,
Wherein the at least one matching image comprises a first matching image generated based on the first group of cameras and a second matching image generated based on the second group of cameras.
The method according to claim 1,
Wherein the plurality of cameras are arranged in a circle,
The optical axis of each of the cameras of the first group is rotated clockwise by a predetermined angle with respect to the center of the circle,
Wherein the optical axis of each of the second group of cameras is rotated by the predetermined angle counterclockwise with respect to the center of the circle.
The method according to claim 1,
And converting the plurality of image frames based on at least one virtual camera located between the plurality of cameras,
Identifying at least one first image pixel generated based on a ray in a direction other than an optical axis direction of each of the plurality of cameras;
Determining at least one second image pixel for the direction of the ray in the at least one virtual camera at a location corresponding to the ray;
Converting the at least one first image pixel based on a depth value of the at least one second image pixel for the at least one virtual camera.
5. The method of claim 4,
Wherein a depth value of the at least one second image pixel for the at least one virtual camera is determined based on a depth value of the at least one second image pixel for each of the plurality of cameras.
5. The method of claim 4,
Wherein a position corresponding to the ray is determined based on an angle that an optical axis of each of the plurality of cameras is rotated from a reference direction.
The method according to claim 1,
And generating at least one matched image from the transformed plurality of image frames,
Further comprising determining at least one seam for the plurality of image frames.
8. The method of claim 7,
Wherein said at least one shim is determined as a line in a vertical direction.
The method according to claim 1,
Performing a calibration for each of the plurality of cameras before generating a plurality of image frames captured by each of the plurality of cameras;
Further comprising storing at least one parameter for the calibration.
5. The method of claim 4, wherein the coordinate value for the at least one second image pixel is determined by the following equation:

Wherein _Pw is a coordinate value for the at least one second image pixel, d is a depth value of the at least one second image pixel for the at least one virtual camera,

An angle formed by the ray in the horizontal direction with the optical axis of the camera,

Is an angle formed by the ray in a direction perpendicular to the optical axis of the camera, and vp is a view point of the at least one second image pixel with respect to the at least one virtual camera.

In an electronic device,
Determining a plurality of image frames generated by a plurality of cameras, converting the plurality of image frames based on at least one virtual camera located elsewhere than the plurality of cameras, A control unit for generating at least one registration image from the registration image,
And a display that displays a portion of the at least one registered image.
12. The method of claim 11,
The plurality of cameras including a first group of cameras and a second group of cameras,
Wherein the at least one matching image comprises a first matching image generated based on the first group of cameras and a second matching image generated based on the second group of cameras.
13. The method of claim 12,
Wherein the plurality of cameras are arranged in a circle,
Wherein the optical axis of each of the first group of cameras is rotated clockwise by a predetermined angle with respect to the center of the circle,
Wherein the optical axis of each of the second group of cameras is rotated by the predetermined angle counterclockwise with respect to the center of the circle.
12. The apparatus of claim 11, wherein the controller is further configured to determine, based on at least one virtual camera located between the plurality of cameras, that the optical axis direction of each of the plurality of cameras Identifying at least one first image pixel generated based on a ray in a non-directional manner and identifying at least one second image pixel for a direction of the ray in the at least one virtual camera at a location corresponding to the ray, And to convert the at least one first image pixel based on a depth value of the at least one second image pixel for the at least one virtual camera.
15. The method of claim 14, wherein the depth value of the at least one second image pixel for the at least one virtual camera is determined based on a depth value of the at least one second image pixel for each of the plurality of cameras .
15. The method of claim 14,
Wherein an optical axis of each of the plurality of cameras is determined based on an angle that is rotated from a reference direction.
12. The apparatus of claim 11, wherein the control unit determines at least one seam for the plurality of image frames to generate at least one matched image from the transformed plurality of image frames, And perform blending on the plurality of image frames based on at least one shim.
18. The apparatus of claim 17, wherein the at least one shim is determined as a line in a vertical direction.
12. The apparatus according to claim 11,
Performing a calibration for each of the plurality of cameras and storing at least one parameter for the calibration before generating a plurality of image frames captured by each of the plurality of cameras.
15. The apparatus of claim 14, wherein a coordinate value for the at least one second image pixel is determined by:

Wherein _Pw is a coordinate value for the at least one second image pixel, d is a depth value of the at least one second image pixel for the at least one virtual camera,

An angle formed by the ray in the horizontal direction with the optical axis of the camera,

Is an angle formed by the ray in a direction perpendicular to the optical axis of the camera, and vp is a view point of the at least one second image pixel with respect to the at least one virtual camera.

Images