KR102490789B1 - Electronic device and method for changing image provision manner based on movement variation - Google Patents

Abstract

According to various embodiments of the present disclosure, an electronic device includes a housing, a sensor, an image sensor disposed using at least a portion of the housing, and a processor, wherein the processor uses the image sensor to generate an image at a specified frame rate. Obtaining, using the sensor to determine the amount of change in motion of the electronic device, and when the amount of change in motion belongs to a specified first range, transmits the video to a specified external device, and the amount of change in motion falls within a specified second range If it belongs, it may be configured to generate information based on at least one of the image or the amount of motion change, and transmit the generated information to the designated external device. In addition, various embodiments are possible.

Description

Electronic device and method for changing an image provision method based on motion variation

Various embodiments of the present disclosure relate to an electronic device and method for changing an image providing method based on a motion change amount.

Today, smart phones, tablet PCs, portable multimedia players (PMPs), personal digital assistants (PDAs), laptop PCs and wrist watches, and head-mounted devices (HMDs) Various portable electronic devices such as wearable devices such as displays are appearing.

Recently, as electronic devices including cameras capable of capturing 360 degrees including the front as well as the rear of electronic devices have been released, 360 degree images representing the entire 3D space are provided to users.

A 360-degree image is an image captured and stitched by a stitching camera or the like, and is an image capable of providing a 360-degree view when a user changes a location or orientation.

Recently, with the spread of social live that can deliver content in real time, such as 'Facebook Live', 'Periscope' or 'YouTube Live', 360-degree images are being shared in various ways.

However, since a 360-degree video is often taken by a photographer while moving, it is necessary to compensate for shaking.

In particular, a viewer who watches a 360-degree video using an HMD may complain of dizziness due to excessive shaking.

According to various embodiments of the present disclosure, an electronic device includes a housing, a sensor, an image sensor disposed using at least a portion of the housing, and a processor, wherein the processor uses the image sensor to generate an image at a specified frame rate. Obtaining, using the sensor to determine the amount of change in motion of the electronic device, and when the amount of change in motion belongs to a specified first range, transmits the video to a specified external device, and the amount of change in motion falls within a specified second range If it belongs, it may be configured to generate information based on at least one of the image or the amount of motion change, and transmit the generated information to the designated external device.

According to various embodiments of the present disclosure, an electronic device includes a communication module, a display, and a processor, and the processor includes an image generated at a specified frame rate from an external device using the communication module and a movement variation of the external device. Receives information related to, and based on the information related to the amount of motion change, distinguishes a first section belonging to a first range in which the motion change in the received image is designated and a second section belonging to a designated second range, and or generating information corresponding to the second period based on at least one of the motion variation, providing the image during the first period using a display, and providing the generated information during the second period. can be set.

According to various embodiments of the present disclosure, a method for providing an image using an electronic device includes obtaining an image at a specified frame rate, checking a motion variation of the electronic device, and a first motion variation amount specified. If it belongs to the range, an operation of transmitting the video to a designated external device, and if the motion change amount belongs to a designated second range, information is generated based on at least one of the video or the motion change amount, and the generated information is stored. An operation of transmitting to the designated external device may be included.

According to various embodiments of the present disclosure, an electronic device and a method for providing an image thereof may solve inconvenience experienced by a viewer watching a 360 degree image due to shaking and/or rotation of the image.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments.
2 is a block diagram 200 of a camera module 180, in accordance with various embodiments.
3 is a diagram illustrating an image providing system according to various embodiments.
4A is a flowchart illustrating a method of providing an image by an electronic device according to various embodiments.
Referring to FIG. 4B , a screen notifying that an electronic device is rotating is shown according to various embodiments.
Referring to FIG. 4C , a method of adjusting a frame rate according to an exemplary embodiment is illustrated.
5 is a diagram illustrating a method of generating a corrected image according to an amount of motion change by an electronic device, according to various embodiments.
6A to 6C are diagrams illustrating a method of generating a corrected image according to an amount of motion change by an electronic device, according to various embodiments.
7 is a flowchart illustrating a method of providing an image by an electronic device according to various embodiments.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, a sensor module ( 176), interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the display device 160 or the camera module 180) may be omitted or one or more other components may be added. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , process commands or data stored in the volatile memory 132 , and store resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or application processor 120), and a secondary processor 123 (eg, a graphics processing unit, The image signal processor 120, the sensor hub processor 120, or the communication processor 120) may be included. Additionally or alternatively, the secondary processor 123 may be configured to use less power than the main processor 121 or to be specialized for a designated function. The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the co-processor 123 (eg image signal processor 120 or communication processor 120) is functionally related to other components (eg camera module 180 or communication module 190). can be implemented as part of

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input device 150 may receive a command or data to be used for a component (eg, the processor 120) of the electronic device 101 from an outside of the electronic device 101 (eg, a user). The input device 150 may include, for example, a microphone, mouse, or keyboard.

The sound output device 155 may output sound signals to the outside of the electronic device 101 . The audio output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 160 may include a touch circuitry set to detect a touch or a sensor circuit (eg, a pressure sensor) set to measure the intensity of force generated by the touch. there is.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors 120, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 388 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors 120 that operate independently of a processor (eg, the application processor 120) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, the corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network 199 (eg, a cellular network, the Internet, or It may communicate with the external electronic device 101 through a computer network (eg, a telecommunications network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified and authenticated.

The antenna module 197 may transmit or receive signals or power to the outside (eg, the external electronic device 101). According to one embodiment, the antenna module 197 may include one or more antennas, from which at least one suitable for a communication scheme used in a communication network such as the first network 198 or the second network 199 of antennas may be selected by the communication module 190, for example. A signal or power may be transmitted or received between the communication module 190 and the external electronic device 101 through the selected at least one antenna.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the electronic devices 102 and 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external devices among the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

2 is a block diagram 200 of a camera module 180, in accordance with various embodiments. Referring to FIG. 2 , the camera module 180 includes a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (eg, a buffer memory), or an image signal processor. (260). The lens assembly 210 may collect light emitted from a subject that is an image capturing target. The lens assembly 210 may include one or more lenses. According to one embodiment, the camera module 180 may include a plurality of lens assemblies 210 . In this case, the camera module 180 may be, for example, a dual camera, a 360-degree camera, or a spherical camera. The plurality of lens assemblies 210 have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly is at least as good as another lens lens assembly. It can have one other lens property. The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens. The flash 220 may emit a light source used to enhance light emitted from a subject. The flash 220 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp.

The image sensor 230 may acquire an image corresponding to the subject by converting light transmitted from the subject through the lens assembly 210 into an electrical signal. According to one embodiment, the image sensor 230 is, for example, an image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, It may include a plurality of image sensors having a property, or a plurality of image sensors having other properties. Each image sensor included in the image sensor 230 may be implemented as, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 240 reacts to the movement of the camera module 180 or the inspection device 101 including the camera module 180 to at least partially compensate for a negative effect (eg, image shaking) caused by the movement on a captured image. At least one lens or image sensor 230 included in the assembly 210 may be moved in a specific direction or controlled (eg, read-out timing is adjusted, etc.). According to one embodiment, the image stabilizer 240 may be implemented as, for example, an optical image stabilizer, and a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180. ) to detect the movement.

The memory 250 may at least temporarily store at least a portion of an image acquired through the image sensor 230 for a next image processing task. For example, when image acquisition is delayed according to the shutter, or when a plurality of images are acquired at high speed, the acquired original image (eg, high resolution image) is stored in the memory 250, and a copy corresponding to it is stored in the memory 250. An image (eg, a low resolution image) may be previewed through the display device 160 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a part of the original image stored in the memory 250 may be obtained and processed by the image signal processor 260 , for example. According to one embodiment, the memory 250 may be configured as at least a part of the memory 130 or as a separate memory operated independently of the memory 130 .

The image signal processor 260 performs image processing (eg, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, an image signal processor ( 260 may perform control (eg, exposure time control, lead-out timing control, etc.) for at least one of the components included in the camera module 180 (eg, the image sensor 230). The image processed by the signal processor 260 is stored again in the memory 250 for further processing or an external component of the camera module 180 (eg, the memory 130, the display device 160, the electronic device 102 ), the electronic device 104, or the server 108) According to one embodiment, the image signal processor 260 is configured as at least a part of the processor 120 or independently of the processor 120. In the case of a separate processor, the images processed by the image signal processor 260 are displayed as they are or after additional image processing by the processor 120. can be displayed through

According to one embodiment, the inspection device 101 may include two or more camera modules 180 each having different properties or functions. In this case, for example, at least one camera module 180 may be a wide-angle camera or a front camera, and at least one other camera module may be a telephoto camera or a rear camera.

3 is a diagram illustrating an image providing system according to various embodiments.

Referring to FIG. 3, the image providing system includes a first electronic device 310 (eg, the electronic device 101 of FIG. 1) producing a 360-degree image, and a 360-degree image produced by the first electronic device 310. The second electronic device 320 (for example, the electronic device 102 or 104 of FIG. 1 ) receiving and reproducing the , and the network 330 connecting the first electronic device 310 and the second electronic device 320 . ) (eg, network 199 of FIG. 1).

According to various embodiments, the first electronic device 310 may acquire a plurality of images using a camera module (eg, the camera module 180 of FIG. 1 ). For example, the camera module may include at least two or more image sensors (eg, the image sensor 230 of FIG. 2 ) configured to capture images in different directions, and a plurality of images may be acquired using the at least two or more image sensors. can According to an embodiment, the first electronic device 310 may create a 360-degree image using the acquired plurality of images. For example, the first electronic device 310 may stitch a plurality of images obtained using a processor (eg, the processor 120 of FIG. 1 or the image signal processor 260 of FIG. 2 ), It can be made into a video. As another example, the first electronic device 310 may be connected to another electronic device (eg, a host device) in a wired/wireless manner, and transmits an image signal obtained by the first electronic device 310 to the other electronic device. The other electronic device may create a 360-degree video.

According to various embodiments, the second electronic device 320 including at least one display module may provide a 360-degree image received from the first electronic device 310 to the user. The second electronic device 320 includes, for example, a portable communication device including a display (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a home appliance and a wrist watch, an HMD (Head- Mounted Display) may include various types of devices such as wearable devices.

Due to the characteristics of 360-degree video, different playback functions or features may be provided to the user depending on which electronic device is used. For example, if an HMD is used, since at least one display is directly placed in front of the user's eyeballs, the electronic device can provide a virtual reality (VR) space to the user, and the user can view 360 degrees in the virtual reality space. You can also watch the video.

According to various embodiments, the second electronic device 320 can share the 360-degree video being captured by the first electronic device 310 in real time and play it back in a streaming manner. According to another embodiment, the second electronic device 320 may completely download the 360-degree video from the first electronic device 310 and play it back.

The first electronic device 310 according to various embodiments may be connected to the second electronic device 320 through the network 330 . The network 330 may be configured as a wired and/or wireless communication network. Wireless communication is, for example, LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications) may include cellular communication using at least one of the like. According to one embodiment, wireless communication, for example, WiFi (wireless fidelity), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, near field communication (NFC), magnetic secure transmission (Magnetic Secure Transmission), radio It may include at least one of a frequency (RF) and a body area network (BAN).

According to an embodiment, at least one of these components may be omitted or one or more other components may be added to the image providing system. For example, the video providing system may include a server (eg, the server 108 of FIG. 1 ). The first electronic device 310 and the second electronic device 320 are connected to each other through a server and can exchange data through the server. As another example, the first electronic device 310 may process an image or create a 360-degree video using a server. The 360-degree video generated by the server may be transmitted to the second electronic device 320 through the first electronic device 310, or the server may directly transmit it to the second electronic device 320.

4A is a flowchart illustrating a method of providing an image by an electronic device according to various embodiments.

Referring to FIG. 4A , in operation 410, a processor (eg, processor 120 of FIG. 1 or FIG. 2 ) of an electronic device (eg, electronic device 101 of FIG. 1 or first electronic device 310 of FIG. 3 ). The image signal processor 260 of may obtain an image at a specified frame rate using an image sensor (eg, the image sensor 230 of FIG. 2 ). For example, the image sensor may obtain an image by converting light collected by a lens unit (eg, the lens assembly 210 of FIG. 2 ) into an electrical signal. The image generated by the image sensor may be a raw image. The processor may control at least two or more image sensors to obtain at least two or more raw images corresponding to each of the two or more image sensors. The processor may simultaneously control at least two or more image sensors to acquire temporally synchronized images according to a specified frame rate, and images obtained from each image sensor may be stored in a memory (eg, the memory 130 of FIG. 1 or the memory 130 of FIG. 2 ). It can be divided and accumulated in the memory 250).

In operation 420, the processor of the electronic device may use a sensor (eg, the sensor module 176 of FIG. 1 ) to determine the movement variation of the electronic device.

According to various embodiments, the processor may use a sensor to recognize a state (eg, movement and/or rotation) of the electronic device and determine a movement variation of the electronic device. For example, the processor may determine whether the electronic device moves and/or rotates by using an acceleration sensor. As another example, the processor may check the state of the electronic device using the gyro sensor.

According to various embodiments, the processor may combine measurement values measured by a plurality of sensors in order to more accurately determine the state of the electronic device. For example, by combining a value measured by an acceleration sensor and/or a gyro sensor with a value measured by a geomagnetic sensor, it is possible to more accurately determine the amount of movement change of the electronic device.

In operation 430, the processor of the electronic device transmits the captured image to a specified external device (eg, the electronic device 102 or 104 of FIG. 1 or the second electronic device 320 of FIG. )) can be sent.

According to various embodiments, the first range may be designated as a range in which a user who is provided with a 360-degree image may not feel uncomfortable due to a relatively small amount of motion variation of the electronic device. For example, if the angular velocity at which the user feels dizzy is 'A', a case where the amount of motion change is smaller than 'A' may be designated as the first range. In this case, the electronic device can directly provide the image being captured or the captured image to the designated external device so that the designated external device can reproduce the 360-degree video.

In operation 440, the processor of the electronic device generates information based on at least one of an image or a motion change amount when the motion change amount falls within the specified second range, and sends the generated information to the specified external device (eg, FIG. 1 ). It can be transmitted to the electronic device 102 or 104 or the second electronic device 320 of FIG. 3 .

According to various embodiments, the second range may be designated as a range in which a user receiving a 360-degree image may feel uncomfortable due to a large movement change of the electronic device. For example, if the angular velocity at which the user feels dizzy is 'A', a case where the amount of motion change is greater than 'A' may be designated as the second range.

According to various embodiments, the processor may generate information based on at least one of an image or a motion variation. For example, the processor may correct and/or edit at least a portion of the image to adjust the 360-degree image so that the user does not feel uncomfortable. As another example, the processor may generate information related to the movement of the electronic device and transmit the information to the external device so that the external device recognizes the state of the electronic device.

According to various embodiments, the generated information may include information related to a motion change amount belonging to the second range. For example, while the motion change amount falls within the specified second range, the processor may temporarily suspend transmission of the image being captured, generate a message indicating a state of the electronic device, and transmit the message to the external device. The external device may receive the message from the electronic device and recognize the state of the electronic device while the movement variation of the electronic device falls within the designated second range. According to one embodiment, the user of the external device can recognize the state of the electronic device based on the message. For example, the processor may generate a message including a phrase and/or an emoticon notifying that the electronic device is rotating, transmit the message to the external device, and display the message on the display of the external device so that the user recognizes it. . Referring to FIG. 4B , a screen notifying that an electronic device is rotating is shown according to various embodiments. For example, the external device 320 may improve the user experience of the user using the external device 320 by stopping the image being captured or reproducing the captured image and displaying the screen SC1 shown in FIG. 4B. .

While transmission of the image being captured is temporarily suspended, the processor may continuously check whether the amount of change in motion falls within a specified first range, and when the amount of change in motion falls within the first range, the processor transmits the image being captured to an external device. An external device can be used to play video.

According to various embodiments, the generated information may include an edited image from which a specified number of frames are excluded from the image. The edited video may be generated by, for example, adjusting the frame rate of the video being captured. According to an embodiment, the processor may adjust an image captured at a first frame rate to an image of a second frame rate having a lower speed than the first frame rate. When an image is provided at a high frame rate, a smooth and natural display is possible, but an image captured while the amount of motion change falls within the designated second range may cause a user receiving the image to feel dizzy. Accordingly, the processor may compensate and provide the image being captured at a frame rate lower than the reference frame rate while the motion change amount falls within the designated second range, and through this, the user receiving the image may not feel relatively dizzy. . For example, the processor may modify an image captured at 30 to 60 fps (frames per second) to a 10 to 20 fps image. Referring to FIG. 4C , a method of adjusting a frame rate according to an exemplary embodiment is illustrated. For example, in an image captured at 30 fps, by removing two frames out of every three frames, an image of 10 fps may be obtained. Since an image adjusted to a low frame rate does not cause dizziness to a user, user experience can be improved.

While providing an image of a low frame rate to an external device, the processor continuously checks whether the amount of change in motion falls within a specified first range, and when the amount of change in motion falls within the first range, the image being captured is returned to the reference frame rate. can be corrected and transmitted to an external device.

According to various embodiments, the generated information may include information on other electronic devices related to the electronic device. For example, in a situation in which real-time broadcasting is relayed using a plurality of electronic devices, information on other electronic devices close to the electronic device may be provided to maintain continuity of a scene being provided. For example, a user who has been provided with an image using a first electronic device may receive information related to a second electronic device in which the amount of change in motion of the first electronic device falls within the first range while the amount of change in motion of the first electronic device falls within the second range. 2 You can watch the video captured by the electronic device. The processor of the first electronic device checks whether the motion change amount falls within the specified first range, and when the motion change amount falls within the first range, the image being captured may be transmitted to the external device again.

According to various embodiments, the generated information may include an image corrected according to a motion change amount. For example, the processor may correct an image being photographed according to a reference direction while the change in motion of the electronic device falls within the designated second range, and provide the corrected image to the external device. The reference direction may be variously determined according to predetermined rules, such as, for example, a moving direction of an electronic device or a direction in which an object is located. According to an embodiment, the processor may reassemble images acquired by a plurality of image sensors according to a reference direction and produce a 360-degree image.

5 is a diagram illustrating a method of generating a corrected image according to an amount of motion change by an electronic device, according to various embodiments.

5(a) shows a general method of generating a frame without considering motion of an electronic device. The processor may control each of the image sensors (eg, cam1, cam2, and cam3) synchronized in time according to the frame rate, and obtain a plurality of images corresponding to each of the image sensors. The processor may generate one frame of a 360-degree video by stitching the plurality of acquired images. According to one embodiment, the processor may generate a frame by analyzing features (eg, location, distance, or color) included in each image and connecting matched features. The processor may accumulate frames according to the frame rate and produce a 360-degree video. Since the 360-degree video produced according to this method is based on the field of view of the image sensor, when the image sensor moves (eg, rotates) to capture an image, the screen is also output while rotating. In some cases, a user watching this may feel dizzy due to fast screen switching.

5(b) is a diagram illustrating a method of generating a frame by recombining an image according to a motion of an electronic device. The processor may recombine the plurality of images acquired by each of the image sensors (eg, cam1, cam2, and cam3) before stitching the plurality of images. For example, the processor may determine a reference direction (eg, a moving direction of an electronic device or a direction in which an object is located) and reconstruct images according to the reference direction. According to an embodiment, the processor may check a reference image facing the reference direction and adjust other images to be disposed around the reference image. For example, the processor may use at least a part of an image obtained by a specific image sensor (eg, cam3) as a reference image, or may use at least a part of an image obtained by a first image sensor (eg, cam1) and a second image sensor (eg, cam1). cam3) may use at least a part of the acquired image as a reference image. When arrangements of other images are adjusted around the reference image, the processor may create one frame by stitching the images. By generating and accumulating a plurality of frames in this way, a 360-degree video can be produced. The 360-degree video produced according to this method can provide a user with a non-rotating screen with a fixed field of view in the reference direction. As a result, the user can be provided with an endless and continuous image, and user convenience can be increased.

According to another embodiment, the processor stitches a plurality of images acquired by each of the image sensors (eg, cam1, cam2, and cam3), and then converts (eg, rotates) the direction of the stitched images according to a reference direction. A 360-degree image with a fixed field of view in a reference direction may be generated.

6A to 6C are diagrams illustrating a method of generating a corrected image according to an amount of motion change by an electronic device, according to various embodiments.

According to various embodiments, an electronic device may include a ball-shaped housing. The electronic device may include a plurality of image sensors disposed using at least a portion of the housing. For example, the electronic device may be composed of a soccer ball, and when a soccer game is relayed, a 360-degree image taken from the soccer ball may be delivered to a viewer in real time. 6A to 6C show a ball-shaped electronic device, but various embodiments of the present invention are not intended to be limited thereto, and the electronic device may be configured in various shapes. Also, at least two or more image sensors may be disposed in various forms.

Referring to FIG. 6A , a case in which an electronic device configured in a ball shape rotates and moves is illustrated. For example, the electronic device rotates in the direction of the roll Φ and may move in a direction parallel to the z-axis.

6b and 6c illustrate a method of providing an image taken by the electronic device of FIG. 6a according to various embodiments.

6A illustrates a case in which an electronic device moves with a motion change amount belonging to a designated first range. According to one embodiment, the field of view of the viewer viewing the 360-degree video in real time may coincide with the field of view of camera #1. In this case, if the viewer watches the 360-degree video in a static state, the viewer can view a screen that matches the field of view of camera #1. For example, a viewer can watch a screen that rotates in a direction of roll (Φ) and moves.

6C shows a case where the electronic device moves with a motion change amount belonging to a designated second range.

The processor may set the movement direction of the electronic device as a reference direction and generate a corrected image according to the amount of change in movement. For example, the processor may recombine the image in consideration of at least one of an arrangement structure of a camera (eg, an image sensor), a frame rate, and a rotational angular velocity of a rotating electronic device. Taking the electronic device of FIG. 6A as an example, cameras in the electronic device are arranged at intervals of 90 degrees, and the frame rate may be 30 fps.

In this case, if the rotational angular velocity of the electronic device is measured as 7.5π/s, the reference screen of the first frame is determined using the image acquired by camera #1, and the second frame is a part of the image acquired by camera #1. and #3 may be determined using a part of the image acquired by the camera. The third frame may be determined using a part of an image acquired by camera #3 and a part of image acquired by camera #5. The 360-degree video produced according to this method can provide a non-rotating screen to the user by fixing the field of view in the reference direction.

According to various embodiments, all or part of a series of processes performed by an electronic device (eg, the electronic device 101 in FIG. 1 or the first electronic device 310 in FIG. 3 ) may be performed by a designated external device (eg, the electronic device 101 in FIG. It can be performed simultaneously or separately by the first electronic device 102 or 104 or the second electronic device 320 of FIG. 3 or a server (eg, the server 108 of FIG. 1 ).

7 is a flowchart illustrating a method of providing an image by an electronic device according to various embodiments.

Referring to operation 710, an electronic device (eg, the electronic device 102 or 104 of FIG. 1, the second electronic device 320 of FIG. 3 or the server 108 of FIG. 1) uses a communication module to use an external device ( Example: It is possible to receive an image generated at a specified frame rate from the electronic device 101 of FIG. 1 and information related to a motion variation of the external device.

According to various embodiments, an electronic device may receive an unprocessed image (eg, a raw image) acquired by an image sensor of an external device or an image obtained by processing the image.

According to various embodiments, an electronic device may receive information related to a movement variation of an external device. For example, the electronic device may receive a measurement value of a sensor included in the external device or data about a variation in movement of the external device determined by the external device using the sensor.

Referring to operation 720, the electronic device may distinguish a first section belonging to a first range in which the amount of motion change in the received image is specified and a second section belonging to a second specified range, based on the information related to the amount of motion change in the received image. .

For example, if the angular velocity at which the user can feel dizzy is 'A', a section with a smaller motion change than 'A' is designated as a first section, and a section with a motion change greater than or equal to 'A' is designated as a second section. It can be specified as an interval.

Referring to operation 730, the electronic device may generate information corresponding to the second section based on at least one of the image and the amount of motion change. For example, the electronic device may correct and/or edit at least a portion of the image corresponding to the second section to adjust the 360-degree image so that the user does not feel uncomfortable. As another example, the electronic device may generate information related to the movement of the external device and display information allowing the user to recognize the state of the external device.

According to various embodiments, the generated information may include information related to a motion change amount belonging to the second range. For example, the electronic device may temporarily stop providing an image corresponding to the second section, generate a message indicating a state of the external device, and display the message on the display. For example, the screen shown in FIG. 4B may be displayed.

According to various embodiments, the generated information may include an edited image from which a specified number of frames are excluded from the image. For example, the electronic device may correct and provide the image corresponding to the second section at a frame rate lower than the reference frame rate. For example, the frame rate may be adjusted using the method shown in FIG. 4C.

According to various embodiments, the generated information may include information on other electronic devices related to the electronic device. For example, during the second period, an image captured by another external device may be provided by receiving information related to another external device.

According to various embodiments, the generated information may include an image corrected according to a motion change amount. For example, the electronic device may correct the image corresponding to the second section according to the reference direction and display the corrected image on the display. For example, an image may be corrected using the method shown in FIGS. 6A to 6C.

Referring to operation 740, the electronic device may provide the image during the first section and the generated information during the second section using a display.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B," "A, B or C," "at least one of A, B and C," and "A Each of the phrases such as "at least one of , B, or C" may include all possible combinations of the items listed together in the corresponding one of the phrases. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in this document may include a unit implemented by hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. Device-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smartphones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the components described above may include a singular entity or a plurality of entities. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

Claims (20)

In electronic devices,
housing;
sensor;
an image sensor disposed using at least a portion of the housing; and
It includes a processor, the processor comprising:
Obtaining an image at a specified frame rate using the image sensor;
Checking the movement variation of the electronic device using the sensor;
When the motion change amount falls within a specified first range, transmitting the video to a specified external device;
When the motion variation amount belongs to a designated second range, information is generated based on at least one of the image or the motion variation amount, and the generated information is set to be transmitted to the designated external device.
The generated information includes an edited image from which a specified number of frames are excluded from the image. According to claim 1,
The generated information includes information related to a motion change amount belonging to the second range. delete According to claim 1,
The generated information includes information of another electronic device related to the electronic device. According to claim 1,
The generated information includes an image corrected according to the motion change amount. The method of claim 5, wherein the processor,
set the reference direction,
An electronic device configured to generate the corrected image according to the reference direction. According to claim 6,
The electronic device of claim 1, wherein the reference direction includes one of a movement direction of the electronic device and a direction in which an object is located. In electronic devices,
communication module;
display; and
It includes a processor, the processor comprising:
Receiving an image generated at a specified frame rate from an external device and information related to a motion variation of the external device using the communication module;
Based on the information related to the amount of motion change, a first period belonging to a first range in which the motion change in the received image is designated and a second period belonging to a designated second range are distinguished,
Based on at least one of the image and the amount of motion change, generating information corresponding to the second section;
It is set to provide the image during the first period using a display and to provide the generated information during the second period,
The generated information includes an edited image from which a specified number of frames are excluded from the image. According to claim 8,
The generated information includes information related to a motion change amount belonging to the second range. delete According to claim 8,
The generated information includes information of another external device related to the external device. According to claim 8,
The generated information includes an image corrected according to the motion change amount. ◈Claim 13 was abandoned when the registration fee was paid.◈ The method of claim 12, wherein the processor,
set the reference direction,
An electronic device configured to generate the corrected image according to the reference direction. ◈Claim 14 was abandoned when the registration fee was paid.◈ According to claim 13,
The reference direction includes a moving direction of the external device or a direction in which an object is located. ◈Claim 15 was abandoned when the registration fee was paid.◈ A method for providing an image using an electronic device,
Obtaining an image at a specified frame rate;
checking a movement variation of the electronic device;
transmitting the image to a designated external device when the amount of motion change falls within a designated first range; and
generating information based on at least one of the image or the motion variation and transmitting the generated information to the designated external device when the motion variation amount belongs to a designated second range;
The generated information includes an edited image from which a specified number of frames are excluded from the image. ◈Claim 16 was abandoned when the registration fee was paid.◈ According to claim 15,
The generated information includes information related to a motion change amount belonging to the second range. delete ◈Claim 18 was abandoned when the registration fee was paid.◈ According to claim 15,
The generated information includes information of another electronic device related to the electronic device. ◈Claim 19 was abandoned when the registration fee was paid.◈ According to claim 15,
The generated information includes an image corrected according to the amount of motion change. ◈Claim 20 was abandoned when the registration fee was paid.◈ According to claim 19,
an operation of setting a reference direction; and
and generating the corrected image according to the reference direction.

Images