KR20220099827A - Electronic apparatus and control method thereof - Google Patents

Abstract

Disclosed is an electronic device. The electronic device includes a first camera, a second camera, and a processor connected to the first camera and the second camera to control the electronic device. When a front image of the electronic device is acquired through the first camera, the processor can identify a direction of the face of a user from the front image, identify a location of an object behind the electronic device based on the orientation of the face of the user, and crop an area of a rear image of the electronic device acquired through the second camera based on the location of the object.

Description

Electronic device and its control method { ELECTRONIC APPARATUS AND CONTROL METHOD THEREOF }

The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for acquiring an image by photographing an object and a method for controlling the same.

Recently, as the performance of the built-in camera in a smartphone has rapidly developed, the demand for video recording using a smartphone has rapidly increased. Accordingly, smartphone manufacturers are developing products focusing on enhancing camera performance and usability.

However, there has not been much progress in shooting a moving object (subject) so far. When the object is still, there is little difficulty in shooting a video, but when shooting a moving object, the user must gaze at the screen of the smartphone and focus on the movement of the subject in the image. Otherwise, the object goes out of the camera angle of view of the smartphone or off the center of the screen. In particular, when shooting a moving object with a smartphone, the object may be close-up in order to take a picture so that the object can be seen clearly on the screen. It is highly probable that this will be created

That is, there is a problem in that the user cannot focus on the movement of the actual object while photographing the object, and must keep an eye on the smartphone screen.

Accordingly, there is a need to develop a function that allows the user to take a picture without missing an object without looking at the screen of the smartphone.

SUMMARY OF THE INVENTION The present disclosure has been made in accordance with the above-mentioned necessity, and an object of the present disclosure is to provide an electronic device for acquiring an image including an object to be photographed by a user by identifying the location of the object, and a method for controlling the same.

According to an embodiment of the present disclosure for achieving the above object, an electronic device includes a first camera, a second camera, and a processor connected to the first camera and the second camera to control the electronic device, , when the front image of the electronic device is obtained through the first camera, the processor identifies the user's face direction from the front image, and determines the position of the object in the rear of the electronic device based on the user's face direction and, based on the position of the object, a region of the rear image of the electronic device acquired through the second camera may be cropped.

In addition, the processor acquires a plurality of rear images through the second camera, crops one region of each of the plurality of rear images based on the position of the object, and based on the cropped plurality of rear image regions A video including a plurality of acquired frames may be acquired.

In addition, the processor acquires a plurality of front images through the first camera while the plurality of rear images are acquired, and when it is identified from the plurality of front images that the direction of the user's face is changed, the changed user's face The changed position of the object may be identified based on the direction, and a different region may be cropped in each of the plurality of rear images based on the position of the object and the changed position of the object.

Further, the processor is further configured to identify a location in the image of the object from each of the plurality of back images based on the location of the object, and to crop a different region in each of the plurality of back images based on the location in the image of the object. can

In addition, when the position of the object is outside the angle of view of the second camera, the processor may provide guide information warning that the object is not captured.

In addition, the processor acquires a plurality of front images through the first camera while a plurality of rear images are obtained through the second camera, and the user's face direction is changed from the plurality of front images When the object to which the face direction faces is changed from the first object to the second object, the plurality of objects based on at least one of the first object or the second object based on a time when the face direction of the user faces the second object One region of each of the rear images of may be cropped.

In addition, the processor is configured to crop one region of each of the plurality of rear images based on the first object when a time in which the user's face direction faces the second object is less than a first threshold time, and the user's face direction If the time towards the second object is longer than the first threshold time and less than the second threshold time, crop a region of each of the plurality of rear images based on the first object and the second object, When the time for the face direction toward the second object is equal to or greater than the second threshold time, one region of each of the plurality of rear images may be cropped based on the second object.

In addition, when an image capturing command or a moving picture capturing command through the second camera is input, the processor may turn on the first camera to photograph the front of the electronic device to obtain the front image.

In addition, the processor may identify the user's face from the front image, and identify the user's face direction based on a plurality of feature points inside the user's face.

Meanwhile, according to an embodiment of the present disclosure, a method of controlling an electronic device includes: when a front image of the electronic device is acquired through a first camera, identifying a direction of the user's face from the front image; identifying the position of the object at the rear of the electronic device based on .

In addition, the method further includes acquiring a plurality of rear images through the second camera, wherein the cropping includes cropping a region of each of the plurality of rear images based on the position of the object, the control method comprising: The method may further include acquiring a video including a plurality of frames acquired based on the plurality of cropped rear image regions.

And, the acquiring of the plurality of rear images includes acquiring a plurality of front images through the first camera while the plurality of rear images are acquired, and the step of identifying the position of the object is performed from the plurality of front images. When it is identified that the user's face direction is changed, the changed position of the object is identified based on the changed face direction of the user, and the cropping includes the plurality of the plurality of objects based on the position of the object and the changed position of the object. You can crop a different area in each of the back images.

In addition, the cropping may include identifying a position in the image of the object from each of the plurality of rear images based on the position of the object, and selecting a different region in each of the plurality of rear images based on the position in the image of the object. can be cropped

The method may further include providing guide information warning that the object is not captured when the position of the object is outside the angle of view of the second camera.

In addition, the method further includes acquiring a plurality of front images through the first camera while a plurality of rear images are acquired through the second camera, wherein the cropping includes the user's face from the plurality of front images When the direction is changed and the object to which the user's face direction is directed is changed from the first object to the second object, one of the first object or the second object based on the time when the user's face direction faces the second object One region of each of the plurality of rear images may be cropped based on at least one.

In addition, the cropping includes cropping one region of each of the plurality of rear images based on the first object when the time in which the face direction of the user faces the second object is less than a first threshold time, and When a time in which a face direction faces the second object is longer than the first threshold time and less than a second threshold time, crop a region of each of the plurality of rear images based on the first object and the second object, wherein If the time for the user's face toward the second object is equal to or greater than the second threshold time, one region of each of the plurality of rear images may be cropped based on the second object.

The method may further include, when a video capturing command is input through the second camera, turning on the first camera to photograph the front of the electronic device to obtain the front image.

In addition, the identifying of the user's face direction may include identifying the user's face from the front image, and identifying the user's face direction based on a plurality of feature points inside the user's face.

Meanwhile, according to an embodiment of the present disclosure, in a non-transitory computer-readable recording medium storing a program for executing a method of operating an electronic device, the operating method includes obtaining a front image of the electronic device through a first camera , identifying the user's face direction from the front image, identifying the position of the object at the rear of the electronic device based on the user's face direction, and acquiring through a second camera based on the position of the object and cropping a region of the rear image of the electronic device.

According to various embodiments of the present disclosure as described above, the electronic device identifies the direction of the user's face, identifies the position of the object based on the direction of the user's face, and selects a region of an image obtained based on the position of the object. By cropping, an image including an object that the user wants to photograph may be acquired.

In addition, since the electronic device automatically identifies the object based on the direction of the user's face, the user can take a picture while looking at the object without the need for the user to look at the shooting screen during shooting, thereby increasing user convenience.

1A is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.
1B is a block diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure.
2 is a diagram for explaining a structure of an electronic device according to an embodiment of the present disclosure.
3A to 3D are diagrams for explaining a method of identifying a user according to an embodiment of the present disclosure.
4 is a diagram for explaining a method of identifying a location of an object according to an embodiment of the present disclosure.
5 is a view for explaining a cropping operation according to an embodiment of the present disclosure.
6 is a diagram for explaining a difference between a Field of View (FoV) and a Field of Interest (FoI) according to an embodiment of the present disclosure.
7A and 7B are diagrams for explaining a change in FoI according to various embodiments of the present disclosure.
8 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

The terms used in the embodiments of the present disclosure are selected as currently widely used general terms as possible while considering the functions in the present disclosure, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in specific cases, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

In the present specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

The expression “at least one of A and/or B” is to be understood as indicating either “A” or “B” or “A and B”.

As used herein, expressions such as "first," "second," "first," or "second," can modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In this specification, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

Hereinafter, an embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings.

1A is a block diagram illustrating a configuration of an electronic device 100 according to an embodiment of the present disclosure. However, the present invention is not limited thereto, and the electronic device 100 may be implemented in a state in which some of the components of FIG. 1A are omitted.

The electronic device 100 is a device that acquires an image by photographing an object, and may be a device such as a smartphone, a tablet PC, a digital camera, or smart glasses. However, the present invention is not limited thereto, and the electronic device 100 may be any device as long as it is capable of acquiring an image by photographing an object.

Referring to FIG. 1A , an electronic device 100 includes a first camera 110 , a second camera 120 , and a processor 130 .

The first camera 110 and the second camera 120 are configured to capture a still image or a moving image under the control of the processor 130 . The first camera 110 and the second camera 120 may capture still images at a specific point in time, but may also continuously capture still images.

The first camera 110 and the second camera 120 may capture an image of an area around the electronic device 100 . For example, the first camera 110 and the second camera 120 may capture images around the electronic device 100 with an angle of view of 71 degrees. However, the present invention is not limited thereto, and the first camera 110 and the second camera 120 may capture images around the electronic device 100 at different angles of view.

The first camera 110 may be arranged to photograph the front of the electronic device 100 , and the second camera 120 may be arranged to photograph the rear of the electronic device 100 . For example, the electronic device 100 is implemented as a bar type, the first camera 110 is disposed on the front of the electronic device 100 , and the second camera 120 is disposed on the rear of the electronic device 100 . ) can be placed. In particular, the first camera 110 may be disposed on the front side of the electronic device 100 together with a display to be described later.

Each of the first camera 110 and the second camera 120 may include a lens, a shutter, an aperture, a solid-state imaging device, an analog front end (AFE), and a timing generator (TG). The shutter controls the time at which the light reflected on the subject enters each of the first camera 110 and the second camera 120, and the iris mechanically increases or decreases the size of the opening through which light enters, thereby reducing the amount of light incident on the lens. can be adjusted The solid-state imaging device may output an image of the photocharge as an electrical signal when light reflected by the subject is accumulated as photocharges. The TG outputs a timing signal for reading out pixel data of the solid-state imaging device, and the AFE may sample and digitize the electrical signal output from the solid-state imaging device.

The processor 130 controls the overall operation of the electronic device 100 . Specifically, the processor 130 may be connected to each component of the electronic device 100 to control the overall operation of the electronic device 100 . For example, the processor 130 may be connected to the first camera 110 and the second camera 120 to control the operation of the electronic device 100 .

According to an embodiment, the processor 130 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON). However, the present invention is not limited thereto, and the central processing unit ( central processing unit (CPU)), micro controller unit (MPU), micro processing unit (MPU), controller, application processor (AP), or communication processor (CP), ARM processor In addition, the processor 130 may be implemented as a SoC (System on Chip) or LSI (large scale integration) in which a processing algorithm is embedded, or an FPGA ( Field programmable gate array) may be implemented.

When the front image of the electronic device 100 is acquired through the first camera 110 , the processor 130 may identify the direction of the user's face from the front image. For example, when an image capturing command or a video capturing command is input through the second camera 120 , the processor 130 turns on the first camera 110 to photograph the front of the electronic device 110 to obtain a front image. can be obtained. In addition, the processor 130 may identify the user's face from the front image and identify the user's face direction based on a plurality of feature points inside the user's face. For example, the processor 130 may identify how much the user's face is rotated with respect to the electronic device 100 based on distances between a plurality of feature points inside the user's face.

However, the present invention is not limited thereto, and the processor 130 may identify the user's gaze direction based on the user's eyes. Hereinafter, it will be described that the user's face direction is used for convenience of explanation, but the user's gaze direction may also be used.

The processor 130 identifies the position of the object at the rear of the electronic device 100 based on the direction of the user's face, and when the rear image of the electronic device 100 is acquired through the second camera 120 , the position of the object A region of the rear image may be cropped based on .

The processor 130 may crop a region of the rear image based on the position of the object in various ways. For example, the processor 130 identifies that the object is located on the right side at the rear of the electronic device 100 based on the direction of the user's face, and determines the right region of the rear image acquired through the second camera 120 . can be cropped Alternatively, the processor 130 divides the image into four regions of the upper left, upper right, lower left, and lower right of the image in a matrix form, identifies an object in one of the four regions based on the direction of the user's face, and the position of the object A region of the rear image may be cropped based on . In this case, the processor 130 may identify the object in one of the four areas using the neural network model. Alternatively, the processor 130 obtains the angle of the object with respect to the front direction of the second camera 120, identifies the position of the object in a partial region of the rear image corresponding to the angle, and A region of the back image may be cropped based on the position.

The processor 130 may crop an area of a preset size from the rear image based on the position of the object. However, the present invention is not limited thereto, and the processor 130 may identify the size of the object in the rear image and determine the size of the area to be cropped based on the identified size of the object.

The processor 130 acquires a plurality of rear images through the second camera 120 , crops one region of each of the plurality of rear images based on the position of the object, and obtains based on the plurality of cropped rear image regions A video including a plurality of frames may be acquired. That is, the processor 130 may capture a video and crop each image frame based on the position of the object.

Here, the processor 130 acquires a plurality of front images through the first camera 110 while a plurality of rear images are acquired, and when it is identified that the direction of the user's face is changed from the plurality of front images, the changed user's face A changed position of the object is identified based on the direction, and a different region may be cropped in each of the plurality of back images based on the position of the object and the changed position of the object.

For example, the processor 130 may acquire a first front image while a first rear image is acquired, and acquire a second rear image while a second rear image following the first rear image is acquired. Then, if the user's face direction of the first front image is changed from the second front image, the processor 130 crops the first rear image based on the position of the object before the user's face direction is changed, and the user's face The second back image may be cropped based on the position of the object after the direction is changed.

Alternatively, the processor 130 may identify a position in the image of the object from each of the plurality of rear images based on the position of the object, and crop a different region in each of the plurality of rear images based on the position in the image of the object. That is, the processor 130 may identify the object from the rear image based on the position of the object, and crop one region of each of the rear images by further considering the identified position in the image of the object. In this case, the accuracy of cropping may be improved compared to the case of simply cropping using the position of the object.

Meanwhile, when the position of the object is outside the angle of view of the second camera 120 , the processor 130 may provide guide information warning that the object is not captured. For example, when the object is positioned outside the angle of view of the second camera 120 as the object moves, the processor 130 may display guide information warning that the object is not captured or output a warning sound as a sound. can Alternatively, the processor 130 may provide a warning through the blinking of an LED.

Meanwhile, the processor 130 acquires a plurality of front images through the first camera 110 while a plurality of rear images are acquired through the second camera 120 , and the direction of the user's face is changed from the plurality of front images When the object to which the user's face direction faces is changed from the first object to the second object, the plurality of rearwards based on at least one of the first object or the second object based on the time when the user's face direction faces the second object One region of each image may be cropped.

For example, the processor 130 crops one region of each of the plurality of rear images based on the first object when the time for the user's face direction toward the second object is less than the first threshold time, and the user's face direction is If the time towards the second object is longer than the first threshold time and less than the second threshold time, one area of each of the plurality of rear images is cropped based on the first object and the second object, and the user's face direction is the second object If the facing time is equal to or greater than the second threshold time, one region of each of the plurality of rear images may be cropped based on the second object.

That is, when the user stares at another place for a while (when the time for the user's face direction toward the second object is less than the first threshold time), the processor 130 identifies the object that the user wants to photograph as not changed, and You can crop the back image based on the original object. Then, the processor 130 crops the rear image based on the first object and the second object when the time for the user's face toward the second object is longer than the first threshold time and less than the second threshold time, so that the cropped image is abrupt. can be prevented from being changed. For example, while cropping the plurality of rear images, the processor 130 may crop the area including the first object in a stepwise manner, and if the second object is included, the area near the first object may be cropped in a stepwise manner. can In addition, the processor 130 identifies that the user's object of interest has been changed only when a threshold time has elapsed for the time the user gazes at the second object (when the user's face direction is equal to or greater than the second threshold time) can do.

However, the present invention is not limited thereto, and the processor 130 crops one region of each of the plurality of rear images based on the first object when the time for the user's face toward the second object is less than the first threshold time, and the user If the time in which the face direction of ' is directed toward the second object is equal to or greater than the first threshold time, one region of each of the plurality of rear images may be cropped based on the second object.

In the above, it has been described that the object is changed based on the direction of the user's face, but the present invention is not limited thereto. For example, the processor 130 may identify the position of the object based on the direction of the user's face, and identify the object from the rear image. The processor 130 may perform this operation for each of the plurality of rear images, and when the shape of an object identified from each of the plurality of rear images is changed, the above operation may be performed.

1B is a block diagram illustrating a detailed configuration of an electronic device 100 according to an embodiment of the present disclosure. The electronic device 100 may include a first camera 110 , a second camera 120 , and a processor 130 . Also, according to FIG. 1B , the electronic device 100 may further include at least one of a display 140 , a memory 150 , a user interface 160 , a communication interface 170 , and a speaker 180 . Among the components shown in FIG. 1B , detailed descriptions of parts overlapping with those shown in FIG. 1A will be omitted.

The display 140 may be implemented with various types of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), and the like. The display 140 may also include a driving circuit, a backlight unit, and the like, which may be implemented in a form such as an a-si TFT, a low temperature poly silicon (LTPS) TFT, or an organic TFT (OTFT). Meanwhile, the display 140 may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display, or the like.

The display 140 may be disposed in front of the electronic device 100 . However, the present invention is not limited thereto, and the display 140 may have any shape as long as it is disposed on the same surface as the first camera 110 .

The display 140 may display the rear image captured by the second camera 120 in real time under the control of the processor 130 . Also, the display 140 may provide guide information for warning that the object is not photographed under the control of the processor 130 .

The memory 150 may refer to hardware that stores information such as data in an electrical or magnetic form so that the processor 130 can access it. To this end, the memory 150 may be implemented as hardware at least one of non-volatile memory, volatile memory, flash memory, hard disk drive (HDD) or solid state drive (SSD), RAM, ROM, etc. .

At least one instruction or module required for the operation of the electronic device 100 or the processor 130 may be stored in the memory 150 . Here, the instruction is a unit of code for instructing the operation of the electronic device 100 or the processor 130 and may be written in machine language, which is a language that a computer can understand. A module may be a set of instructions that perform a specific task of a unit of work.

The memory 150 may store data that is information in units of bits or bytes that can represent characters, numbers, images, and the like. For example, the memory 150 may store a face identification module, a face direction identification module, an object location identification module, an object identification module, a crop module, and the like. Here, at least one of the face identification module, the face direction identification module, the object position identification module, the object identification module, and the crop module may be implemented as a neural network model.

The memory 150 is accessed by the processor 130 , and reading/writing/modification/deletion/update of instructions, modules, or data may be performed by the processor 130 .

The user interface 160 may be implemented as a button, a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen capable of performing the above-described display function and manipulation input function together. Here, the button may be various types of buttons such as a mechanical button, a touch pad, a wheel, etc. formed in an arbitrary area such as the front, side, or rear of the exterior of the main body of the electronic device 100 .

The communication interface 170 is configured to communicate with various types of external devices according to various types of communication methods. For example, the electronic device 100 may receive various modules from an external device through the communication interface 170 .

The communication interface 170 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

The Wi-Fi module and the Bluetooth module perform communication using a WiFi method and a Bluetooth method, respectively. In the case of using a Wi-Fi module or a Bluetooth module, various types of connection information such as an SSID and a session key are first transmitted and received, and various types of information can be transmitted/received after a communication connection is made using this. The infrared communication module communicates according to the infrared data association (IrDA) technology, which wirelessly transmits data in a short distance using infrared that is between visible light and millimeter waves.

In addition to the above-described communication method, the wireless communication module includes Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE Advanced (LTE-A), 4th Generation (4G), 5G It may include at least one communication chip that performs communication according to various wireless communication standards, such as (5th Generation).

Alternatively, the communication interface 170 may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, or the like.

In addition, the communication interface 170 may include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module for performing communication using a pair cable, a coaxial cable, or an optical fiber cable.

The speaker 180 may receive an electrical signal, and may radiate sound by vibrating a diaphragm based on the input electrical signal. The speaker 180 may output a sound warning that the object is not photographed under the control of the processor 130 .

As described above, the electronic device 100 identifies the position of the object based on the direction of the user's face, and crops a region of the image obtained based on the identified position of the object, so that the user can keep an eye on the shooting screen while shooting. You can acquire an image containing an object without any need.

Hereinafter, the operation of the electronic device 100 will be described in more detail with reference to FIGS. 2 to 7 . 2 to 7 , individual embodiments will be described for convenience of description. However, the individual embodiments of FIGS. 2 to 7 may be embodied in any combination.

2 is a view for explaining the structure of the electronic device 100 according to an embodiment of the present disclosure.

The electronic device 100 may include the first camera 110 on the front side and the second camera 120 on the rear side. Also, the electronic device 100 may include the first camera 110 and the display 140 on the front side and the second camera 120 on the rear side.

In addition, the electronic device 100 may be embodied in any number of forms, and the photographing directions of the first camera 110 and the second camera 120 are opposite to each other, and the first camera 110 and the second camera ( 120) may be used in any form as long as it is a device capable of simultaneously driving.

3A to 3D are diagrams for explaining a method of identifying a user according to an embodiment of the present disclosure.

,

로서 표현하면, 하기의 수학식 1이 성립한다.The processor 130 may identify the location of the user from the front image. First, as shown in FIG. 3A , the user's position is determined with respect to the position of the first camera 110 .

,

Expressed as , the following Equation 1 is established.

[Equation 1]

Here, a field of view (FoV) is a value expressing the size of an area that can be photographed by the first camera as an angle, and w represents a length from the leftmost to the right in the FoV.

Equation 1 can be changed to an expression for w as in Equation 2 .

[Equation 2]

를 획득할 수 있다.The processor 130 through Equation 3 based on the user position of the front image

can be obtained.

[Equation 3]

은 전방 이미지의 중심으로부터 이미지 내의 사용자까지의 거리이고,

은 전방 이미지의 가로 길이이다.here,

is the distance from the center of the front image to the user in the image,

is the horizontal length of the front image.

Meanwhile, the processor 130 may identify the user's face from the front image as shown in FIG. 3B and identify the user's face direction based on a plurality of feature points inside the user's face. For example, the processor 130 identifies the user's face from the front image through the model provided by Dlib having 68 feature points, and uses OpenCV to transform the 2D data into 3D to identify the direction of the user's face can do. However, the present invention is not limited thereto, and the processor 130 may identify the user's face and the user's face direction from the front image in any number of ways.

로 표현될 수 있다.The direction of the user's face is based on the line formed by the first camera 110 and the user.

can be expressed as

로 이용할 수 있다. 사용자와 전자 장치(100) 간의 거리인

는 팔꿈치부터 어깨까지의 거리 정도로 사람마다 큰 차이가 없기 때문에 기설정된 값이 이용될 수 있다.On the other hand, the processor 130 is a preset value of Equation 3

is available as The distance between the user and the electronic device 100 is

is the distance from the elbow to the shoulder, and there is no significant difference for each person, so a preset value may be used.

를 획득할 수도 있다. 예를 들어, 프로세서(130)는 도 3c에 도시된 바와 같이, 같은 방향을 촬영하는 두 개의 카메라를 이용하여 두 장의 이미지를 획득하고, 도 3d에 도시된 바와 같이, 삼각형의 대칭을 이용하여 수학식 4와 같이 계산될 수 있다.Alternatively, the processor 130 may be configured in the same manner as in FIGS. 3C and 3D.

can also be obtained. For example, the processor 130 acquires two images using two cameras that shoot in the same direction as shown in FIG. 3C , and performs math using the symmetry of a triangle as shown in FIG. 3D . It can be calculated as Equation 4.

[Equation 4]

이며, f는 초점 거리이고, b는 카메라 간의 거리이며,

는 객체의 위치 변화이다.where Z is the distance to the object

where f is the focal length, b is the distance between cameras,

is the change in the position of the object.

가 획득될 수도 있다.It has been described that two cameras are required to explain [Equation 4], but the present invention is not limited thereto. For example, using a single camera to take two shots at different points in a similar way

may be obtained.

4 is a diagram for explaining a method of identifying a location of an object according to an embodiment of the present disclosure.

를 획득할 수 있다.In FIG. 4 , it is assumed that the user gazes at the upper right corner, and the processor 130 performs an angle of the object with respect to the front direction of the second camera 110 through the process shown in Equation 5 below.

can be obtained.

[Equation 5]

,

,

는 도 3a 내지 도 3d를 통해 획득된 상태이고,

는 도 3c 및 도 3d와 같은 방식으로 획득될 수 있고,

는

,

를 이용한 탄젠트 함수를 통해 획득될 수 있다.here,

,

,

is a state obtained through FIGS. 3A to 3D,

can be obtained in the same way as in FIGS. 3C and 3D,

Is

,

It can be obtained through a tangent function using

는 가로축 성분만이 반영된 값이다. 프로세서(130)는 동일한 방식으로 세로축 성분만이 반영된 제2 카메라(110)의 정면 방향에 대한 객체의 각도를 획득할 수 있다. 이러한 과정을 통해 객체의 위치가 좀더 정확하게 획득될 수 있다.Meanwhile, the angle of the object with respect to the front direction of the second camera 110 obtained in FIGS. 3A to 4 .

is a value in which only the horizontal axis component is reflected. The processor 130 may acquire the angle of the object with respect to the front direction of the second camera 110 in which only the vertical axis component is reflected in the same manner. Through this process, the position of the object can be obtained more accurately.

However, the present invention is not limited thereto, and the processor 130 may identify the position of the object in consideration of only one of the horizontal axis component and the vertical axis component.

5 is a view for explaining a cropping operation according to an embodiment of the present disclosure.

The upper view of FIG. 5 is a view in which a user photographs an object (person) through the electronic device 100 , and the middle view of FIG. 5 illustrates a rear image obtained through the second camera 120 . In this process, the processor 130 may acquire a front image through the first camera 110 and identify the direction of the user's face.

The bottom view of FIG. 5 is a result of cropping a region of the rear image based on the direction of the user's face, and may be finally stored in the memory 150 .

6 is a diagram for explaining a difference between a Field of View (FoV) and a Field of Interest (FoI) according to an embodiment of the present disclosure.

FoV is a first area photographed by the second camera 120 , FoI is a second area to which the user's face is directed, and the first area may include a second area and a shaded area.

When a rear image such as the first region is captured, the processor 130 may crop the rear image to the second region and store the cropped image in the memory 150 .

7A and 7B are diagrams for explaining a change in FoI according to various embodiments of the present disclosure.

The processor 130 may identify the position of the object based on the direction of the user's face, and may crop a region of the rear image based on the position of the object. For example, as shown in FIG. 7A , the processor 130 may identify the position of the airplane based on the direction of the user's face, and may crop the left region of the rear image based on the position of the airplane.

In this case, the processor 130 may identify the object in the rear image based on the position of the object. In particular, when a plurality of objects are identified in the rear image based on the position of the object, the processor 130 may determine one of the plurality of objects in consideration of at least one of sizes and movements of the plurality of objects. For example, when a cloud and an airplane are identified in the rear image based on the position of the object, the processor 130 may determine the airplane as the final object because the movement of the airplane is faster than the cloud. In this case, the processor 130 may compare a plurality of rear images.

The processor 130 may crop a region of the rear image based on the finally determined object.

When the final object is determined, the processor 130 may reduce the number of times the first camera 110 is photographed. For example, the processor 130 may acquire only one front image through the first camera 110 while acquiring five rear images through the second camera 120 . The processor 130 identifies the direction of the user's face from one front image, identifies the position of the object based on the direction of the user's face, and the object identified in the rear image based on the position of the object is the same as the final object When the first camera 110 operates in a reduced state, the number of shots of the first camera 110 may be increased again if the object identified in the rear image is different from the final object.

Alternatively, the processor 130 may crop a region of the rear image in consideration of only the direction of the user's face. For example, as shown in FIG. 7B , the processor 130 crops the left region in the first rear image and crops the right region in the second rear image following the first rear image based on the user's face direction. You may.

Alternatively, when a plurality of objects move in an area corresponding to the user's face direction, the processor 130 may crop one area of the rear image in consideration of only the user's face direction.

8 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

First, when a front image of the electronic device is acquired through the first camera, the user's face direction is identified from the front image ( S810 ). Then, the position of the object in the rear of the electronic device is identified based on the direction of the user's face ( S820 ). Then, a region of the rear image of the electronic device obtained through the second camera is cropped based on the position of the object ( S830 ).

Here, the method further includes acquiring a plurality of rear images through a second camera, and the cropping ( S830 ) crops one region of each of the plurality of rear images based on the position of the object, and the control method includes the cropped image. The method may further include acquiring a moving picture including a plurality of frames acquired based on the plurality of rear image regions.

And, the step of acquiring a plurality of rear images is to acquire a plurality of front images through the first camera while the plurality of rear images are acquired, and the step of identifying the position of the object (S820) is the user's image from the plurality of front images. If the face direction is identified as changed, the changed position of the object is identified based on the changed face direction of the user, and the cropping step (S830) is a different area in each of the plurality of rear images based on the position of the object and the changed position of the object. can be cropped.

Here, the cropping (S830) may identify a position in the image of the object from each of the plurality of rear images based on the position of the object, and crop different regions in each of the plurality of rear images based on the position in the image of the object. have.

Meanwhile, when the position of the object is outside the angle of view of the second camera, the method may further include providing guide information warning that the object is not captured.

And, further comprising the step of acquiring a plurality of front images through the first camera while the plurality of rear images are acquired through the second camera, and the step of cropping (S830) is the direction of the user's face from the plurality of front images When the object to which the user's face direction is changed is changed from the first object to the second object, based on at least one of the first object or the second object based on the time when the user's face direction faces the second object One region of each of the rear images may be cropped.

Here, in the cropping step (S830), if the time for the user's face toward the second object is less than the first threshold time, crop one area of each of the plurality of rear images based on the first object, and the user's face direction is If the time towards the second object is longer than the first threshold time and less than the second threshold time, one area of each of the plurality of rear images is cropped based on the first object and the second object, and the user's face direction is the second object If the facing time is equal to or greater than the second threshold time, one region of each of the plurality of rear images may be cropped based on the second object.

Meanwhile, the method may further include, when a video capturing command is input through the second camera, turning on the first camera to photograph the front of the electronic device to obtain a front image.

In the step of identifying the user's face direction ( S810 ), the user's face may be identified from the front image, and the user's face direction may be identified based on a plurality of feature points inside the user's face.

According to various embodiments of the present disclosure as described above, the electronic device identifies the direction of the user's face, identifies the position of the object based on the direction of the user's face, and selects a region of an image obtained based on the position of the object. By cropping, an image including an object that the user wants to photograph may be acquired.

In addition, since the electronic device automatically identifies the object based on the direction of the user's face, the user can take a picture while looking at the object without the need for the user to look at the shooting screen during shooting, thereby increasing user convenience.

Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer). can The device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, the electronic device A) according to the disclosed embodiments. When the instruction is executed by the processor, the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

In addition, according to an embodiment of the present disclosure, the method according to the various embodiments described above may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

In addition, according to an embodiment of the present disclosure, the various embodiments described above are stored in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. can be implemented in In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing the processing operation of the device according to the above-described various embodiments may be stored in a non-transitory computer-readable medium. When the computer instructions stored in the non-transitory computer-readable medium are executed by the processor of the specific device, the specific device performs the processing operation in the device according to the various embodiments described above. The non-transitory computer-readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specific examples of the non-transitory computer-readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, each of the components (eg, a module or a program) according to the above-described various embodiments may be composed of a single or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be omitted. Components may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, parallel, iteratively, or heuristically, or at least some operations are executed in a different order, are omitted, or other operations are added. can be

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is commonly used in the technical field pertaining to the present disclosure without departing from the gist of the present disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of

100: electronic device 110: first camera
120: second camera 130: processor
140: display 150: memory
160: user interface 170: communication interface
180: speaker

Claims (19)

In an electronic device,
a first camera;
a second camera; and
a processor connected to the first camera and the second camera to control the electronic device;
The processor is
When the front image of the electronic device is obtained through the first camera, the user's face direction is identified from the front image,
Identifies the position of the object in the rear of the electronic device based on the direction of the user's face,
An electronic device that crops a region of the rear image of the electronic device acquired through the second camera based on the position of the object. The method of claim 1,
The processor is
Acquire a plurality of rear images through the second camera,
Cropping a region of each of the plurality of back images based on the position of the object,
An electronic device for acquiring a moving picture including a plurality of frames acquired based on the plurality of cropped rear image regions. 3. The method of claim 2,
The processor is
acquiring a plurality of front images through the first camera while the plurality of rear images are acquired;
When it is identified from the plurality of front images that the direction of the user's face is changed, the changed position of the object is identified based on the changed face direction of the user,
Cropping a different region in each of the plurality of rear images based on the position of the object and the changed position of the object. 4. The method of claim 3,
The processor is
identify a location in the image of the object from each of the plurality of back images based on the location of the object;
and crop a different region in each of the plurality of back images based on a position in the image of the object. According to claim 1,
The processor is
When the position of the object is outside the angle of view of the second camera, the electronic device provides guide information warning that the object is not captured. According to claim 1,
The processor is
acquiring a plurality of front images through the first camera while a plurality of rear images are acquired through the second camera;
When the direction of the user's face is changed from the plurality of front images and the object to which the direction of the user's face is directed is changed from the first object to the second object, the direction of the user's face is changed based on the time toward the second object Cropping one region of each of the plurality of rear images based on at least one of the first object and the second object. 7. The method of claim 6,
The processor is
Cropping a region of each of the plurality of rear images based on the first object when the time for the user's face direction toward the second object is less than a first threshold time,
If the time in which the user's face is directed toward the second object is longer than the first threshold time and less than the second threshold time, a region of each of the plurality of rear images is cropped based on the first object and the second object. and
and cropping a region of each of the plurality of rear images based on the second object when a time in which the user's face is directed toward the second object is equal to or greater than the second threshold time. According to claim 1,
The processor is
When an image capturing command or a video capturing command through the second camera is input, the electronic device is configured to turn on the first camera to photograph the front of the electronic device to obtain the front image. According to claim 1,
The processor is
The electronic device identifies the user's face from the front image and identifies the user's face direction based on a plurality of feature points inside the user's face. A method for controlling an electronic device, comprising:
when a front image of the electronic device is acquired through a first camera, identifying a direction of the user's face from the front image;
identifying the position of the object at the rear of the electronic device based on the direction of the user's face; and
and cropping a region of the rear image of the electronic device acquired through a second camera based on the position of the object. 11. The method of claim 10,
Further comprising; acquiring a plurality of rear images through the second camera,
The cropping step is
Cropping a region of each of the plurality of back images based on the position of the object,
The control method is
Acquiring a moving picture including a plurality of frames acquired based on the plurality of cropped rear image regions; further comprising a control method. 12. The method of claim 11,
Acquiring the plurality of rear images comprises:
acquiring a plurality of front images through the first camera while the plurality of rear images are acquired;
The step of identifying the location of the object,
When it is identified from the plurality of front images that the direction of the user's face is changed, the changed position of the object is identified based on the changed face direction of the user,
The cropping step is
Cropping a different region in each of the plurality of back images based on the position of the object and the changed position of the object. 13. The method of claim 12,
The cropping step is
identify a location in the image of the object from each of the plurality of back images based on the location of the object;
Cropping a different region in each of the plurality of back images based on a position in the image of the object. 11. The method of claim 10,
When the position of the object is outside the angle of view of the second camera, providing guide information warning that the object is not captured; the control method further comprising a. 11. The method of claim 10,
Acquiring a plurality of front images through the first camera while a plurality of rear images are obtained through the second camera; further comprising,
The cropping step is
When the direction of the user's face is changed from the plurality of front images and the object to which the direction of the user's face is directed is changed from the first object to the second object, the direction of the user's face is changed based on the time toward the second object Cropping a region of each of the plurality of rear images based on at least one of the first object and the second object. 16. The method of claim 15,
The cropping step is
Cropping a region of each of the plurality of rear images based on the first object when the time for the user's face direction toward the second object is less than a first threshold time,
If the time in which the user's face is directed toward the second object is longer than the first threshold time and less than the second threshold time, a region of each of the plurality of rear images is cropped based on the first object and the second object. and
When a time in which the user's face direction faces the second object is equal to or greater than the second threshold time, cropping a region of each of the plurality of rear images based on the second object. 11. The method of claim 10,
When a video recording command is input through the second camera, turning on the first camera to photograph the front of the electronic device to obtain the front image. 11. The method of claim 10,
The step of identifying the direction of the user's face,
A control method for identifying the user's face from the front image and identifying the user's face direction based on a plurality of feature points inside the user's face. In a non-transitory computer-readable recording medium storing a program for executing a method of operating an electronic device,
The method of operation is
when a front image of the electronic device is acquired through a first camera, identifying a direction of the user's face from the front image;
identifying the position of the object at the rear of the electronic device based on the direction of the user's face; and
and cropping a region of the rear image of the electronic device acquired through a second camera based on the position of the object.

Images