WO2022149704A1 - Dispositif électronique et procédé de commande associé - Google Patents

Dispositif électronique et procédé de commande associé Download PDF

Info

Publication number
WO2022149704A1
WO2022149704A1 PCT/KR2021/016426 KR2021016426W WO2022149704A1 WO 2022149704 A1 WO2022149704 A1 WO 2022149704A1 KR 2021016426 W KR2021016426 W KR 2021016426W WO 2022149704 A1 WO2022149704 A1 WO 2022149704A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
camera
face
electronic device
images
Prior art date
Application number
PCT/KR2021/016426
Other languages
English (en)
Korean (ko)
Inventor
이철준
박지훈
김윤성
Original Assignee
삼성전자주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 삼성전자주식회사 filed Critical 삼성전자주식회사
Publication of WO2022149704A1 publication Critical patent/WO2022149704A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for acquiring an image by photographing an object and a method for controlling the same.
  • an object of the present disclosure is to provide an electronic device for acquiring an image including an object to be photographed by a user by identifying the location of the object, and a method for controlling the same.
  • an electronic device includes a first camera, a second camera, and a processor connected to the first camera and the second camera to control the electronic device, , when the front image of the electronic device is obtained through the first camera, the processor identifies the user's face direction from the front image, and determines the position of the object in the rear of the electronic device based on the user's face direction and, based on the position of the object, a region of the rear image of the electronic device acquired through the second camera may be cropped.
  • the processor acquires a plurality of rear images through the second camera, crops one region of each of the plurality of rear images based on the position of the object, and based on the cropped plurality of rear image regions A video including a plurality of acquired frames may be acquired.
  • the processor acquires a plurality of front images through the first camera while the plurality of rear images are acquired, and when it is identified from the plurality of front images that the direction of the user's face is changed, the changed user's face
  • the changed position of the object may be identified based on the direction, and a different region may be cropped in each of the plurality of rear images based on the position of the object and the changed position of the object.
  • the processor is further configured to identify a location in the image of the object from each of the plurality of back images based on the location of the object, and to crop a different region in each of the plurality of back images based on the location in the image of the object.
  • the processor may provide guide information warning that the object is not captured.
  • the processor acquires a plurality of front images through the first camera while a plurality of rear images are acquired through the second camera, and the user's face direction is changed from the plurality of front images
  • the plurality of objects based on at least one of the first object or the second object based on a time when the face direction of the user faces the second object
  • One region of each of the rear images of may be cropped.
  • the processor is configured to crop one region of each of the plurality of rear images based on the first object when a time in which the user's face direction faces the second object is less than a first threshold time, and the user's face direction If the time towards the second object is longer than the first threshold time and less than the second threshold time, crop a region of each of the plurality of rear images based on the first object and the second object, When the time for the face direction toward the second object is equal to or greater than the second threshold time, one region of each of the plurality of rear images may be cropped based on the second object.
  • the processor may turn on the first camera to photograph the front of the electronic device to obtain the front image.
  • the processor may identify the user's face from the front image, and identify the user's face direction based on a plurality of feature points inside the user's face.
  • a method of controlling an electronic device includes: when a front image of the electronic device is acquired through a first camera, identifying a direction of the user's face from the front image; identifying the position of the object at the rear of the electronic device based on .
  • the method further includes acquiring a plurality of rear images through the second camera, wherein the cropping includes cropping a region of each of the plurality of rear images based on the position of the object, the control method comprising: The method may further include acquiring a video including a plurality of frames acquired based on the plurality of cropped rear image regions.
  • the acquiring of the plurality of rear images includes acquiring a plurality of front images through the first camera while the plurality of rear images are acquired, and the step of identifying the position of the object is performed from the plurality of front images.
  • the changed position of the object is identified based on the changed face direction of the user, and the cropping includes the plurality of positions based on the position of the object and the changed position of the object. You can crop a different area in each of the back images.
  • the cropping may include identifying a position in the image of the object from each of the plurality of rear images based on the position of the object, and selecting a different region in each of the plurality of rear images based on the position in the image of the object. can be cropped
  • the method may further include providing guide information warning that the object is not captured when the position of the object is outside the angle of view of the second camera.
  • the method further includes acquiring a plurality of front images through the first camera while a plurality of rear images are acquired through the second camera, wherein the cropping includes the user's face from the plurality of front images
  • the cropping includes the user's face from the plurality of front images
  • the cropping includes cropping one region of each of the plurality of rear images based on the first object when the time in which the face direction of the user faces the second object is less than a first threshold time, and When a time in which a face direction faces the second object is longer than the first threshold time and less than a second threshold time, crop a region of each of the plurality of rear images based on the first object and the second object, wherein If the time for the user's face toward the second object is equal to or greater than the second threshold time, one region of each of the plurality of rear images may be cropped based on the second object.
  • the method may further include, when a video capturing command is input through the second camera, turning on the first camera to photograph the front of the electronic device to obtain the front image.
  • the identifying of the user's face direction may include identifying the user's face from the front image, and identifying the user's face direction based on a plurality of feature points inside the user's face.
  • the operating method includes obtaining a front image of the electronic device through a first camera , identifying the user's face direction from the front image, identifying the position of the object at the rear of the electronic device based on the user's face direction, and acquiring through a second camera based on the position of the object and cropping a region of the rear image of the electronic device.
  • the electronic device identifies the direction of the user's face, identifies the position of the object based on the direction of the user's face, and selects a region of an image obtained based on the position of the object. By cropping, an image including an object that the user wants to photograph may be acquired.
  • the electronic device automatically identifies the object based on the direction of the user's face, the user can take a picture while looking at the object without the need for the user to look at the shooting screen during shooting, thereby increasing user convenience.
  • 1A is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.
  • 1B is a block diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure.
  • FIG. 2 is a diagram for explaining a structure of an electronic device according to an embodiment of the present disclosure.
  • 3A to 3D are diagrams for explaining a method of identifying a user according to an embodiment of the present disclosure.
  • FIG. 4 is a diagram for explaining a method of identifying a location of an object according to an embodiment of the present disclosure.
  • FIG. 5 is a view for explaining a cropping operation according to an embodiment of the present disclosure.
  • FIG. 6 is a diagram for explaining a difference between a Field of View (FoV) and a Field of Interest (FoI) according to an embodiment of the present disclosure.
  • FoV Field of View
  • FoI Field of Interest
  • FIG. 7A and 7B are diagrams for explaining a change in FoI according to various embodiments of the present disclosure.
  • FIG. 8 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.
  • expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.
  • the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.
  • a device eg, an artificial intelligence electronic device
  • FIG. 1A is a block diagram illustrating a configuration of an electronic device 100 according to an embodiment of the present disclosure.
  • the present invention is not limited thereto, and the electronic device 100 may be implemented in a state in which some of the components of FIG. 1A are omitted.
  • the electronic device 100 is a device that acquires an image by photographing an object, and may be a device such as a smartphone, a tablet PC, a digital camera, or smart glasses. However, the present invention is not limited thereto, and the electronic device 100 may be any device as long as it is capable of acquiring an image by photographing an object.
  • an electronic device 100 includes a first camera 110 , a second camera 120 , and a processor 130 .
  • the first camera 110 and the second camera 120 are configured to capture a still image or a moving image under the control of the processor 130 .
  • the first camera 110 and the second camera 120 may capture still images at a specific point in time, but may also continuously capture still images.
  • the first camera 110 and the second camera 120 may capture an image of an area around the electronic device 100 .
  • the first camera 110 and the second camera 120 may capture images around the electronic device 100 with an angle of view of 71 degrees.
  • the present invention is not limited thereto, and the first camera 110 and the second camera 120 may capture images around the electronic device 100 at different angles of view.
  • the first camera 110 may be arranged to photograph the front of the electronic device 100
  • the second camera 120 may be arranged to photograph the rear of the electronic device 100
  • the electronic device 100 is implemented as a bar type
  • the first camera 110 is disposed on the front of the electronic device 100
  • the second camera 120 is disposed on the rear of the electronic device 100 .
  • the first camera 110 may be disposed on the front side of the electronic device 100 together with a display to be described later.
  • Each of the first camera 110 and the second camera 120 may include a lens, a shutter, an aperture, a solid-state imaging device, an analog front end (AFE), and a timing generator (TG).
  • the shutter controls the time at which the light reflected on the subject enters each of the first camera 110 and the second camera 120, and the iris mechanically increases or decreases the size of the opening through which light enters, thereby reducing the amount of light incident on the lens. can be adjusted
  • the solid-state imaging device may output an image of the photocharge as an electrical signal when light reflected by the subject is accumulated as photocharges.
  • the TG outputs a timing signal for reading out pixel data of the solid-state imaging device, and the AFE may sample and digitize the electrical signal output from the solid-state imaging device.
  • the processor 130 controls the overall operation of the electronic device 100 .
  • the processor 130 may be connected to each component of the electronic device 100 to control the overall operation of the electronic device 100 .
  • the processor 130 may be connected to the first camera 110 and the second camera 120 to control the operation of the electronic device 100 .
  • the processor 130 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON).
  • DSP digital signal processor
  • TCON time controller
  • the present invention is not limited thereto, and the central processing unit (central processing unit (CPU)), micro controller unit (MPU), micro processing unit (MPU), controller, application processor (AP), or communication processor (CP), ARM processor
  • the processor 130 may be implemented as a SoC (System on Chip) or LSI (large scale integration) in which a processing algorithm is embedded, or an FPGA ( Field programmable gate array) may be implemented.
  • SoC System on Chip
  • LSI large scale integration
  • FPGA Field programmable gate array
  • the processor 130 may identify the direction of the user's face from the front image. For example, when an image capturing command or a video capturing command is input through the second camera 120 , the processor 130 turns on the first camera 110 to photograph the front of the electronic device 110 to obtain a front image. can be obtained.
  • the processor 130 may identify the user's face from the front image and identify the user's face direction based on a plurality of feature points inside the user's face. For example, the processor 130 may identify how much the user's face is rotated with respect to the electronic device 100 based on distances between a plurality of feature points inside the user's face.
  • the present invention is not limited thereto, and the processor 130 may identify the user's gaze direction based on the user's eyes.
  • the user's face direction is used for convenience of explanation, but the user's gaze direction may also be used.
  • the processor 130 identifies the position of the object at the rear of the electronic device 100 based on the direction of the user's face, and when the rear image of the electronic device 100 is acquired through the second camera 120 , the position of the object A region of the rear image may be cropped based on .
  • the processor 130 may crop a region of the rear image based on the position of the object in various ways. For example, the processor 130 identifies that the object is located on the right side at the rear of the electronic device 100 based on the direction of the user's face, and determines the right region of the rear image acquired through the second camera 120 . can be cropped Alternatively, the processor 130 divides the image into four regions of the upper left, upper right, lower left, and lower right of the image in a matrix form, identifies an object in one of the four regions based on the direction of the user's face, and the position of the object A region of the rear image may be cropped based on . In this case, the processor 130 may identify the object in one of the four areas using the neural network model.
  • the processor 130 obtains the angle of the object with respect to the front direction of the second camera 120, identifies the position of the object in a partial region of the rear image corresponding to the angle, and A region of the back image may be cropped based on the position.
  • the processor 130 may crop an area of a preset size from the rear image based on the position of the object.
  • the present invention is not limited thereto, and the processor 130 may identify the size of the object in the rear image and determine the size of the area to be cropped based on the identified size of the object.
  • the processor 130 acquires a plurality of rear images through the second camera 120 , crops one region of each of the plurality of rear images based on the position of the object, and obtains based on the plurality of cropped rear image regions
  • a video including a plurality of frames may be acquired. That is, the processor 130 may capture a video and crop each image frame based on the position of the object.
  • the processor 130 acquires a plurality of front images through the first camera 110 while a plurality of rear images are acquired, and when it is identified that the direction of the user's face is changed from the plurality of front images, the changed user's face A changed position of the object is identified based on the direction, and a different region may be cropped in each of the plurality of back images based on the position of the object and the changed position of the object.
  • the processor 130 may acquire a first front image while a first rear image is acquired, and acquire a second rear image while a second rear image following the first rear image is acquired. Then, if the user's face direction of the first front image is changed from the second front image, the processor 130 crops the first rear image based on the position of the object before the user's face direction is changed, and the user's face The second back image may be cropped based on the position of the object after the direction is changed.
  • the processor 130 may identify a position in the image of the object from each of the plurality of rear images based on the position of the object, and crop a different region in each of the plurality of rear images based on the position in the image of the object. That is, the processor 130 may identify the object from the rear image based on the position of the object, and crop one region of each of the rear images by further considering the identified position in the image of the object. In this case, the accuracy of cropping may be improved compared to the case of simply cropping using the position of the object.
  • the processor 130 may provide guide information warning that the object is not captured. For example, when the object is positioned outside the angle of view of the second camera 120 as the object moves, the processor 130 may display guide information warning that the object is not captured or output a warning sound as a sound. can Alternatively, the processor 130 may provide a warning through the blinking of an LED.
  • the processor 130 acquires a plurality of front images through the first camera 110 while a plurality of rear images are acquired through the second camera 120 , and the direction of the user's face is changed from the plurality of front images
  • the plurality of rearwards based on at least one of the first object or the second object based on the time when the user's face direction faces the second object
  • One region of each image may be cropped.
  • the processor 130 crops one region of each of the plurality of rear images based on the first object when the time for the user's face direction toward the second object is less than the first threshold time, and the user's face direction is If the time towards the second object is longer than the first threshold time and less than the second threshold time, one area of each of the plurality of rear images is cropped based on the first object and the second object, and the user's face direction is the second object If the facing time is equal to or greater than the second threshold time, one region of each of the plurality of rear images may be cropped based on the second object.
  • the processor 130 identifies the object that the user wants to photograph as not changed, and You can crop the back image based on the original object. Then, the processor 130 crops the rear image based on the first object and the second object when the time for the user's face toward the second object is longer than the first threshold time and less than the second threshold time, so that the cropped image is abrupt. can be prevented from being changed. For example, while cropping the plurality of rear images, the processor 130 may crop the area including the first object in a stepwise manner, and if the second object is included, the area near the first object may be cropped in a stepwise manner.
  • the processor 130 identifies that the user's object of interest has been changed only when a threshold time has elapsed for the time the user gazes at the second object (when the user's face direction is equal to or greater than the second threshold time) can do.
  • the present invention is not limited thereto, and the processor 130 crops one region of each of the plurality of rear images based on the first object when the time for the user's face toward the second object is less than the first threshold time, and the user If the time in which the face direction of ' is directed toward the second object is equal to or greater than the first threshold time, one region of each of the plurality of rear images may be cropped based on the second object.
  • the processor 130 may identify the position of the object based on the direction of the user's face, and identify the object from the rear image.
  • the processor 130 may perform this operation for each of the plurality of rear images, and when the shape of an object identified from each of the plurality of rear images is changed, the above operation may be performed.
  • FIG. 1B is a block diagram illustrating a detailed configuration of an electronic device 100 according to an embodiment of the present disclosure.
  • the electronic device 100 may include a first camera 110 , a second camera 120 , and a processor 130 . Also, according to FIG. 1B , the electronic device 100 may further include at least one of a display 140 , a memory 150 , a user interface 160 , a communication interface 170 , and a speaker 180 .
  • FIG. 1B detailed descriptions of parts overlapping with those shown in FIG. 1A will be omitted.
  • the display 140 may be implemented with various types of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), and the like.
  • the display 140 may also include a driving circuit, a backlight unit, and the like, which may be implemented in a form such as an a-si TFT, a low temperature poly silicon (LTPS) TFT, or an organic TFT (OTFT).
  • the display 140 may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display, or the like.
  • the display 140 may be disposed in front of the electronic device 100 .
  • the present invention is not limited thereto, and the display 140 may have any shape as long as it is disposed on the same surface as the first camera 110 .
  • the display 140 may display the rear image captured by the second camera 120 in real time under the control of the processor 130 . Also, the display 140 may provide guide information for warning that the object is not photographed under the control of the processor 130 .
  • the memory 150 may refer to hardware that stores information such as data in an electrical or magnetic form so that the processor 130 can access it. To this end, the memory 150 may be implemented as hardware at least one of non-volatile memory, volatile memory, flash memory, hard disk drive (HDD) or solid state drive (SSD), RAM, ROM, etc. .
  • At least one instruction or module required for the operation of the electronic device 100 or the processor 130 may be stored in the memory 150 .
  • the instruction is a unit of code for instructing the operation of the electronic device 100 or the processor 130 and may be written in machine language, which is a language that a computer can understand.
  • a module may be a set of instructions that perform a specific task of a unit of work.
  • the memory 150 may store data that is information in units of bits or bytes that can represent characters, numbers, images, and the like.
  • the memory 150 may store a face identification module, a face direction identification module, an object location identification module, an object identification module, a crop module, and the like.
  • at least one of the face identification module, the face direction identification module, the object position identification module, the object identification module, and the crop module may be implemented as a neural network model.
  • the memory 150 is accessed by the processor 130 , and reading/writing/modification/deletion/update of instructions, modules, or data may be performed by the processor 130 .
  • the user interface 160 may be implemented as a button, a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen capable of performing the above-described display function and manipulation input function together.
  • the button may be various types of buttons such as a mechanical button, a touch pad, a wheel, etc. formed in an arbitrary area such as the front, side, or rear of the exterior of the main body of the electronic device 100 .
  • the communication interface 170 is configured to communicate with various types of external devices according to various types of communication methods.
  • the electronic device 100 may receive various modules from an external device through the communication interface 170 .
  • the communication interface 170 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module.
  • each communication module may be implemented in the form of at least one hardware chip.
  • the Wi-Fi module and the Bluetooth module perform communication using a WiFi method and a Bluetooth method, respectively.
  • various types of connection information such as an SSID and a session key are first transmitted and received, and various types of information can be transmitted/received after a communication connection is made using this.
  • the infrared communication module communicates according to the infrared data association (IrDA) technology, which wirelessly transmits data in a short distance using infrared that is between visible light and millimeter waves.
  • IrDA infrared data association
  • the wireless communication module includes Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE Advanced (LTE-A), 4th Generation (4G), 5G It may include at least one communication chip that performs communication according to various wireless communication standards, such as (5th Generation).
  • 3G 3rd Generation
  • 3GPP 3rd Generation Partnership Project
  • LTE Long Term Evolution
  • LTE-A LTE Advanced
  • 4G 4th Generation
  • 5G 5G
  • It may include at least one communication chip that performs communication according to various wireless communication standards, such as (5th Generation).
  • the communication interface 170 may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, or the like.
  • the communication interface 170 may include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module for performing communication using a pair cable, a coaxial cable, or an optical fiber cable.
  • LAN local area network
  • Ethernet Ethernet
  • wired communication module for performing communication using a pair cable, a coaxial cable, or an optical fiber cable.
  • the speaker 180 may receive an electrical signal, and may radiate sound by vibrating a diaphragm based on the input electrical signal.
  • the speaker 180 may output a sound warning that the object is not photographed under the control of the processor 130 .
  • the electronic device 100 identifies the position of the object based on the direction of the user's face, and crops a region of the image obtained based on the identified position of the object, so that the user can keep an eye on the shooting screen while shooting. You can acquire an image containing an object without any need.
  • FIGS. 2 to 7 the operation of the electronic device 100 will be described in more detail with reference to FIGS. 2 to 7 .
  • 2 to 7 individual embodiments will be described for convenience of description. However, the individual embodiments of FIGS. 2 to 7 may be embodied in any combination.
  • FIG. 2 is a view for explaining the structure of the electronic device 100 according to an embodiment of the present disclosure.
  • the electronic device 100 may include the first camera 110 on the front side and the second camera 120 on the rear side. Also, the electronic device 100 may include the first camera 110 and the display 140 on the front side and the second camera 120 on the rear side.
  • the electronic device 100 may be embodied in any number of forms, and the photographing directions of the first camera 110 and the second camera 120 are opposite to each other, and the first camera 110 and the second camera ( 120) may be used in any form as long as it is a device capable of simultaneously driving.
  • 3A to 3D are diagrams for explaining a method of identifying a user according to an embodiment of the present disclosure.
  • the processor 130 may identify the location of the user from the front image. First, as shown in FIG. 3A , the user's position is determined with respect to the position of the first camera 110 . , Expressed as , the following Equation 1 is established.
  • a field of view is a value expressing the size of an area that can be photographed by the first camera as an angle, and w represents a length from the leftmost to the right in the FoV.
  • Equation 1 can be changed to an expression for w as in Equation 2 .
  • the processor 130 through Equation 3 based on the user position of the front image can be obtained.
  • the processor 130 may identify the user's face from the front image as shown in FIG. 3B and identify the user's face direction based on a plurality of feature points inside the user's face. For example, the processor 130 identifies the user's face from the front image through the model provided by Dlib having 68 feature points, and uses OpenCV to transform the 2D data into 3D to identify the direction of the user's face can do.
  • the present invention is not limited thereto, and the processor 130 may identify the user's face and the user's face direction from the front image in any number of ways.
  • the direction of the user's face is based on the line formed by the first camera 110 and the user. can be expressed as
  • the processor 130 sets the preset value in Equation 3 is available as The distance between the user and the electronic device 100 is is the distance from the elbow to the shoulder, and there is no significant difference for each person, so a preset value may be used.
  • the processor 130 may be configured in the same manner as in FIGS. 3C and 3D. can also be obtained. For example, the processor 130 acquires two images using two cameras that shoot in the same direction as shown in FIG. 3C , and performs math using the symmetry of a triangle as shown in FIG. 3D . It can be calculated as Equation 4.
  • FIG. 4 is a diagram for explaining a method of identifying a location of an object according to an embodiment of the present disclosure.
  • FIGS. 3A to 3D is a state obtained through FIGS. 3A to 3D, can be obtained in the same way as in FIGS. 3C and 3D, Is , It can be obtained through a tangent function using
  • the angle of the object with respect to the front direction of the second camera 110 obtained in FIGS. 3A to 4 . is a value in which only the horizontal axis component is reflected.
  • the processor 130 may acquire the angle of the object with respect to the front direction of the second camera 110 in which only the vertical axis component is reflected in the same manner. Through this process, the position of the object can be obtained more accurately.
  • the present invention is not limited thereto, and the processor 130 may identify the position of the object in consideration of only one of the horizontal axis component and the vertical axis component.
  • FIG. 5 is a view for explaining a cropping operation according to an embodiment of the present disclosure.
  • the upper view of FIG. 5 is a view in which a user photographs an object (person) through the electronic device 100 , and the middle view of FIG. 5 illustrates a rear image obtained through the second camera 120 .
  • the processor 130 may acquire a front image through the first camera 110 and identify the direction of the user's face.
  • the bottom view of FIG. 5 is a result of cropping a region of the rear image based on the direction of the user's face, and may be finally stored in the memory 150 .
  • FIG. 6 is a diagram for explaining a difference between a Field of View (FoV) and a Field of Interest (FoI) according to an embodiment of the present disclosure.
  • FoV Field of View
  • FoI Field of Interest
  • FoV is a first area photographed by the second camera 120
  • FoI is a second area to which the user's face is directed
  • the first area may include a second area and a shaded area.
  • the processor 130 may crop the rear image to the second region and store the cropped image in the memory 150 .
  • FIG. 7A and 7B are diagrams for explaining a change in FoI according to various embodiments of the present disclosure.
  • the processor 130 may identify the position of the object based on the direction of the user's face, and may crop a region of the rear image based on the position of the object. For example, as shown in FIG. 7A , the processor 130 may identify the position of the airplane based on the direction of the user's face, and may crop the left region of the rear image based on the position of the airplane.
  • the processor 130 may identify the object in the rear image based on the position of the object. In particular, when a plurality of objects are identified in the rear image based on the position of the object, the processor 130 may determine one of the plurality of objects in consideration of at least one of sizes and movements of the plurality of objects. For example, when a cloud and an airplane are identified in the rear image based on the position of the object, the processor 130 may determine the airplane as the final object because the movement of the airplane is faster than the cloud. In this case, the processor 130 may compare a plurality of rear images.
  • the processor 130 may crop a region of the rear image based on the finally determined object.
  • the processor 130 may reduce the number of times the first camera 110 is photographed. For example, the processor 130 may acquire only one front image through the first camera 110 while acquiring five rear images through the second camera 120 . The processor 130 identifies the direction of the user's face from one front image, identifies the position of the object based on the direction of the user's face, and the object identified in the rear image based on the position of the object is the same as the final object.
  • the first camera 110 operates in a reduced number of times, if the object identified in the rear image is different from the final object, the number of times of photographing by the first camera 110 may be increased again.
  • the processor 130 may crop a region of the rear image in consideration of only the direction of the user's face. For example, as shown in FIG. 7B , the processor 130 crops the left region in the first rear image and crops the right region in the second rear image following the first rear image based on the user's face direction. You may.
  • the processor 130 may crop one area of the rear image in consideration of only the user's face direction.
  • FIG. 8 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.
  • the user's face direction is identified from the front image ( S810 ). Then, the position of the object in the rear of the electronic device is identified based on the direction of the user's face ( S820 ). Then, a region of the rear image of the electronic device obtained through the second camera is cropped based on the position of the object ( S830 ).
  • the method further includes acquiring a plurality of rear images through a second camera, and the cropping ( S830 ) crops one region of each of the plurality of rear images based on the position of the object, and the control method includes the cropped image.
  • the method may further include acquiring a moving picture including a plurality of frames acquired based on the plurality of rear image regions.
  • the step of acquiring a plurality of rear images is to acquire a plurality of front images through the first camera while the plurality of rear images are acquired
  • the step of identifying the position of the object (S820) is the user's image from the plurality of front images. If the face direction is identified as changed, the changed position of the object is identified based on the changed face direction of the user, and the cropping step (S830) is a different area in each of the plurality of rear images based on the position of the object and the changed position of the object. can be cropped.
  • the cropping (S830) may identify a position in the image of the object from each of the plurality of rear images based on the position of the object, and crop different regions in each of the plurality of rear images based on the position in the image of the object. have.
  • the method may further include providing guide information warning that the object is not captured.
  • the step of cropping is the direction of the user's face from the plurality of front images
  • the object to which the user's face direction is changed is changed from the first object to the second object, based on at least one of the first object or the second object based on the time when the user's face direction faces the second object
  • One region of each of the rear images may be cropped.
  • the cropping step (S830) if the time for the user's face toward the second object is less than the first threshold time, crop one area of each of the plurality of rear images based on the first object, and the user's face direction is If the time towards the second object is longer than the first threshold time and less than the second threshold time, one area of each of the plurality of rear images is cropped based on the first object and the second object, and the user's face direction is the second object If the facing time is equal to or greater than the second threshold time, one region of each of the plurality of rear images may be cropped based on the second object.
  • the method may further include, when a video capturing command is input through the second camera, turning on the first camera to photograph the front of the electronic device to obtain a front image.
  • the user's face may be identified from the front image, and the user's face direction may be identified based on a plurality of feature points inside the user's face.
  • the electronic device identifies the direction of the user's face, identifies the position of the object based on the direction of the user's face, and selects a region of an image obtained based on the position of the object. By cropping, an image including an object that the user wants to photograph may be acquired.
  • the electronic device automatically identifies the object based on the direction of the user's face, the user can take a picture while looking at the object without the need for the user to look at the shooting screen during shooting, thereby increasing user convenience.
  • the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer).
  • the device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, the electronic device A) according to the disclosed embodiments.
  • the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor.
  • Instructions may include code generated or executed by a compiler or interpreter.
  • the device-readable storage medium may be provided in the form of a non-transitory storage medium.
  • 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.
  • the method according to the various embodiments described above may be included in a computer program product and provided.
  • Computer program products may be traded between sellers and buyers as commodities.
  • the computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play StoreTM).
  • an application store eg, Play StoreTM
  • at least a part of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.
  • the various embodiments described above are stored in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. can be implemented in In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.
  • non-transitory computer-readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, a cache, a memory, and the like, and can be read by a device.
  • Specific examples of the non-transitory computer-readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.
  • each of the components may be composed of a singular or a plurality of entities, and some sub-components of the above-described corresponding sub-components may be omitted, or other sub-components may be omitted. Components may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, parallel, iteratively, or heuristically, or at least some operations are executed in a different order, are omitted, or other operations are added. can be

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Studio Devices (AREA)
  • Image Analysis (AREA)

Abstract

Un dispositif électronique est divulgué. Le dispositif électronique comprend une première caméra, une seconde caméra et un processeur connecté à la première caméra et à la seconde caméra de façon à commander le dispositif électronique, lorsqu'une image avant du dispositif électronique est obtenue par l'intermédiaire de la première caméra, le processeur pouvant identifier la direction du visage d'un utilisateur à partir de l'image avant, identifier la position d'un objet à l'arrière du dispositif électronique sur la base de la direction du visage de l'utilisateur et rogner une région d'une image arrière du dispositif électronique acquise par l'intermédiaire de la seconde caméra sur la base de la position de l'objet.
PCT/KR2021/016426 2021-01-07 2021-11-11 Dispositif électronique et procédé de commande associé WO2022149704A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2021-0002168 2021-01-07
KR1020210002168A KR20220099827A (ko) 2021-01-07 2021-01-07 전자 장치 및 그 제어 방법

Publications (1)

Publication Number Publication Date
WO2022149704A1 true WO2022149704A1 (fr) 2022-07-14

Family

ID=82358019

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/016426 WO2022149704A1 (fr) 2021-01-07 2021-11-11 Dispositif électronique et procédé de commande associé

Country Status (2)

Country Link
KR (1) KR20220099827A (fr)
WO (1) WO2022149704A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015126451A (ja) * 2013-12-27 2015-07-06 レノボ・シンガポール・プライベート・リミテッド 画像の記録方法、電子機器およびコンピュータ・プログラム
KR20170008064A (ko) * 2015-07-13 2017-01-23 연세대학교 산학협력단 객체 인식을 이용하는 객체 촬영 방법 및 시스템
KR20180002534A (ko) * 2016-06-29 2018-01-08 포브, 아이엔씨. 외부 촬상 시스템, 외부 촬상 방법, 외부 촬상 프로그램
KR20190080243A (ko) * 2017-12-28 2019-07-08 엘에스산전 주식회사 증강 현실 제공 방법
KR102119659B1 (ko) * 2013-09-23 2020-06-08 엘지전자 주식회사 영상표시장치 및 그것의 제어 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102119659B1 (ko) * 2013-09-23 2020-06-08 엘지전자 주식회사 영상표시장치 및 그것의 제어 방법
JP2015126451A (ja) * 2013-12-27 2015-07-06 レノボ・シンガポール・プライベート・リミテッド 画像の記録方法、電子機器およびコンピュータ・プログラム
KR20170008064A (ko) * 2015-07-13 2017-01-23 연세대학교 산학협력단 객체 인식을 이용하는 객체 촬영 방법 및 시스템
KR20180002534A (ko) * 2016-06-29 2018-01-08 포브, 아이엔씨. 외부 촬상 시스템, 외부 촬상 방법, 외부 촬상 프로그램
KR20190080243A (ko) * 2017-12-28 2019-07-08 엘에스산전 주식회사 증강 현실 제공 방법

Also Published As

Publication number Publication date
KR20220099827A (ko) 2022-07-14

Similar Documents

Publication Publication Date Title
WO2017014415A1 (fr) Appareil de capture d'image et son procédé de fonctionnement
EP3642802A1 (fr) Appareil d'édition d'image utilisant une carte de profondeur et son procédé
WO2020036343A1 (fr) Dispositif électronique et procédé de commande correspondant
WO2019039771A1 (fr) Dispositif électronique pour mémoriser des informations de profondeur en relation avec une image en fonction des propriétés d'informations de profondeur obtenues à l'aide d'une image, et son procédé de commande
WO2018236058A1 (fr) Dispositif électronique pour fournir des informations de propriété d'une source de lumière externe pour un objet d'intérêt
WO2019059562A1 (fr) Dispositif électronique comprenant une pluralité de caméras utilisant un mode d'obturateur roulant
WO2017090848A1 (fr) Dispositif de photographie et son procédé d'utilisation
WO2021133025A1 (fr) Dispositif électronique comprenant un capteur d'image et son procédé de fonctionnement
WO2019039870A1 (fr) Dispositif électronique capable de commander un effet d'affichage d'image, et procédé d'affichage d'image
WO2021158057A1 (fr) Dispositif électronique et procédé d'affichage d'image sur le dispositif électronique
WO2019139404A1 (fr) Dispositif électronique et procédé de traitement d'image correspondante
WO2019017585A1 (fr) Dispositif électronique de commande de la mise au point d'une lentille et procédé de commande associé
WO2019045517A1 (fr) Procédé de commande de synchronisation d'une pluralité de capteurs d'image et dispositif électronique destiné à sa mise en œuvre
WO2021221341A1 (fr) Dispositif de réalité augmentée et son procédé de commande
WO2014073939A1 (fr) Procédé et appareil de capture et d'affichage d'image
WO2019160237A1 (fr) Dispositif électronique, et procédé de commande d'affichage d'images
WO2022149704A1 (fr) Dispositif électronique et procédé de commande associé
WO2016208833A1 (fr) Appareil de viseur et son procédé de fonctionnement
WO2019190250A1 (fr) Procédé de synthèse d'image sur un objet réfléchissant en fonction d'un attribut d'objet réfléchissant inclus dans une image différente et dispositif électronique
WO2021162396A1 (fr) Procédé de génération d'éléments de contenu et dispositif électronique associé
WO2017034321A1 (fr) Technique de prise en charge de photographie dans un dispositif possédant un appareil photo et dispositif à cet effet
WO2020085718A1 (fr) Procédé et dispositif de génération d'avatar sur la base d'une image corrigée
WO2020122513A1 (fr) Procédé de traitement d'image bidimensionnelle et dispositif d'exécution dudit procédé
WO2024048914A1 (fr) Dispositif d'affichage pour acquérir une ressource holographique et procédé de commande associé
WO2024025089A1 (fr) Dispositif d'affichage pour afficher un objet ra et son procédé de commande

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21917866

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21917866

Country of ref document: EP

Kind code of ref document: A1