WO2022055174A1

WO2022055174A1 - Electronic apparatus and control method thereof

Info

Publication number: WO2022055174A1
Application number: PCT/KR2021/011830
Authority: WO
Inventors: 이효재; 김지은; 유재영
Original assignee: 삼성전자(주)
Priority date: 2020-09-08
Filing date: 2021-09-02
Publication date: 2022-03-17
Also published as: KR20220032867A

Abstract

Disclosed are an electronic apparatus and a control method thereof. The electronic apparatus comprises: an interface unit; a display; and a processor for identifying user-related information of a first image acquired by a first camera, identifying a user area in a second image acquired by a second camera on the basis of the identified user-related information, in response to switching from the first camera to another second camera, and transmitting, through the interface unit, a call video corresponding to the user area in the identified second image to a counterpart device of a video call.

Description

Electronic device and its control method

The present disclosure relates to an electronic device that provides a video call function and a control method therefor.

Recently, various services using a video call function, such as video conferencing, are becoming more popular at home or at work. As an example of various services, the existing video call may be maintained by switching from a device using a video call function to a device using another camera. In this case, depending on the location of the camera or the user, an image of a completely different frame may be transmitted to the other party when the camera is switched, and an unwanted background or exposure of a person may occur differently from the user's intention. This may cause problems such as invasion of users' privacy. In addition, since video calls are transmitted to unnecessary objects to the other party, call concentration may be hindered, which may be a factor in reducing work efficiency in video conferencing and the like.

In the case of providing a video call function, an electronic device that provides a better service and a control method thereof are provided.

According to an aspect of the present disclosure, an electronic device includes: an interface unit; display; and identifying user-related information of the first image acquired by the first camera, and in response to switching from the first camera to another second camera, by the second camera based on the identified user-related information and a processor for identifying a user area in the obtained second image, and transmitting a call image corresponding to the user area in the identified second image to a counterpart device of the video call through the interface unit.

The user-related information includes information about at least one of a location or a size of a user area in the first image, and the processor is configured to: Based on the location or size of the user area in the first image, the user in the second image area can be identified.

The user-related information includes information about an angle of view of the first camera, and the processor is configured to identify a user area in the second image based on a relationship between the angle of view of the first camera and the angle of view of the second camera. can

The processor may receive the user-related information from an external device including the first camera through the interface unit.

The processor may transmit the first image to the counterpart device while the user area in the second image is identified in response to switching from the first camera to the second camera, or transmit the first image by replacing it with another image .

The electronic device may further include the second camera.

The electronic device may further include a user input unit, and the processor may identify a user area in the second image based on a user input received through the user input unit.

The user input may include a user voice input.

The electronic device further includes a user input unit, and the processor adjusts a user area in the second image based on a user input received through the user input unit, and displays a call image corresponding to the adjusted user area. can be transmitted to the counterpart device.

The processor may display a user interface (UI) indicating a user area in the second image, and adjust the user area in the second image based on the user input received using the UI.

The processor may adjust a user area in the second image based on a user input received through an external device, and transmit a call image corresponding to the adjusted user area to the counterpart device.

The processor may transmit information about the user area in the second image to the external device through the interface unit so that the external device adjusts the user area in the second image.

According to another aspect of the present disclosure, a method of controlling an electronic device includes: identifying user-related information of a first image acquired by a first camera; identifying a user area in a second image acquired by the second camera based on the identified user-related information in response to switching from the first camera to another second camera; and transmitting the call image corresponding to the user area in the identified second image to the other party device of the video call through the interface unit.

The user-related information includes information about at least one of a location and a size of the user area in the first image, and the identifying of the user area in the second image includes the location or size of the user area in the first image. It may include identifying a user area in the second image based on the

The user-related information includes information about an angle of view of the first camera, and the identifying a user area in the second image includes: based on a relationship between the angle of view of the first camera and the angle of view of the second camera It may include the step of identifying the user area in the second image.

The method of controlling the electronic device may include receiving the user-related information from an external device including the first camera through the interface unit.

The method for controlling the electronic device includes transmitting the first image to the counterpart device or replacing it with another image while the user area in the second image is identified in response to the switching from the first camera to the second camera It may include the step of transmitting.

The identifying of the user region in the second image may include identifying the user region in the second image based on a user input received through the user input unit.

The method of controlling the electronic device may include: adjusting a user area in the second image based on a user input received through an external device; and transmitting a call image corresponding to the adjusted user area to the counterpart device.

According to another aspect of the present disclosure, in a recording medium storing a computer program including a code for performing a control method of an electronic device as a computer readable code, the control method of the electronic device includes: identifying user-related information of the first image obtained by the method; identifying a user area in a second image acquired by the second camera based on the identified user-related information in response to switching from the first camera to another second camera; and transmitting the call image corresponding to the user area in the identified second image to the other party device of the video call through the interface unit.

According to the present disclosure, it is possible to provide a natural camera switching service by minimizing an image difference that occurs according to a camera switching during a video call. In addition, it is possible to contribute to privacy protection by preventing exposure of a background or a person not desired by the user in the image by switching the camera. Through this, it is possible to increase the concentration of the call by transmitting the object required in the video call to the other party, which can be a factor to increase the work efficiency in the video conference.

The above and other aspects, features and advantages of the embodiments of the present disclosure will become more apparent from the following description with reference to the accompanying drawings.

1 is a diagram illustrating an image difference according to a camera according to an exemplary embodiment.

2 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

3 is a diagram illustrating an operation flowchart of an electronic device according to an exemplary embodiment.

4 is a diagram illustrating an operation of an electronic device according to an exemplary embodiment.

5 is a diagram illustrating an operation of an electronic device according to an exemplary embodiment.

6 is a diagram illustrating an operation of an electronic device according to an exemplary embodiment.

7 is a diagram illustrating an operation of an electronic device according to an exemplary embodiment.

8 is a diagram illustrating an operation of an electronic device according to an exemplary embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numbers or symbols refer to components that perform substantially the same functions, and the size of each component in the drawings may be exaggerated for clarity and convenience of description. However, the technical spirit of the present disclosure and its core configuration and operation are not limited to the configuration or operation described in the following embodiments. In describing the present disclosure, if it is determined that a detailed description of a known technology or configuration related to the present disclosure may unnecessarily obscure the subject matter of the present disclosure, the detailed description thereof will be omitted.

In an embodiment of the present disclosure, terms including an ordinal number such as first, second, etc. are used only for the purpose of distinguishing one element from another element, and the expression of the singular is plural, unless the context clearly dictates otherwise. includes the expression of In addition, in the embodiments of the present disclosure, terms such as 'consisting', 'comprising', 'having' and the like are one or more other features or numbers, steps, operations, components, parts, or the presence of a combination thereof. Or it should be understood that the possibility of addition is not precluded in advance. In addition, in an embodiment of the present disclosure, a 'module' or 'unit' performs at least one function or operation, may be implemented as hardware or software, or may be implemented as a combination of hardware and software, and may be integrated into at least one module and can be implemented. Also, in an embodiment of the present disclosure, at least one of the plurality of elements refers to all of the plurality of elements, as well as each one or a combination thereof excluding the rest of the plurality of elements.

1 is a diagram illustrating an image according to an embodiment.

1 shows an image A(1) and an image B(2). Image A(1) and image B(2) are captured by different cameras in the same situation, for example, image A(1) is captured by a camera C1 built into a mobile phone, which is a mobile electronic device , it is assumed that the image B(2) was captured by the camera C2 mounted on the TV, which is an electronic device having a fixed position.

In the case of image A(1), the distance between the mobile phone and the user is close, and the camera C1 built into the mobile phone has a relatively narrow angle of view compared to the camera C2 mounted on the TV, so that only the user is displayed. In the case of image B(2), the distance between the TV and the user is greater than that of the mobile phone, and the user is relatively out of the center of the angle of view of the camera C2. In addition, the angle of view of the camera C2 mounted on the TV is wider than the angle of view of the mobile phone camera C1, so that other people including the user are displayed together.

In this way, the image may be displayed differently depending on the angle of view of the imaging camera, the distance between the camera and the object to be imaged, the position of the camera, and the like. Hereinafter, when the screen changes, such as when the camera is switched from the camera (C1) built into the mobile phone to the camera (C2) mounted on the TV, a scene not intended by the user is displayed, etc. Describe techniques to avoid possible problems.

According to an embodiment, the electronic device 100 may be implemented as a display device including a display 121 capable of displaying images, such as the image A(1) and the image B(2) shown in FIG. 1 . . As an example, the electronic device 100 may include a TV, a computer, a smart phone, a tablet, a portable media player, a wearable device, a video wall, an electronic picture frame, and the like. However, the spirit of the present disclosure is not limited to the example implementation of the electronic device 100, and as another embodiment, the electronic device 100 transmits a signal to an external display connected by wire or wirelessly, such as an image such as a set-top box. It may be implemented as a processing device.

Hereinafter, configurations of an electronic device according to an embodiment will be described with reference to the drawings.

As shown in FIG. 2 , the electronic device 100 may include an interface unit 110 .

The interface unit 110 may include a wired interface unit 111 . The wired interface unit 111 includes a connector or port to which an antenna capable of receiving a broadcast signal according to a broadcasting standard such as terrestrial/satellite broadcasting can be connected, or a cable capable of receiving a broadcast signal according to a cable broadcasting standard can be connected. do. As another example, the electronic device 100 may have a built-in antenna capable of receiving a broadcast signal. Wired interface unit 111 is an HDMI port, DisplayPort, DVI port, Thunderbolt, composite video, component (component) video, super video (super video), SCART, such as, such as, video and / or audio transmission standards according to It may include a connector or a port, and the like. The wired interface unit 111 may include a connector or port according to a universal data transmission standard, such as a USB port. The wired interface unit 111 may include a connector or a port to which an optical cable can be connected according to an optical transmission standard. The wired interface unit 111 is connected to an external microphone or an external audio device having a microphone, and may include a connector or a port capable of receiving or inputting an audio signal from the audio device. The wired interface unit 111 is connected to an audio device such as a headset, earphone, or external speaker, and may include a connector or port capable of transmitting or outputting an audio signal to the audio device. The wired interface unit 111 may include a connector or port according to a network transmission standard such as Ethernet. For example, the wired interface unit 111 may be implemented as a LAN card connected to a router or a gateway by wire.

The wired interface unit 111 is wired through the connector or port in a 1:1 or 1:N (N is a natural number) method with an external device such as a set-top box, an optical media player, or an external display device, speaker, server, etc. By being connected, a video/audio signal is received from the corresponding external device or a video/audio signal is transmitted to the corresponding external device. The wired interface unit 111 may include a connector or port for separately transmitting video/audio signals.

In addition, according to an embodiment, the wired interface unit 111 is embedded in the electronic device 100 , but may be implemented in the form of a dongle or a module to be detachably attached to the connector of the electronic device 100 .

The interface unit 110 may include a wireless interface unit 112 . The wireless interface unit 112 may be implemented in various ways corresponding to the implementation form of the electronic device 100 . For example, the wireless interface unit 112 is a communication method RF (radio frequency), Zigbee (Zigbee), Bluetooth (bluetooth), Wi-Fi (Wi-Fi), UWB (Ultra WideBand) and NFC (Near Field Communication), etc. Wireless communication can be used. The wireless interface unit 112 may be implemented as a wireless communication module that performs wireless communication with an AP according to a Wi-Fi method, or a wireless communication module that performs one-to-one direct wireless communication such as Bluetooth. The wireless interface unit 112 may transmit and receive data packets to and from the server by wirelessly communicating with the server on the network. The wireless interface unit 112 may include an IR transmitter and/or an IR receiver capable of transmitting and/or receiving an IR (Infrared) signal according to an infrared communication standard. The wireless interface unit 112 may receive or input a remote control signal from a remote control or other external device through an IR transmitter and/or an IR receiver, or transmit or output a remote control signal to another external device. As another example, the electronic device 100 may transmit/receive a remote control signal to and from the remote control or other external device through the wireless interface unit 112 of another method such as Wi-Fi or Bluetooth.

When the video/audio signal received through the interface unit 110 is a broadcast signal, the electronic device 100 may further include a tuner for tuning the received broadcast signal for each channel.

When the electronic device 100 is implemented as a display device, it may include the display unit 120 . The display unit 120 includes a display panel capable of displaying an image on the screen. The display panel is provided with a light-receiving structure such as a liquid crystal type or a self-luminous structure such as an OLED type. The display unit 120 may further include additional components according to the structure of the display panel. For example, if the display panel is a liquid crystal type, the display unit 120 includes a liquid crystal display panel and a backlight unit for supplying light. and a panel driving substrate for driving the liquid crystal of the liquid crystal display panel.

The electronic device 100 may include a user input unit 130 .

The user input unit 130 transmits various preset control commands or non-limited information to the processor 180 in response to a user's input. The user input unit 130 includes various input means capable of receiving a user input.

In an embodiment, the user input unit 130 may include a keypad (or input panel) including buttons such as a power key, a number key, and a menu key provided in the electronic device 100 .

In an embodiment, the user input unit 130 may include an input device that generates a preset command/data/information/signal to be able to remotely control the electronic device 100 and transmits the generated command/data/information/signal to the electronic device 100 . The input device includes, for example, a remote control, a game console, a keyboard, a mouse, and the like, and is separated from the electronic device 100 to receive a user input. can

At least one button for receiving a user input may be provided in the remote control. In an embodiment, the remote control may be provided with a touch sensing unit that receives a user's touch input (touch gesture) and/or a motion sensing unit that detects movement of the remote control itself by the user. In an embodiment, the input device includes a terminal device such as a smartphone on which a remote control application is installed, and in this case, a user's touch input through the touch screen can be received.

The input device becomes an external device capable of wireless communication with the main body of the electronic device 100 , and wireless communication includes Bluetooth, infrared communication, RF communication, wireless LAN, Wi-Fi Direct, and the like.

In an embodiment, the user input unit 130 may include a motion sensing unit that detects a motion of the user's hand, that is, a hand gesture (hereinafter, also referred to as a gesture). The electronic device 100 may output data by detecting a movement distance, a movement speed, an area of a movement region, and the like, of the hand by the motion detection unit.

In an embodiment, the user input unit 130 may include a touch sensing unit capable of detecting a user's touch on the bezel area around the display 121 .

In an embodiment, a sound receiving unit 130 such as a microphone 150 capable of receiving a voice uttered by a user may be included in the user input unit 130 .

In an embodiment, the user input unit 130 may receive a user input for a set distance that is a standard for privacy processing for image data. For example, a user input for setting or adjusting (changing) a set distance or a reference position corresponding to the set distance may be received through the user input unit 130 .

The electronic device 100 may include a storage unit 140 . The storage unit 140 stores digitized data. The storage unit 140 is a nonvolatile storage capable of preserving data regardless of whether power is supplied or not, and data to be processed by the processor 180 is loaded, and data is stored when power is not provided. This includes memory with volatile properties that cannot be used. Storage includes flash-memory, hard-disc drive (HDD), solid-state drive (SSD), read-only memory (ROM), etc., and memory includes buffer and random access memory (RAM). etc.

The electronic device 100 may include a microphone 150 . The microphone 150 collects sounds of the external environment including the user's voice. The microphone 150 transmits the collected sound signal to the processor 180 . The electronic device 100 may include a microphone 150 for collecting user voices or may receive a voice signal from an external device such as a remote controller having a microphone or a smart phone through the interface unit 110 . A remote controller application may be installed in an external device to control the electronic device 100 or perform functions such as voice recognition. In the case of an external device installed with such an application, a user voice can be received, and the external device can transmit/receive and control data using the electronic device 100 and Wi-Fi/BT or infrared rays, etc. A plurality of interface units 110 that can be implemented may exist in the electronic device 100 .

The electronic device 100 may include a speaker 160 . The speaker 160 outputs audio data processed by the processor 180 as sound. The speaker 160 may include a unit speaker provided to correspond to audio data of one audio channel, and may include a plurality of unit speakers to respectively correspond to audio data of a plurality of audio channels. As another embodiment, the speaker 160 may be provided separately from the electronic device 100 . In this case, the electronic device 100 may transmit audio data to the speaker 160 through the interface unit 110 .

The electronic device 100 may include a sensor 170 . The sensor 170 may detect a state of the electronic device 100 or a state around the electronic device 100 , and transmit the sensed information to the processor 180 . The sensor 170 includes a camera.

In addition, a magnetic sensor, an acceleration sensor, a temperature/humidity sensor, an infrared sensor, a gyroscope sensor, a position sensor (eg, GPS), a barometric pressure sensor, a proximity sensor, and an RGB sensor (illuminance sensor) It may include at least one, but is not limited thereto. Since a function of each sensor can be intuitively inferred from the name of a person skilled in the art, a detailed description thereof will be omitted.

The electronic device 100 may include a processor 180 . The processor 180 includes one or more hardware processors implemented with a CPU, a chipset, a buffer, a circuit, etc. mounted on a printed circuit board, and may be implemented as a system on chip (SOC) depending on a design method. When the electronic device 100 is implemented as a display device, the processor 180 includes modules corresponding to various processes such as a demultiplexer, a decoder, a scaler, an audio digital signal processor (DSP), and an amplifier. Here, some or all of these modules may be implemented as SOC. For example, a module related to image processing such as a demultiplexer, decoder, and scaler may be implemented as an image processing SOC, and an audio DSP may be implemented as a chipset separate from the SOC.

When a voice signal for the user's voice is obtained by the microphone 150 or the like, the processor 180 may convert the voice signal into voice data. In this case, the voice data may be text data obtained through a speech-to-text (STT) process for converting a voice signal into text data. The processor 180 identifies a command indicated by the voice data and performs an operation according to the identified command. The voice data processing process and the command identification and execution process may all be executed in the electronic device 100 . However, in this case, since the system load and required storage capacity required for the electronic device 100 become relatively large, at least a part of the process is performed by at least one server communicatively connected to the electronic device 100 through a network. can be performed.

The processor 180 according to an embodiment may call at least one command among commands of software stored in a storage medium readable by a machine such as the electronic device 100 and execute it. . This enables a device such as the electronic device 100 to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (eg, electromagnetic wave), and this term refers to cases in which data is semi-permanently stored in a storage medium and temporary storage. It does not distinguish between cases where

On the other hand, the processor 180 identifies the user-related information of the first image acquired by the first camera, and corresponds to the switch from the first camera to another second camera, based on the identified user-related information, the second Data analysis, processing, to identify the user area in the second image acquired by the camera, and transmit the call image corresponding to the user area in the identified second image to the other party device of the video call through the interface unit 110; And at least a part of the result information generation may be performed using at least one of machine learning, a neural network, and a deep learning algorithm as a rule-based or artificial intelligence algorithm.

The processor 180 identifies the user-related information of the first image acquired by the first camera, and identifies the user area in the second image acquired by the second camera based on the identified user-related information to model the artificial intelligence It can be converted into a form suitable for use as an input of AI models can be created through learning. Here, being made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined action rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created means burden. The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and the plurality of weights.

Inference prediction is a technology for logically reasoning and predicting by judging information. It is a technology that uses knowledge based reasoning, optimization prediction, preference-based planning, and recommendation. includes

For example, the processor 180 may perform the functions of the learning unit and the recognition unit together. The learning unit may perform a function of generating a learned neural network, and the recognition unit may perform a function of recognizing (or inferring, predicting, estimating, determining) data using the learned neural network. The learning unit may create or update the neural network. The learning unit may acquire learning data to generate a neural network. For example, the learning unit may acquire the learning data from the storage 140 or the outside. The learning data may be data used for learning of the neural network, and the neural network may be trained by using the data obtained by performing the above-described operation as learning data.

The learning unit may perform a preprocessing operation on the acquired training data before training the neural network using the training data, or may select data to be used for learning from among a plurality of training data. For example, the learning unit may process the learning data in a preset format, filter it, or add/remove noise to process the learning data into a form suitable for learning. The learner may generate a neural network set to perform the above-described operation by using the preprocessed learning data.

The learned neural network network may be composed of a plurality of neural network networks (or layers). Nodes of the plurality of neural networks have weights, and the plurality of neural networks may be connected to each other so that an output value of one neural network is used as an input value of another neural network. Examples of neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN) and It can include models such as Deep Q-Networks.

Meanwhile, the recognizer may acquire target data to perform the above-described operation. The target data may be obtained from the storage 140 or from the outside. The target data may be data to be recognized by the neural network. The recognizer may perform preprocessing on the acquired target data before applying the target data to the learned neural network, or select data to be used for recognition from among a plurality of target data. For example, the recognition unit may process the target data into a preset format, filter, or add/remove noise to process the target data into a form suitable for recognition. The recognizer may obtain an output value output from the neural network by applying the preprocessed target data to the neural network. The recognition unit may obtain a probability value or a reliability value together with the output value.

For example, the control method of the electronic device 100 according to an embodiment may be provided by being included in a computer program product. The computer program product may include instructions for software executed by the processor 180 , as described above. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg, CD-ROM), or via an application store (eg, Play Store™) or between two user devices (eg, smartphones). It may be distributed directly, online (eg, downloaded or uploaded). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to an embodiment, the electronic device 100 may support a video call (video call) function. As the video call application is installed and executed in the electronic device 100 , the video call function is activated, so that a video call service may be provided to the user. In the present disclosure, a video call includes a video conference (video conference) with one or two or more counterparts.

According to an embodiment, the processor 180 identifies user-related information of the first image acquired by the first camera ( S310 ).

The first camera is an external type, and the processor 180 receives data obtained by the first camera through the interface unit 110 . At this time, in the case of voice or sound, it is possible to collect a voice signal from the microphone 150 provided in the electronic device 100 or receive a voice signal from an external device such as a remote controller or smart phone having a microphone through the interface unit 110 . can

User-related information means information related to the user area, and more specifically, due to the nature of the video call, the user's face (features, face shape, hairstyle, etc.), the user's body (object), the user's voice, It includes objects or backgrounds that appear in front of or behind the user, the position of the face on the screen, the aspect ratio, and the relationship between the aspect ratio and the face/object ratio.

According to an embodiment, the processor 180 identifies a user area in the second image acquired by the second camera based on the identified user-related information, in response to switching from the first camera to another second camera. (S320).

In this case, the second camera may be of an external type like the first camera, and on the other hand, as another embodiment, the electronic device 100 may include a camera capable of capturing the surroundings of the electronic device 100 . When the second camera is an external type, it may exist in the form of being mounted on the electronic device 100 by being connected through the wired interface 111 or the wireless interface 112 as an embodiment.

The processor 180 may receive a signal requesting to switch from the first camera to the second camera that is another camera. The camera switching request may be received from an external device through the interface unit 110 or may be received through the user input unit 130 .

The user area refers to an area in which the user is located, and includes, for example, a face area including the user's face and an object area including the body. The processor 180 may identify the user's face using an image recognition technology, for example, an infrared sensor or a technology for extracting feature points. In addition, in the area recognized as a face, an area within a predetermined range based on the central point may be set as the face area, and an area including the face and the user's body may be set as the object area.

A more specific embodiment will be described with reference to FIG. 4 .

According to an embodiment, the processor 180 transmits a call image corresponding to the user area in the identified second image to the other party device of the video call through the interface unit 110 (S330).

As the camera is switched from the first camera to the second camera, there may be differences in displayed images due to differences in the angle of view of the camera or the distance and location between the user and the camera.

In one embodiment, in order to prevent an unnecessary or unwanted portion of the image from being displayed due to the difference in images, the processor 180 determines the user in the second image identified based on the user-related information identified in the first image. The call image corresponding to the area is transmitted to the other party's device of the video call. A more specific embodiment will be described with reference to FIG. 5 .

According to an embodiment, it is possible to provide a natural camera switching service by minimizing a difference in an image that occurs according to a camera switching during a video call. In addition, it is possible to contribute to privacy protection by preventing exposure of a background or a person not desired by the user in the image by switching the camera. Through this, it is possible to increase the concentration of the call by transmitting the object required in the video call to the other party, which can be a factor to increase the work efficiency in the video conference.

This figure shows a state in which the user area in the second image 20 is identified based on the location or size of the user area in the first image 10 in relation to S320 of FIG. 3 .

As described above, the

user areas

410 and 420 refer to areas in which the user is located, for example, face

areas

411 and 421 including the user's face and

object areas

412 and 422 including the body. ) is included.

Identifying the user area means identifying a characteristic sufficient to identify the area corresponding to the first image in another image by comparing it with other images later. More specifically, it includes identifying the location or size characteristic of the user area with respect to the entire image area or the screen area.

The processor 180 first identifies a face region 411 and an object region 412 in the first image 10 displayed by the first camera, and extracts information related to each region.

For example, the processor 180 recognizes the user's features, face shape, eyebrow shape, and hairstyle in the face region 411 , and in the object region 412 , the shape, type, color, and user of clothes worn by the user. Recognize poses, etc.

In addition, the processor 180 may identify a user area in the image based on a user input received through the user input unit 130 . In this case, the user input includes a user voice input. The processor 180 may store the recognized user's face and the received user's voice in the storage 150 in correspondence with each other.

On the other hand, in the case of the first image 10 , the first camera is an external type, and the processor 180 receives data obtained by the first camera through the interface unit 110 , and the first camera through the interface unit 110 . Information related to the first image 10 may be received together from an external device including a camera.

The user-related information according to an embodiment includes information about at least one of a location and a size of a user area in an image. The processor 180 may identify the user area 420 in the second image 20 based on the location or size of the user area 410 in the first image 10 .

The processor 180 identifies the position on the screen of the face region 411, the position on the screen of the object region 412, the aspect ratio, the relationship between the aspect ratio and the ratio of the face region 411, the object region 412, etc. Based thereon, the user area 420 in the second image 20 may be identified based on the location or size of the user area 410 in the first image 10 .

As shown in FIG. 4 , the vertical and horizontal lengths of the screen on which the first image 10 is displayed are x and y, respectively, and the vertical and horizontal lengths of the face region 411 identified in the first image 10 are x', y', the vertical and horizontal lengths of the object region 412 are x'' and y''. In this case, the ratios of the screen and each area are set to x:y, x':y', and x'':y'' for convenience of calculation.

The processor 180 also identifies the user area 420 including the face area 421 and the object area 422 in the second image 20 acquired by the second camera. The vertical and horizontal lengths of the second image 20 are a and b, respectively, and the vertical and horizontal lengths of the face region 421 identified in the second image 20 are a′, b′, and the object region 422 . The length and width are a'',b''. In this case, the ratios of the screen and each area are set to a:b, a':b', a'':b'' for convenience of calculation. In addition, this drawing is illustrated without considering the perspective, and the sizes of the first image 10 and the second image 20 may differ from what is shown in that a may be much larger than x. Hereinafter, the same applies to the rest of the drawings.

In this case, in the case of the second image 20 acquired by the second camera, a plurality of people including the user are displayed, and the processor 180 may identify a user area for each person. The processor 180 specifies a user area 420 corresponding to the user area 410 identified in the first image 10 from among the user areas identified in the second image 20 .

Alternatively, for faster processing, the processor 180 identifies the user area 420 corresponding to the user area 410 in the first image 10 in the second image 20, processing the user area identification for the remaining persons. can be stopped

In addition to this, the user-related information includes information about the angle of view of the camera. The processor 180 may identify the user area in the second image based on the relationship between the angle of view of the first camera and the angle of view of the second camera.

According to an embodiment, a region corresponding to the user region of the first image may be more accurately identified in the converted second image by identifying the user region in the first image.

This figure shows a state in which a call image corresponding to the user area in the identified second image is transmitted to the other party device of the video call through the interface in relation to S330 of FIG. 3 .

The configuration for identifying the user area in the first image 10 and the user area in the second image 20 is the same as described in FIG. 4 .

As described above with reference to FIG. 4 , the processor 180 identifies the user area 410 in the first image 10 and includes the corresponding user area 420 in the second image 20 . A partial image 431 of the second image 20 is identified. The partial image 431 means a call image to be transmitted to the counterpart device of the video call. The processor 180 may transmit a call image corresponding to the user area in the identified second image to the other party device of the video call through the interface unit 110 . According to an embodiment, the processor 180 may obtain a partial image 431 by cropping an unnecessary portion of the second image 20 .

In addition, the processor 180 may adjust some acquired images 431 .

As an example, the processor 180 is configured to display the first image 10 based on an aspect ratio (x:y) and a ratio (x':y') of the user area 410 and the size of the user area 410 . , the relationship between the aspect ratio (x:y) in which the identified partial image 431 is displayed and the ratio (x':y', x'':y'') of the user area 410 on which the first image 10 is displayed While maintaining , the adjusted image 510 may be displayed by adjusting it to fit the screen ratio of the display 121 .

In the embodiment of FIG. 5 , the aspect ratio of the display 121 and the screen ratio at which the image 10 is displayed are different. In this case, in light of the fact that the screen on which the first image 10 is displayed is vertically long, the processor 180 determines the length (x) of the screen on which the first image is displayed and the face region 411 of the first image. Based on x: x', which is the ratio of the vertical (x') of the screen of The ratio a:a' of the length (a') may be adjusted to correspond to the ratio x:x'. However, the present invention is not limited thereto, and the processor 180 may adjust not only the ratio of the area but also the location of the area (where the area is located in the horizontal and vertical directions on the screen, etc.). At this time, in this drawing, it is assumed that the size of the screen of the display 121 and the size of the second image 20 are the same.

6 , a specific example of the operation of the above-described embodiment will be described.

The user is conducting a video call using the external device 200 . The external device may be, for example, a mobile phone. The user intends to conduct a video call using the electronic device 100 while making a video call with the camera built into the external device 200 .

The processor 180 executes a video call application based on receiving a user input requesting camera change through any one of the external device 200 , the interface unit 110 , or the user input unit 130 , and performs a video call. Activate the function.

At this time, the processor 180 receives data about the image 610 displayed on the external device 200 to identify a user area of the image 610 or transmits information about the user area of the image 610 to the interface unit ( 110) can be received. When the processor 180 receives information about the user area from the external device 200 through the interface 110 , the processor 180 automatically searches for a user based on the received information and is included in the received information. The image may be cropped and displayed on the display 121 at the same ratio as the screen ratio.

The processor 180 may acquire an image using another camera connected to the electronic device 100 , and display the image 620 on the display 121 so that the acquired image corresponds to a user area of the image 610 . . The processor 180 may transmit the identified image 620 to the counterpart device of the video call through the internal interface unit 110 .

As the video call function performed in the external device 200 is activated in the electronic device 100, the image acquired based on the switched camera is adjusted to correspond to the image 610 as described in the preceding figure.

The processor 180 transmits the image 610 to the counterpart device while preparing the display of the image 620 in response to the transition from the first camera to the second camera, that is, while the user area is identified, or transfers the image 610 to another image. can be transferred instead. This is because, when the image displayed by the second camera is transmitted as it is without the display of the image 620 being prepared, there is a possibility that an image that the user does not want is still exposed. Accordingly, various embodiments may be applied, such as displaying the image 610 with a partial delay before displaying the image 620 , blurring the screen itself, or displaying a screen indicating that the screen is being prepared.

7 is a diagram illustrating an operation of an electronic device according to an exemplary embodiment. 7 illustrates an operation of the electronic device after the processor 180 transmits the image 620 to the counterpart device in FIG. 6 .

The electronic device 100 according to an embodiment may further include a user input unit 130 . The processor 180 may adjust the user area in the image 620 based on the user input received through the user input unit 130 and transmit a call image corresponding to the adjusted user area to the counterpart device.

The processor 180 may display a user interface (UI) indicating the user area within the image 620 and adjust the user area within the image 620 based on a user input received using the UI.

After the image 620 is displayed to correspond to the user area of the image 610 , the user may make various adjustments, such as additionally displaying the intended part or displaying it by enlarging/reducing the image 610 .

The user input unit 130 includes various input means capable of receiving a user input for adjusting an image, for example, a physical button or a touch pad (or input panel) provided in the electronic device 100 . The processor 180 may display a graphic user interface (GUI) capable of adjusting an image on the display 121 and receive a user input for adjusting the image through a touch pad. Alternatively, it is separated from the electronic device 100 and is separated from an input device that transmits a controllable signal through the wired/wireless interface unit 110, for example, from a remote control, a keyboard, a mouse, etc. A user input for adjusting an image in the displayed GUI may be received.

If you want to display another person sitting next to the user as shown in FIG. 7 , the image 620 is adjusted to the image 710 based on the user input received through the processor 180 user input unit 130 . displayed on the display 121 .

According to an embodiment, first, by naturally switching the camera, unnecessary parts are prevented from being displayed in the image before and after the change, and the user's convenience can be enhanced by allowing the user to transmit the desired image.

8 is a diagram illustrating an operation of an electronic device according to an exemplary embodiment. FIG. 8 shows an operation state of the electronic device after the processor 180 displays the image 620 in FIG. 6 .

FIG. 8 shows that the operation described with reference to FIG. 7 is performed using the external device 200 . The processor 180 may be connected to the external device 200 by wire or wirelessly through the interface unit 110 .

The processor 180 transmits data of the image 620 and information on the user area within the image 620 so that the external device 200 adjusts the user area within the image 620 through the interface unit 110 to the external device ( 200) is sent.

In this embodiment, the external device 200 includes a terminal device such as a smart phone in which a remote control application is installed, and in this case, a user's touch input through the touch screen can be received. The external device 200 is capable of wireless communication with the main body of the electronic device 100 , and the wireless communication includes Bluetooth, infrared communication, RF communication, wireless LAN, Wi-Fi Direct, and the like.

The external device 200 adjusts the image 620 based on a user input through software capable of manually adjusting the magnification. In the scaling software, you can set the cropped range and coordinates against the original data.

The external device 200 receives a user input to enlarge the image 620 , enlarges the image 620 , drags and drops it to the electronic device 100 , and transmits it again.

At this time, the user input that can be performed in the external device 200 is not limited to a technique that can be applied in relation to image editing as well as enlargement, and the image is transmitted from the external device 200 back to the electronic device 100 . The technique may be variously implemented in addition to drag and drop.

In addition, since remote technology is applied, adjustment in real time may be possible regardless of transmission technology such as drag and drop.

The processor 180 displays the image 810 received through the interface unit 110 on the display 121 or converts the image 620 into the image 810 based on a user input received through the interface unit 110 . adjust to be

According to an embodiment, in adjusting an image, an external terminal device such as a wireless terminal may be used to conveniently manipulate the image.

As mentioned above, although preferred embodiments of the present disclosure have been described in detail, the present disclosure is not limited thereto, and may be variously practiced within the scope of the claims.

Claims

In an electronic device,

interface unit;

display; and

Identifies the user-related information of the first image acquired by the first camera,

In response to switching from the first camera to another second camera, based on the identified user-related information, identify a user area in the second image acquired by the second camera,

and a processor for transmitting a call image corresponding to the user area in the identified second image to a counterpart device of the video call through the interface unit.
According to claim 1,

The user-related information includes information about at least one of the location and size of the user area in the first image,

The processor is

An electronic device for identifying a user area in the second image based on a location or size of the user area in the first image.
According to claim 1,

The user-related information includes information about the angle of view of the first camera,

The processor is

An electronic device for identifying a user area in the second image based on a relationship between the angle of view of the first camera and the angle of view of the second camera.
According to claim 1,

The processor is

An electronic device receiving the user-related information from an external device including the first camera through the interface unit.
According to claim 1,

The processor is

In response to the switching from the first camera to the second camera, while the user area in the second image is identified, the electronic device transmits the first image to the counterpart device or replaces it with another image for transmission.
According to claim 1,

The electronic device further comprising the second camera.
According to claim 1,

Further comprising a user input unit,

The processor is

An electronic device for identifying a user area in the second image based on a user input received through the user input unit.
8. The method of claim 7,

The user input is an electronic device including a user voice input.
According to claim 1,

Further comprising a user input unit,

The processor is

Based on the user input received through the user input unit, adjust the user area in the second image,

An electronic device for transmitting a call image corresponding to the adjusted user area to the counterpart device.
10. The method of claim 9,

The processor is

Displaying a UI (User Interface) indicating a user area in the second image,

An electronic device for adjusting a user area in the second image based on the user input received using the UI.
The method of claim 1,

The processor is

Based on a user input received through an external device, adjust the user area in the second image,

An electronic device for transmitting a call image corresponding to the adjusted user area to the counterpart device.
12. The method of claim 11,

The processor is

The electronic device transmits information on the user area in the second image to the external device through the interface unit so that the external device adjusts the user area in the second image.
In the control method of an electronic device,

identifying user-related information of a first image acquired by a first camera;

identifying a user area in a second image acquired by the second camera based on the identified user-related information in response to switching from the first camera to another second camera; and

and transmitting a call image corresponding to the user area in the identified second image to a counterpart device of the video call through an interface unit.
14. The method of claim 13,

The user-related information includes information about at least one of the location and size of the user area in the first image,

The step of identifying the user area in the second image,

and identifying the user area in the second image based on the location or size of the user area in the first image.
A computer-readable code, comprising a computer program stored thereon, comprising a code for performing a control method of an electronic device

The control method of the electronic device,

identifying user-related information of a first image acquired by a first camera;

identifying a user area in a second image acquired by the second camera based on the identified user-related information in response to switching from the first camera to another second camera; and

and transmitting the call image corresponding to the user area in the identified second image to the other party device of the video call through the interface unit.