KR20220061763A - Electronic device providing video conference and method for providing video conference thereof - Google Patents

Abstract

Disclosed are an electronic device and method for providing a video conference service to provide participants with an optimized screen according to the flow of a presentation. According to various embodiments of the present invention, the electronic device comprises a communication module, an input module, a display, a memory, and a processor operatively connected to the communication module, the input module, the display, and the memory. The processor is configured to analyze operating content information to generate content analysis information; analyze received presenter video information to generate presenter video analysis information; divide a content execution screen for the content information into a plurality of areas on the basis of the content analysis information; select at least one region to place the presenter image information among the plurality of regions of the content execution screen on the basis of the content analysis information and the presenter image analysis information; and arrange and synthesize the presenter image information in a selected region from the content execution screen.

Description

ELECTRONIC DEVICE PROVIDING VIDEO CONFERENCE AND METHOD FOR PROVIDING VIDEO CONFERENCE THEREOF

Various embodiments of the present document relate to an electronic device, for example, to an electronic device that provides a video conference and a method for providing a video conference.

Recently, the use of remote video conferencing technology has increased due to an increase in the proportion of untact business and the development of high-speed wireless communication networks. Remote video conferencing-related technologies aim to deliver video and audio information of users clearly and quickly to meeting participants, and aim for a system in which dozens or hundreds of participants can participate depending on the service. In addition, it provides a function to share various additional contents such as videos and data, not just transmitting the video input to the camera.

The current video conferencing services independently provide the presenter's video screen and the presentation material (content) screen or file shared by the presenter, which allows the user to selectively view the desired video according to the user's choice. . However, there is a service that divides the video of the presenter and the content of the presenter and displays them on one screen. In this case, the presenter's video may cover a part of the content, and if you want to see the part of the location, you may need to change the location of the presenter's video from time to time. Such user intervention may reduce the concentration of the presenter or viewer.

In addition, it is also conceivable that the presentation material and the presenter appear in the same video at the same time, and the presenter makes an offline presentation with the content in the background as a video, and transmits it to the viewer. In this case, it may be difficult to identify which part of the content part or the presenter part is out of focus in the entire video or the part of the content pointed out or explained by the presenter. Even in this case, the viewer's concentration may be reduced.

An electronic device providing a video conferencing service according to various embodiments of the present disclosure includes a communication module, an input module, a display, a memory, and operatively connected to the communication module, the input module, the display, and the memory. a processor, wherein the processor generates content analysis information by analyzing the content information being executed, generates the presenter image analysis information by analyzing the received presenter image information, and adds the content information to the content information based on the content analysis information. divides the content execution screen for the content execution screen into a plurality of regions, and selects at least one region in which the presenter image information is to be arranged among the plurality of regions of the content execution screen based on the content analysis information and the presenter image analysis information, and The presenter image information may be set to be synthesized by arranging it in a selected area among areas of the content execution screen.

A method of providing a video conferencing service according to various embodiments disclosed herein includes the steps of generating content analysis information by analyzing running content information, analyzing the received presenter image information to generate presenter image analysis information; dividing the content execution screen for the content information into a plurality of regions based on the content analysis information; the presenter image information among a plurality of regions of the content execution screen based on the content analysis information and the presenter image analysis information The method may include selecting at least one area in which to place , and synthesizing the presenter image information into a selected area among areas of the content execution screen.

A server device according to various embodiments disclosed herein includes a communication interface, a memory, and a processor operatively connected to the communication interface and the memory, wherein the processor analyzes presenter image information and presenter image from an external electronic device Receive information, content execution screen information, and content analysis information, divide the content execution screen into a plurality of regions based on the content analysis information, and arrange the presenter image information among the plurality of regions of the content execution screen. It may be set to select one region and synthesize the presenter image information into the selected region among regions of the content execution screen.

An electronic device according to various embodiments disclosed herein includes a communication module, an input module, a display, a memory, and a processor operatively connected to the communication module, the input module, the display, and the memory, wherein the The processor receives image information, analyzes the image information to extract presenter image information and content image information, analyzes the extracted content image information to generate content analysis information, and analyzes the presenter image to obtain a presenter image generating analysis information, dividing the content image information into a plurality of regions based on the content analysis information, selecting at least one region in which the presenter image information is to be placed among the plurality of regions of the content image information, and It may be set to synthesize the presenter image information into a selected region among the regions of the content image information.

According to various embodiments, an optimized screen according to the flow of a presentation may be provided to participants by providing a video synthesized by dynamically arranging content and a presenter video among video conference participants, and between participants participating in a video conference Efficient information sharing and opinion exchange are possible.

1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure;
2 illustrates various devices for providing a video conference according to various embodiments.
3 is a block diagram of an electronic device according to various embodiments of the present disclosure;
4 illustrates an embodiment of providing video conferencing according to various embodiments.
5 is a block diagram for providing a video conference of an electronic device according to various embodiments of the present disclosure;
6 is a flowchart of a method of providing a video conference by an electronic device according to various embodiments of the present disclosure;
7 is a flowchart of a method for providing a video conference by an electronic device and a server according to various embodiments of the present disclosure;
8 illustrates an embodiment of analyzing a presenter image according to various embodiments of the present disclosure.
9 is a flowchart of a method for an electronic device to analyze a presenter image according to various embodiments of the present disclosure;
10, 11, 12, and 13 are diagrams illustrating an embodiment of analyzing content according to various embodiments.
14 is a flowchart of a method of analyzing content by an electronic device according to various embodiments of the present disclosure;
15 is a diagram illustrating an example in which regions of a content screen are divided and a presenter image is dynamically arranged according to various embodiments of the present disclosure;
16 is a flowchart of a method for an electronic device to divide a region of a content screen and dynamically arrange a presenter image, according to various embodiments of the present disclosure;

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1 , in a network environment 100 , the electronic device 101 communicates with the electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 may be included. In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be

The processor 120, for example, executes software (eg, the program 140) to execute at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120 . It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be stored in the volatile memory 132 , and may process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 uses less power than the main processor 121 or is set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The secondary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or when the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which artificial intelligence is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176 ) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 or an external electronic device (eg, a sound output module 155 ) directly or wirelessly connected to the electronic device 101 . Sound may be output through the electronic device 102 (eg, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 includes various technologies for securing performance in a high frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. Technologies such as full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements specified in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to an embodiment, the wireless communication module 192 may include a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency ( Example: downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be chosen. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 is to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 illustrates various devices for providing a video conference according to various embodiments.

According to various embodiments, the electronic device 210 (eg, the electronic device 101 of FIG. 1 ) may use the presenter image information and the content information to synthesize and output the presenter image with the content information. The electronic device 210 may transmit the synthesized image to the server device 220 . The presenter video information may include an image, a video, and/or a voice. The content information may be visual or audiovisual material for presentation, and may include various multimedia data. For example, the content information includes text, an image, a moving picture, data that can be executed as an application executable in the electronic device 210 or output on the display of the electronic device 210 (eg, Power Point TM data); It may include a combination of at least some of them. The information in which the content information and the presenter's image information are synthesized may be image information or an image in which the content screen and the presenter's image information are synthesized.

According to various embodiments, the electronic device 210 may analyze presenter image information and generate presenter image analysis information. The electronic device 210 may transmit the presenter image analysis information to the server device 220 . According to various embodiments, the electronic device 210 may analyze content information and generate content analysis information. The electronic device 210 may transmit content analysis information to the server device 220 .

Referring to FIG. 2 , the electronic device may be implemented as an electronic device such as a smart phone or a tablet PC (personal computer), but is not limited thereto, and various types having communication functions (eg, wired communication and/or wireless communication) The devices of may correspond to the electronic device 210 according to various embodiments of the present document. According to various embodiments, the electronic device 210 may communicate with the server device 220 using wired and/or wireless network communication (eg, the second network 199 of FIG. 1 ). Wired and/or wireless network communication may be using a telecommunication network (eg, a cellular network, LAN).

According to various embodiments, the server device 220 synthesizes the presenter image with the content information by using the presenter image information and the content information, and uses the synthesized image with another device (eg, the electronic device 210 and/or an external device). may be transmitted to the electronic device 230 ). According to various embodiments, the server device 220 may analyze the presenter image information received from the electronic device 210 and generate presenter image analysis information. According to various embodiments, the server device 220 may receive the presenter image analysis information generated by the electronic device 210 from the electronic device 210 . According to various embodiments, the server device 220 may analyze content information and generate content analysis information. According to various embodiments, the server device 220 may receive the content analysis information generated by the electronic device 210 from the electronic device 210 . The server device 220 generates a presenter image or content information received from the electronic device 210 , and uses the presenter image analysis information and content analysis information received from the electronic device 210 or directly to generate the presenter image in the content information. can be synthesized. The information in which the content information and the presenter's image information are synthesized may be image information or an image in which the content screen and the presenter's image information are synthesized. The server device 220 may transmit the synthesized image to the electronic device 210 and/or the external electronic device 230 . According to various embodiments, the server device 220 may communicate with the electronic device 210 and/or the external electronic device 230 using wired and/or wireless network communication. Wired and/or wireless network communication may be using a telecommunication network (eg, a cellular network, LAN, Internet).

3 is a block diagram of an electronic device according to various embodiments of the present disclosure;

Referring to FIG. 3 , an electronic device 300 according to various embodiments may include a communication module 310 , an input module 320 , a memory 330 , an output module 340 , and a processor 350 . , even if at least some of the illustrated configuration is omitted or substituted, there will be no hindrance to implementing various embodiments of the present document. The electronic device 300 may include at least some of the configuration and/or functions of the electronic device 101 of FIG. 1 and/or the electronic device 210 of FIG. 2 .

According to various embodiments, the communication module 310 may communicate with the server device 220 using wired and/or wireless network communication (eg, the second network 199 of FIG. 1 ). Wired and/or wireless network communication may be using a telecommunication network (eg, a cellular network, LAN). There is no limitation on the long-distance communication supported by the communication module 310, and for example, various communication methods such as a cellular network (eg, 5G, LTE-A), a LAN, and the Internet may be supported. According to various embodiments, the communication module 310 may receive presenter image information and/or content information from an external device (eg, the server device 220 of FIG. 2 ), and synthesize the presenter image information and content information. Various data (eg, presenter image analysis information, content analysis information) may be transmitted to the server device 220 . According to various embodiments, the communication module 310 may receive presenter image information and/or content information from an external device (eg, the server device 220 of FIG. 2 ), and analyze the presenter image information and content information Based on the , an image in which presenter image information and content information are synthesized may be transmitted to the server device 220 .

According to various embodiments, the input module 320 receives a command or data to be used by a component (eg, the processor 350 ) of the electronic device 300 from the outside (eg, a user) of the electronic device 300 . can do. The input module 320 may include, for example, a camera, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen). The input module 320 may include at least a part of the configuration and/or functions of the input module 150 of FIG. 1 and/or the camera module 180 of FIG. 1 . According to various embodiments, the input module 320 may photograph the presenter, transmit presenter image information to another component (eg, the processor 350 ) of the electronic device 300 , and input the user's content (eg clicks, dragging, pointing) can be received.

According to various embodiments, the memory 330 is for temporarily or permanently storing digital data, and may include at least some of the configuration and/or functions of the memory 130 of FIG. 1 . Also, the memory 330 may store at least a portion of the program 140 of FIG. 1 . The memory 330 may store various instructions that may be executed by the processor 350 . Such instructions may include control commands such as logical operations and data input/output that may be recognized and executed by the processor 350 . There is no limitation on the type and/or amount of data that the memory 330 can store, but in this document, a method for providing a video conference according to various embodiments and an operation of the processor 350 performing the method and Only the configuration and function of the related memory will be described.

According to various embodiments, the output module 340 may provide information of the electronic device 300 to the outside. The output module 340 may output presenter image information, content information, and/or an image in which presenter image information and content information are synthesized, and may include, for example, a display and/or a speaker. The output module 340 may include at least a part of the configuration and/or functions of the display module 160 of FIG. 1 , the sound output module 155 of FIG. 1 , and/or the audio module 170 of FIG. 1 .

According to various embodiments, the processor 350 may process an operation or data related to control and/or communication of each component of the electronic device 300 . The processor 350 may include at least some of the configuration and/or functions of the processor 120 of FIG. 1 . The processor 350 is operatively, electrically and/or with components of the electronic device 300 such as the communication module 310 , the input module 320 , the memory 330 , and the output module 340 . Or it may be functionally linked. Each operation of the processor 350 according to various embodiments may be performed in real time. For example, the time for the processor 350 of the electronic device 300 to perform each operation for providing the video conference may be performed substantially simultaneously with the actual conference or within a predetermined time range.

According to various embodiments, the processor 350 may analyze image information. The image information may be, for example, image information corresponding to presenter image information and/or content information. The image information may be data captured by the processor 350 or image information received from an external electronic device. For example, the image information may be information received by the processor 350 from an external device (eg, the server device 220 of FIG. 2 ) using the communication module 310 , or the input module 320 . It may be data captured using . According to various embodiments, the processor 350 may use an artificial intelligence algorithm to analyze the presenter's image, for example, using a neural network (eg, artificial neural network (ANN), convolutional neural network (CNN)) The learned algorithm may be stored in advance, or algorithm data may be received from the server.

According to various embodiments, the processor 350 may extract an image from image information and perform image analysis. According to various embodiments, the processor 350 may analyze an image included in the image information to identify at least one object included in the image. The at least one object included in the image of the image information may include, for example, an image of a person (eg, a presenter), an image designated as presentation material, and a visually identifiable object such as text. The processor 350 may classify at least one object included in the image by category, and may perform an analysis on each object based on the classification or determine to execute a specified operation. For example, the processor 350 may classify an object recognized in an image according to object properties such as a person, an image, text, a screen, and a blackboard. The processor 350 may select the classified object. The selection operation may be an operation for reducing the complexity of image analysis. For example, the processor 350 may select only the objects required for analysis based on the size and frequency of appearance of the classified designated object.

According to various embodiments, the processor 350 may analyze image information and determine whether a presenter and content exist simultaneously in the same image. For example, when the presenter and the content exist in the same image, the corresponding image information may be information obtained by photographing an image in which the presenter makes a presentation with the content as a background offline. The processor 350 may recognize at least one object through analysis of received or photographed image information, classify the recognized at least one object, and determine which image information the image information is based on the object classification. there is. For example, if the main object recognized on the image information is a person, it can be recognized as a presenter image, and when a person and a designated object that can be classified in presentation materials (eg, content information) are recognized together, the image is displayed as a presenter It can be determined that the video is taken with the content as a background.

According to various embodiments, when it is determined that the presenter and the content exist in the same image, the processor 350 may separate and extract the presenter image from the image information. The processor 350 may extract image information of the presenter based on an image boundary of a portion recognized as a presenter using image extraction technology, and may analyze the presenter image information using the extracted presenter image information. According to various embodiments, the processor 350 may separate content existing in an image. For example, the content existing in the image may be a designated flat object such as a screen or a blackboard, and the processor 350 may recognize it as a designated visual object through image information analysis. The processor 350 may separate one or more objects recognized as content through object classification, extract it as an image, and generate one content image. The processor 350 may generate and extract a content image extracted from image information as content information.

According to various embodiments, the processor 350 may analyze the presenter image information. The presenter image information may be, for example, image data being photographed by the processor 350 or image information being received from an external electronic device. For example, the presenter image information may be information that the processor 350 receives from an external device (eg, the server device 220 of FIG. 2 ) using the communication module 310 , or the input module 320 . ) may be used to capture data. Alternatively, the presenter image information may be data obtained by the processor 350 extracting only the presenter image information from other image information. The processor 350 may recognize, classify, and select an image object in the presenter's image information. The analysis of the presenter's video information may be performed through the same or similar method as the above-described video information analysis. When an object that can be classified as a person is recognized in the presenter image information, the processor 350 may analyze gesture information and/or facial feature points. For example, the processor 350 may recognize a change in the recognized image object, and analyze the gesture information using pre-stored data, data received from a server, or an algorithm. According to various embodiments, the processor 350 may analyze the presenter's gesture through presenter image information analysis and generate gesture information. According to various embodiments, the processor 350 may analyze the facial feature points included in the image, and through this, it is possible to identify the presenter's expression information, the presenter's gaze information, and/or whether the presenter is speaking. According to various embodiments, the processor 350 may calculate an image boundary of the recognized object. The processor 350 may calculate an image boundary using pixel information of an image object, extract position information of each object based on the image boundary, and extract only pixel information within the image boundary as image information or presenter image information can be edited. For example, based on the boundary formed by the image object recognized as the presenter, only the image portion of the presenter shape in the presenter image may be cut out to edit the presenter image information. According to various embodiments, the processor 350 analyzes the presenter's video through analysis of the presenter's video, and the presenter's video analysis information including information on whether the speaker speaks, gesture information, gaze information, image boundary information, and presenter image information edited based on the image boundary can create

According to various embodiments, the processor 350 may analyze content information. The content information may be multimedia data that the presenter can use as presentation material. For example, the content information includes text, image, video, and data (eg, Power Point TM) data that can be executed as an application executable in the electronic device 210 or output on the display of the electronic device 210 . ) and a combination of at least some of them. The content information may be data executed or displayed by the processor 350 or multimedia data information received from an external electronic device. The content information may be information received by the processor 350 from an external device (eg, the server device 220 of FIG. 2 ) using the communication module 310 , or executed by the processor 350 , It may be data that is executed and output by the output module 350 . Alternatively, the content information may be image data in which the processor 350 extracts only an image region identified as content from other image information.

According to various embodiments, the processor 350 may determine the type of content information. For example, the type of application required to execute or display content information in the electronic device 300 may be checked. Alternatively, the type of content information may be determined through the type of an application for displaying content information that is already being executed. The type of content information may be divided into a case of video, image, text, or a combination thereof and other applications. A content analysis method may be different based on the determined type of content information.

According to various embodiments, when the content information includes text, an image, a moving picture, or a combination thereof, the processor 350 may analyze the text, image, and/or video. For example, the content information may be in the form of including a plurality of multimedia contents (eg, images and texts) in the same output screen. In this case, the processor 350 may recognize each content as a content object. The processor 350 may identify each recognized content object, and classify and select the identified object. For example, the identified content objects may be classified according to content types, such as images, texts, and images, respectively.

According to various embodiments, when the content information corresponds to data executable as an application other than the above type, the processor 350 may analyze execution data of the application and input data for the corresponding application. For example, the processor 350 may analyze a user interface (UI) of the application. According to various embodiments, the processor 350 uses a simulation technique to check each UI page and/or page on the UI layout such as a pop-up window, based on the simulation of the movement of the UI page of the corresponding application, and connection information for each window. Alternatively, such information may be previously stored in another component (eg, the memory 330 ) of the electronic device. According to various embodiments, the processor 350 may obtain mapping information between UI pages by combining each movable page and/or window for each UI page and/or window of the corresponding application. According to various embodiments, the processor 350 may analyze a UI component (eg, a UI object) that triggers page movement and generation of a pop-up window within the same UI page. This may be possible by reading control information included in each UI component in the same UI page. You can check the current UI page by combining all UI control information in the same page. Such control information and a combination thereof may be analyzed in real time by the processor 350 and may be information previously stored in the electronic device 300 . In addition, the processor 350 may check information about an object identified in the UI page.

According to various embodiments, the processor 350 means that each content object may be included in each content object using image analysis, text analysis (eg, optical character recognition (OCR)), and/or image analysis for each type of content object classified. It is possible to analyze units or extract related text information. The processor 350 may extract content information of each content object using image analysis, text analysis, and/or image analysis. In addition, it is possible to check the position and size of the content object displayed on the screen, and extract information about the size and position. According to various embodiments, the processor 350 may generate information on a logical connection link between content objects by using semantic unit information extracted from the identified content object, related text information, and/or location information. The logical connection link may mean, for example, context information between each content object, and may mean a content path of a logically expected presentation. According to various embodiments, the context information or the logical connection link information is an order of content objects that are highly related to each other by using information extracted from each content object and using similarity of information included in each content object and proximity of distance. and/or a connection link. According to various embodiments, if the processor 350 determines that the presentation of the present presenter is related to the specified object, the object most relevant to the content of the presentation to be performed immediately after the presentation is the current object and the most relevant object on the logical connection link information. It can be expected to be an object that exists in a nearby link. According to various embodiments, the processor 350 may generate content analysis information including at least one of location information, content information, page information, and page mapping information of a content object. According to various embodiments, the processor 350 may transmit content analysis information to a server device (eg, 220 of FIG. 2 ).

According to various embodiments, the processor 350 may synthesize content information and presenter image information. The processor 350 may generate a synthesized image by synthesizing the content information and the presenter image information. According to various embodiments, the content information to be synthesized may mean a screen on which the content information is executed or displayed.

The processor 350 may divide the content screen into one or more regions based on the content analysis information. The content screen may be all or part of a designated screen for displaying content information. The processor 350 may classify the area of the content screen based on the size and position of the content object recognized in the content information. According to various embodiments, the processor 350 may classify an area for each designated UI object by using information about the layout of the content analysis information. According to various embodiments, the processor 350 may divide a designated partial area including the related content object into one area based on content information of the content object. According to various embodiments, the processor 350 may select at least one area among each area of the divided content screen as an area in which the presenter's image is to be arranged. According to various embodiments, the area to be arranged may be selected based on input information of a user (eg, a presenter) for content information, gesture information of the presenter, and/or voice information of the presenter.

According to various embodiments, the processor 350 may receive voice information of a presenter. For example, audio data included in the presenter's video information may be received as presenter's voice information. Alternatively, voice information input by the presenter may be received using the microphone of the input module 320 . According to various embodiments, the processor 350 may analyze the presenter's voice information to determine the presenter's current presentation. For example, the processor 350 may recognize the speaker's voice information and analyze it for each morpheme or semantic unit, or convert the speaker's voice information into text and perform content analysis using the text information. According to various embodiments, the processor 350 may analyze the presenter's voice information and select a divided and designated area on the content screen into which the presenter's image information is to be merged by using the information on the content of the present presentation. For example, the content analysis information, which is a result of analyzing the content object included in the content information, may include the content information of the content object, and the processor 350 compares the presenter voice information with the content object content information to determine the current progress. It is possible to detect a content object most related to the presentation content. In this case, the processor 350 may arrange the presenter image in a region that does not cover the corresponding content object among the plurality of divided regions of the content screen, and/or arrange the presenter image in an area adjacent to the corresponding content object. .

According to various embodiments, the input information for the content information may be a user input received from the input module 320 . For example, the content input information may be input information for content information, such as mouse manipulation input information and keyboard manipulation input information. When the mouse pointer is positioned at a designated position as a result of the manipulation of the mouse movement on the content, the processor 350 may recognize this and determine that the presentation is in progress with respect to the content object at the corresponding position. In this case, the processor 350 may not obstruct the view of the corresponding content object or may select an area adjacent to the corresponding object as the area in which the presenter's image information is to be placed. According to various embodiments, the content input information may be included in the content analysis information.

According to various embodiments, the presenter's gesture information may be information on a result of analyzing the presenter's behavior. For example, the presenter gesture information may be included in the presenter image analysis information. According to various embodiments, the processor 350 may identify a content object related to the content of a current presentation based on the presenter's gesture information by using the presenter's image analysis information. The gesture information may include information about the presenter's hand gesture, the presenter's gaze, and an action of the presenter pointing to content using a designated tool. According to various embodiments, a region in which the presenter image information is to be arranged may be selected based on information on a logical connection link between content objects included in the content analysis information. The processor 350 determines that, when a presentation proceeds from a content object related to the present presentation content to the next content object, the content most relevant to the presentation content to be presented later based on the information on the logical link link between the content objects objects can be predicted. The processor 350 selects an area most suitable for the predicted content object when selecting an area to be placed in the future, when the current presenter image information is disposed in the designated area and the content of the presentation is predicted to proceed with the content object predicted in advance. You can choose.

According to various embodiments, the processor 350 may synthesize the presenter image by arranging the presenter image in at least one selected area among the divided areas of the content screen. The processor 350 may generate a synthesized image by arranging the presenter image in the selected designated area. According to various embodiments, the processor 350 may transmit the synthesized image to the server device (eg, the server device 220 of FIG. 2 ), and the processor 350 may transmit the synthesized image using the output module 340 . can be printed out.

4 illustrates an embodiment of providing a video conference according to various embodiments.

Referring to FIG. 4 , in the video conference, a case where the presenter image information 411 and the content information 412 independently exist ( 410 ) and a case where the presenter and the content exist in the same image ( 420 ) may occur. When the presenter and the content exist in the same video (420), the entire video information of the presenter making an offline presentation using the presentation material (eg, content) corresponds to the presenter video information and is included in the content information at the same time. may be applicable. An electronic device (eg, the electronic device 300 of FIG. 3 ) or a server device (eg, the server device 220 of FIG. 2 ) analyzes the presenter image information 411 and the content information 412 , respectively, and generates an analysis result An image 430 in which a presenter image and a content screen are synthesized may be generated by using the presented presenter image analysis information and content analysis information. According to various embodiments, when an electronic device (eg, the electronic device 300 of FIG. 3 ) or a server device (eg, the server device 220 of FIG. 2 ) exists in the same image as the presenter ( 420 ), Through the full video information analysis, the presenter video and content information are separated and extracted from the total video information, and the presenter video information and content information analysis is performed, and the presenter video analysis information and content analysis information generated through the analysis are used. An image 430 obtained by synthesizing the presenter image and content information may be generated. The image 430 in which the presenter and content are synthesized may be an image in which the presenter or each component of the content is dynamically arranged in real time. An electronic device (eg, the electronic device 300 of FIG. 3 ) or a server device (eg, the server device 220 of FIG. 2 ) generates the synthesized image 430 with another device (eg, the server device 220 of FIG. 2 ). Alternatively, the data may be transmitted to the external electronic device 230 in real time.

5 is a block diagram for providing a video conference of an electronic device according to various embodiments of the present disclosure;

Referring to FIG. 5 , an example of a module (eg, a function processing module 500 ) for executing a function related to providing a video conference in various embodiments may be illustrated. The function processing module 500 (or the processor 350 of FIG. 3 ) according to various embodiments generates presenter image analysis information by analyzing the presenter image information, and generates content analysis information by analyzing the content information. It is possible to provide a video conferencing service for generating an image in which content information and presenter image information are synthesized by using the present presenter image analysis information and content analysis information.

The function processing module 500 according to various embodiments is included as a hardware module in a processor (eg, the processor 350 of FIG. 3 ) including processing circuitry, or a software module ) may be included.

Referring to FIG. 5 , the function processing module 500 includes an image analysis module 510 (video analyzer module), a content analysis module 520 (contents analyzer module), an arrangement area determination module 530 and an image synthesis module ( 540 , and the placement region determination module 530 may include a composition model decision module 531 and a positioning engine module 532 .

According to various embodiments, the function processing module 500 may receive presenter image information. The presenter image information may be a presenter camera video, but is not limited thereto. When presenter video information is input, the video analysis module 510 may extract a portion to be used for providing a video conference. For example, by analyzing the presenter's image, only the area occupied by the image object that can be recognized as the presenter is extracted from the presenter's image, and the presenter's image information can be edited into an image outputting only the presenter's image. Also, the image analysis module 510 may analyze the presenter image information and generate presenter image analysis information including gesture information of the presenter, voice information of the presenter, and information on whether the presenter speaks.

According to various embodiments, the image analysis module 510 may analyze an image in which the presenter and content exist at the same time, separate and extract the presenter image and content information, and transmit only the content information to the content analysis module 520 . In addition, by analyzing the extracted presenter image, it is possible to extract presenter image analysis information. According to various embodiments, the image analysis module 510 may transmit presentation image analysis information to the arrangement area determining module 530 .

According to various embodiments, the content analysis module 520 may analyze content information. For example, the content and configuration of content information that can be used, output, or executed in the background area during a video conference may be analyzed. The content analysis module 520 inputs content information 521 including background content, presenter script data, background object info, and content input data. can receive The content analysis module 520 may recognize the type of content. For example, the content information may have various properties such as PowerPoint, Word, video player, photo viewer, web browser, and terminal. The content information may be multimedia data that the presenter can use as presentation material. For example, the content information includes text, images, moving pictures, data (eg, data that can be executed as an application executable in the electronic device (eg, the electronic device 300 of FIG. 3 ) or output on the display of the electronic device 300 ). : PowerPoint (Power Point TM) data) and a combination of at least some of them. The content information may be data executed or displayed by the electronic device or multimedia data information received from an external electronic device. According to various embodiments, the content information is the content information transmitted to the content analysis module 520 by the image analysis module 510 analyzing an image in which the presenter and the content exist at the same time, separating and extracting the presenter image and the content information. can According to various embodiments, when the content is image information, the content analysis module 520 may transmit content information corresponding to the image information to the image analysis module 510 .

According to various embodiments, when the content information corresponds to data executable as an application, the content analysis module 520 may analyze execution data of the application and input data for the corresponding application. For example, the content analysis module 520 may analyze a user interface (UI) of the application. According to various embodiments, the content analysis module 520 uses a simulation technique and based on the simulation of the movement of the UI page of the corresponding application, each UI page and/or page on the UI layout such as a pop-up window, connection information for each window can be checked. According to various embodiments, the content analysis module 520 may obtain mapping information between UI pages by combining each movable page and/or window for each UI page and/or window of the corresponding application. According to various embodiments, the content analysis module 520 may analyze a UI component (eg, a UI object) that triggers page movement and generation of a pop-up window within the same UI page. This may be possible by reading control information included in each UI component in the same UI page. You can check the current UI page by combining all UI control information in the same page. Such control information and a combination thereof may be analyzed in real time by the content analysis module 520 , and may be information previously stored in the electronic device 300 . In addition, the content analysis module 520 analyzes data on the object identified in the UI page, and related information of the object identified in the application, for example, each location information where the object is located, attribute information of the object, and content information can be checked.

According to various embodiments, the content analysis module 520 may be included in each content object by using image analysis, text analysis (eg, optical character recognition (OCR)) and/or image analysis for each type of content object classified. It is possible to analyze existing semantic units or extract related text information. The content analysis module 520 may extract content information of each content object using image analysis, text analysis, and/or image analysis. In addition, it is possible to check the position and size of the content object displayed on the screen, and extract information about the size and position. According to various embodiments, the content analysis module 520 may generate information on a logical connection link between content objects by using semantic unit information extracted from the identified content object, related text information, and/or location information. The logical connection link may mean, for example, context information between each content object, and may mean a content path of a logically expected presentation. According to various embodiments, the context information or the logical connection link information is an order of content objects that are highly related to each other by using information extracted from each content object and using similarity of information included in each content object and proximity of distance. and/or a connection link. According to various embodiments, when the content analysis module 520 determines that the presentation of the present presenter is related to the specified object, the object most relevant to the content of the presentation to be performed immediately after the presentation is the current object on the logical connection link information. It can be expected that it is an object that exists in the link closest to . According to various embodiments, the content analysis module 520 may generate content analysis information including at least one of location information, content information, page information, and page mapping information of a content object. According to various embodiments, the content analysis module 520 may transmit the content analysis information to the arrangement area determining module 530 .

According to various embodiments, the placement area determination module 530 may include a composition model decision module 531 (a composition model decision module) and a placement engine module 532 (positioning engine module).

According to various embodiments, the synthesis model determination module 531 included in the arrangement area determination module 530 divides the content screen into at least one area in order to generate a synthesized image by merging the content information and the presenter image information. can According to various embodiments, the placement engine module 532 may select a region in which the presenter image information is to be arranged on each divided region of the content screen.

The arrangement area determining module 530 may divide the content screen into one or more areas based on the content analysis information. The content screen may be all or part of a designated screen for displaying content information. The synthesis model determining module 531 may classify the area of the content screen based on the size and position of the content object recognized in the content information. According to various embodiments, the synthesis model determining module 531 may classify an area for each designated UI object by using information about the layout of the content analysis information. According to various embodiments, the synthesis model determining module 531 may classify a designated partial region including a related content object into one region based on content information of the content object.

According to various embodiments, the placement engine module 532 may select at least one area among each area of the divided content screen as an area in which the presenter's image is to be placed. According to various embodiments, the area to be arranged may be selected based on input information of a user (eg, a presenter) for content information, gesture information of the presenter, and/or voice information of the presenter.

According to various embodiments, the placement engine module 532 may receive presenter behavioral data and/or presenter pointer info 533 . The presenter behavior data and/or the presenter pointer information 533 may include gesture information of the presenter, voice information of the presenter, information on whether the presenter speaks or not, and further may include input data for content. Also, the presenter behavior data and/or the presenter pointer information 533 may be information included in the presenter image analysis information and/or content analysis information.

According to various embodiments, the placement engine module 532 may receive voice information of a presenter. For example, audio data included in the presenter's video information may be received as presenter's voice information. Alternatively, voice information input by the presenter may be received using a microphone of an input module (eg, the input module 320 of FIG. 3 ). According to various embodiments, the placement engine module 532 may analyze the presenter's voice information to determine the presenter's current presentation. For example, the placement engine module 532 may recognize the speaker's voice information and analyze it for each morpheme or semantic unit, or convert the speaker's voice information into text and perform content analysis using the text information. According to various embodiments, the placement engine module 532 may analyze the presenter's voice information, and select a divided, designated area on the content screen into which the presenter's image information is to be merged by using the information on the content of the present presentation. For example, the content analysis information, which is a result of analyzing the content object included in the content information, may include content information of the content object, and the placement engine module 532 compares the presenter voice information with the content object content information to currently It is possible to detect a content object most related to an ongoing presentation. In this case, the placement engine module 532 may arrange the presenter image in a region that does not cover the corresponding content object among the plurality of divided regions of the content screen, and/or arranges the presenter image in an area adjacent to the corresponding content object. can

According to various embodiments, the input information for the content information may be a user input received from the input module 320 . For example, the content input information may be input information for content information, such as mouse manipulation input information and keyboard manipulation input information. When the mouse pointer is positioned at a specified position as a result of manipulation of the mouse movement on the content, the placement engine module 532 may recognize this and determine that the presentation of the content object at the corresponding position is in progress. . In this case, the placement engine module 532 may not obstruct the view of the corresponding content object or may select an area adjacent to the corresponding object. According to various embodiments, the content input information may be included in the content analysis information.

According to various embodiments, the presenter's gesture information may be information on a result of analyzing the presenter's behavior. For example, the presenter gesture information may be included in the presenter image analysis information. According to various embodiments, the placement engine module 532 may use the presenter image analysis information to identify a content object related to the content of the present presentation based on the presenter's gesture information. The gesture information may include information about the presenter's hand gesture, the presenter's gaze, and an action of the presenter pointing to content using a designated tool. According to various embodiments, a region in which the presenter image information is to be arranged may be selected based on information on a logical connection link between content objects included in the content analysis information. The placement engine module 532 is the most relevant to the presentation to be made in the future based on the information on the logical connection link between the content objects when the presentation proceeds from the content object related to the present content object to the next content object. High content objects can be predicted. The placement engine module 532 is the most suitable for the predicted content object when selecting an area to be placed in the future, when the current presenter image information is disposed in the designated area and the content of the presentation is predicted to proceed with the content object predicted in advance. You can select an area.

According to various embodiments, the image synthesis module 540 may synthesize the presenter image by arranging the presenter image in at least one selected region among the divided regions of the content screen. The image synthesis module 540 may generate a synthesized image by arranging the presenter image in the selected designated area. According to various embodiments, the image synthesized by the image synthesis module 540 is generated using another component (eg, the communication module 310 of FIG. 3 ) of the electronic device (eg, the electronic device 300 of FIG. 3 ). may be transmitted to a server device (eg, the server device 220 of FIG. 2 ), and the electronic device (eg, the electronic device 300 of FIG. 3 ) transmits an output module (eg, the output module 340 of FIG. 3 ). can be used to output the synthesized image.

6 is a flowchart of a method of providing a video conference by an electronic device according to various embodiments of the present disclosure;

Referring to FIG. 6 , a series of operations according to each step of FIG. 6 may consist of operations of the processor 350 of an electronic device (eg, the electronic device 300 of FIG. 3 ). Alternatively, the series of operations of FIG. 6 may be performed through a server device (eg, the server device 220 of FIG. 2 ). In this document, the operation of the processor of the electronic device is described for convenience, but the subject of the operation is not limited to the electronic device.

According to various embodiments, in operation 610 , the electronic device 300 may analyze image information. According to various embodiments, the electronic device 300 may analyze image information and determine whether a presenter and content exist simultaneously in the same image. For example, when the presenter and the content exist in the same image, the corresponding image information may be information obtained by photographing an image in which the presenter makes a presentation with the content as a background offline. The electronic device 300 may recognize an object through analysis of received or photographed image information, classify the recognized object, and determine what kind of image information the image information is based on the object classification. For example, if the main object recognized on the image information is a person, it can be recognized as a presenter image, and when a person and a designated object that can be classified in presentation materials (eg, content information) are recognized together, the image is displayed as a presenter It can be determined that the video is taken with the content as a background.

Referring to operation 620 , the electronic device 300 may analyze entire image information. The entire image may be an image captured with the presenter and the content as a background. The electronic device 300 may recognize an object through analysis of image information, classify the recognized object, and calculate an image boundary of the recognized object based on the object classification. The electronic device 300 may calculate an image boundary by using pixel information of an image object, extract position information of each object based on the image boundary, and extract only pixel information within the image boundary as image information or presenter image You can edit the information. For example, based on the boundary formed by the image object recognized as the presenter, only the image portion of the presenter shape within the entire image may be cut out to edit the presenter image information. According to various embodiments, the electronic device 300 may separate content existing in an image. For example, the content existing in the image may be a designated flat object, such as a screen or a blackboard, and the electronic device 300 may recognize it as a designated visual object through image information analysis.

Referring to operation 630 , when it is determined that the presenter and the content exist in the same image, the electronic device 300 may separate and extract the presenter image from the image information. The electronic device 300 may extract image information of the presenter based on an image boundary of a portion recognized as a presenter using image extraction technology, and may analyze the presenter image information using the extracted presenter image information . According to various embodiments, the electronic device 300 may separate content existing in an image. For example, the content existing in the image may be a designated flat object, such as a screen or a blackboard, and the electronic device 300 may recognize it as a designated visual object through image information analysis. The electronic device 300 may separate one or more objects recognized as content through object classification, extract it as an image, and generate as one content image. The electronic device 300 may generate and extract a content image extracted from image information as content information.

Referring to operation 640 , the electronic device 300 may analyze presenter image information. According to various embodiments, operations 640 and 650 may not necessarily be sequentially performed. For example, operation 640 and operation 650 may be performed simultaneously in parallel, or the order may be changed so that operation 640 after operation 650 may be performed. According to various embodiments, the electronic device 300 may analyze image information. The image information may be data captured by the electronic device 300 or image information received from an external electronic device. For example, the presenter image information may be information received by the electronic device 300 from an external device (eg, the server device 220 of FIG. 2 ) using the communication module 310 or the input module ( 320) may be data taken. According to various embodiments, the electronic device 300 may use an artificial intelligence algorithm to analyze a presenter's image, for example, store an algorithm learned using a neural network (eg, ANN, CNN) in advance, or a server It is also possible to receive algorithm data from

According to various embodiments, if it is determined in operation 610 that the content and the presenter do not exist in the same image at the same time, the process proceeds to operation 640 or operation 650 depending on whether the analyzed entire image is the presenter image or the content screen. can proceed. According to various embodiments, when it is determined that only the content exists among the cases where it is determined that the content and the presenter do not exist simultaneously in the same image in operation 610, the presenter image information analysis may not be necessary. In this case, the electronic device 300 may directly perform operation 650 in operation 610 without going through operation 640 .

According to various embodiments, the electronic device 300 may extract an image from presenter image information and perform image analysis. According to various embodiments, the electronic device 300 may analyze an image included in the presenter image information to recognize an object included in the image. The object included in the image of the video information may be, for example, a visually identifiable object such as an image of a person (eg, a presenter) and/or an image designated as presentation material. According to various embodiments, when the content and the presenter do not exist simultaneously in the same image in operation 610, only the presenter image may be an identifiable object. The electronic device 300 may classify for each object category included in the image, and may perform an analysis on each object based on the classification or perform a determination for executing a specified operation. For example, the processor may classify the object recognized in the image according to object properties of a person, an image, a text, a screen, and a blackboard. The electronic device 300 may select the classified object. The selection operation may be an operation for reducing the complexity of image analysis. For example, the electronic device 300 may select only the objects required for analysis based on the size and frequency of appearance of the classified designated object.

When an object that can be classified as a person is recognized in the presenter image information, the electronic device 300 may analyze gesture information and/or facial feature points. For example, the electronic device 300 may recognize a change in the recognized image object, and analyze the gesture information using pre-stored data, data received from a server, or an algorithm. According to various embodiments, the electronic device 300 may analyze the presenter's gesture through presenter image information analysis and generate gesture information. According to various embodiments, the electronic device 300 may analyze a feature point of a face included in an image, and through this, may identify the presenter's expression information, the presenter's gaze information, and/or whether the presenter is speaking . According to various embodiments, the electronic device 300 may calculate an image boundary of the recognized object. The electronic device 300 may calculate an image boundary by using pixel information of an image object, extract position information of each object based on the image boundary, and extract only pixel information within the image boundary as image information or presenter image You can edit the information. For example, based on the boundary formed by the image object recognized as the presenter, only the image portion of the presenter shape in the presenter image may be cut out to edit the presenter image information. According to various embodiments of the present disclosure, the electronic device 300 analyzes a presenter image including information on whether the presenter speaks, gesture information, gaze information, image boundary information, and presenter image information edited based on the image boundary through analysis of the presenter image. information can be created.

Referring to operation 650 , the electronic device 300 may analyze content information. The content information may be multimedia data that the presenter can use as presentation material. For example, the content information includes text, image, video, and data (eg, Power Point TM) data that can be executed as an application executable in the electronic device 300 or output on the display of the electronic device 300 . ) and a combination of at least some of them. The content information may be data executed or displayed by the electronic device 300 or multimedia data information received from an external electronic device. The content information may be information received by the electronic device 300 from an external device (eg, the server device 220 of FIG. 2 ) using the communication module 310 , or executed by the electronic device 300 . or data that is executed and output by the output module 350 . Alternatively, the content information may be image data in which the electronic device 300 extracts only an image region identified as content from other image information.

According to various embodiments, the electronic device 300 may determine the type of content information. For example, the type of application required to execute or display content information in the electronic device 300 may be checked. Alternatively, the type of content information may be determined through the type of an application for displaying content information that is already being executed. The type of content information may be divided into a case of video, image, text, or a combination thereof and other applications. A content analysis method may be different based on the determined type of content information.

According to various embodiments, when the content information includes text, an image, a moving picture, or a combination thereof, the electronic device 300 may analyze the text, image, and/or video. For example, the content information may be in the form of including a plurality of multimedia contents (eg, images and texts) in the same output screen. In this case, the electronic device 300 may recognize each content as a content object. The electronic device 300 may identify each recognized content object, and classify and select the identified object. For example, the identified content objects may be classified according to content types, such as images, texts, and images, respectively.

According to various embodiments, when the content information corresponds to data executable as an application other than the above-described type, the electronic device 300 may analyze execution data of the application and input data of the corresponding application. For example, the electronic device 300 may analyze a user interface (UI) of the application. According to various embodiments, the electronic device 300 uses a simulation technique to provide each UI page and/or connection information for each window on a UI layout such as a pop-up window based on a simulation of moving a UI page of a corresponding application. Alternatively, such information may be previously stored in another component (eg, the memory 330 ) of the electronic device. According to various embodiments, the electronic device 300 may obtain mapping information between UI pages by combining each movable page and/or window for each UI page and/or window of the corresponding application. According to various embodiments, the electronic device 300 may analyze a UI component (eg, a UI object) that triggers page movement and generation of a pop-up window within the same UI page. This may be possible by reading control information included in each UI component in the same UI page. You can check the current UI page by combining all UI control information in the same page. Such control information and a combination thereof may be analyzed in real time by the electronic device 300 , and may be information previously stored in the electronic device 300 . Also, the electronic device 300 may check information on an object identified in the UI page.

According to various embodiments, the electronic device 300 may be included in each content object by using image analysis, text analysis (eg, optical character recognition (OCR)) and/or image analysis for each type of content object classified. A semantic unit may be analyzed or related text information may be extracted. The electronic device 300 may extract content information of each content object using image analysis, text analysis, and/or image analysis. In addition, it is possible to check the position and size of the content object displayed on the screen, and extract information about the size and position. According to various embodiments, the electronic device 300 may generate information on a logical connection link between content objects by using semantic unit information extracted from the identified content object, related text information, and/or location information. The logical connection link may mean, for example, context information between each content object, and may mean a content path of a logically expected presentation. According to various embodiments, the context information or the logical connection link information is an order of content objects that are highly related to each other by using information extracted from each content object and using similarity of information included in each content object and proximity of distance. and/or a connection link. According to various embodiments, when the electronic device 300 determines that the presenter's presentation is related to the specified object, the object most relevant to the content of the presentation to be performed immediately after the corresponding presentation is the current object and the current object on the logical connection link information. It can be expected that the object exists in the nearest link. According to various embodiments, the electronic device 300 may generate content analysis information including at least one of location information, content information, page information, and page mapping information of a content object. According to various embodiments, the electronic device 300 may transmit content analysis information to a server device (eg, 220 of FIG. 2 ) through the communication module 310 .

Referring to operation 660 , the electronic device 300 may synthesize content information and presenter image information. The electronic device 300 may generate a synthesized image by synthesizing content information and presenter image information. According to various embodiments, the content information to be synthesized may mean a screen on which the content information is executed or displayed.

The electronic device 300 may divide the content screen into one or more regions based on the content analysis information. The content screen may be all or part of a designated screen for displaying content information. The electronic device 300 may classify the area of the content screen based on the size and position of the content object recognized in the content information. According to various embodiments, the electronic device 300 may classify an area for each designated UI object by using information about the layout of the content analysis information. According to various embodiments, the electronic device 300 may classify a designated partial area including a related content object into one area based on content information of the content object. According to various embodiments, the electronic device 300 may select at least one area among each area of the divided content screen as an area in which the presenter's image is to be arranged. According to various embodiments, the area to be arranged may be selected based on input information of a user (eg, a presenter) for content information, gesture information of the presenter, and/or voice information of the presenter.

According to various embodiments, the electronic device 300 may receive voice information of a presenter. For example, audio data included in the presenter's video information may be received as presenter's voice information. Alternatively, voice information input by the presenter may be received using the microphone of the input module 320 . According to various embodiments, the electronic device 300 may analyze the presenter's voice information to identify the presenter's current presentation. For example, the electronic device 300 may recognize the speaker's voice information and analyze it for each morpheme or semantic unit, or convert the speaker's voice information into text and perform content analysis using the text information. According to various embodiments, the electronic device 300 may analyze the presenter's voice information and select a divided and designated area on the content screen into which the presenter's image information is to be merged by using the information on the currently ongoing presentation. For example, the content analysis information, which is a result of analyzing the content object included in the content information, may include content information of the content object, and the electronic device 300 compares the presenter's voice information with the content object content information to proceed with the current process. It is possible to detect a content object most related to the current presentation. In this case, the electronic device 300 may arrange the presenter image in a region that does not cover the corresponding content object among the plurality of divided regions of the content screen, and/or arrange the presenter image in an area adjacent to the corresponding content object. there is.

According to various embodiments, the input information for the content information may be a user input received from the input module 320 . For example, the content input information may be input information for content information, such as mouse manipulation input information and keyboard manipulation input information. When the mouse pointer is positioned at a designated location as a result of the manipulation of the mouse movement on the content, the electronic device 300 may recognize this and determine that the presentation of the content object at the corresponding location is in progress. In this case, the electronic device 300 may not obstruct the view of the corresponding content object or may select an area adjacent to the corresponding object. According to various embodiments, the content input information may be included in the content analysis information.

According to various embodiments, the presenter's gesture information may be information on a result of analyzing the presenter's behavior. For example, the presenter gesture information may be included in the presenter image analysis information. According to various embodiments, the electronic device 300 may use the presenter's image analysis information to identify a content object related to the content of the present presentation based on the presenter's gesture information. The gesture information may include information about the presenter's hand gesture, the presenter's gaze, and an action of the presenter pointing to content using a designated tool. According to various embodiments, a region in which the presenter image information is to be arranged may be selected based on information on a logical connection link between content objects included in the content analysis information. The electronic device 300 determines that, when a presentation proceeds from a content object related to the present presentation content to the next content object, the electronic device 300 has the highest relevance to the presentation content to be presented later based on the information on the logical connection link between the content objects. Content objects can be predicted. When the current presenter image information is arranged in a specific region and the content of the presentation is predicted to proceed with the content object predicted in advance, the electronic device 300 selects the region to be arranged in the future, the region most suitable for the predicted content object. can be selected.

According to various embodiments, the electronic device 300 may synthesize the presenter image by arranging the presenter image in at least one selected area among the divided areas of the content screen. The electronic device 300 may generate a synthesized image by arranging the presenter image in the selected designated area. According to various embodiments, the electronic device 300 may transmit an image synthesized to the server device (eg, the server device 220 of FIG. 2 ) through the communication module 310 , and the electronic device 300 may transmit an output module ( 340) to output a synthesized image.

7 is a flowchart of a method of providing a video conference by an electronic device and a server according to various embodiments of the present disclosure;

According to various embodiments, the subject providing the video conferencing service may include the electronic device 710 and the server 720 . A video conference service may be implemented through a series of operations of the electronic device 710 (eg, the electronic device 300 of FIG. 3 ) and the server 720 (eg, the server device 220 of FIG. 2 ). In the case of a remote video conference, it may be provided using an end-to-end encryption (E2EE) method. In this case, the server (eg, the server 720) is limited to mediating encrypted data transmission between the client (eg, the electronic device 710) and another client (eg, the external electronic device 730), and the client's Transmission and reception data including video and/or audio cannot be decoded. Accordingly, the subject performing the function of the present invention may be changed according to the presence or absence of a secure connection through E2EE. In the case of the embodiment shown in FIG. 7 , the description is focused on a case in which the E2EE method is not used, but the present invention is not limited thereto.

Referring to FIG. 7 , operations 701 and 703 may not necessarily be sequentially performed. For example, operation 701 and operation 703 may be performed simultaneously in parallel, or the order may be changed so that operation 701 subsequent to operation 703 may be performed. In operation 701, the electronic device 710 may analyze presenter image information. According to various embodiments, the electronic device 710 may extract an image from presenter image information and perform image analysis. According to various embodiments, the electronic device 710 may analyze an image included in the presenter image information to recognize an object included in the image. The object included in the image of the video information may be, for example, a visually identifiable object such as an image of a person (eg, a presenter) or an image designated as presentation material. The electronic device 710 may classify for each object category included in the image, and may perform an analysis on each object based on the classification or perform a determination for executing a specified operation. For example, the electronic device 710 may classify an object recognized in an image according to object properties such as a person, an image, a text, a screen, and a blackboard. The electronic device 710 may select the classified object. The selection operation may be an operation for reducing the complexity of image analysis. For example, the electronic device 710 may select only the objects required for analysis based on the size and frequency of appearance of the classified designated object.

When an object that can be classified as a person is recognized in the presenter image information, the electronic device 710 may analyze gesture information and/or facial feature points. For example, the electronic device 710 may recognize a change in the recognized image object, and analyze the gesture information using pre-stored data, data received from a server, or an algorithm. According to various embodiments, the electronic device 710 may analyze the presenter's gesture through presenter image information analysis and generate gesture information. According to various embodiments, the electronic device 710 may analyze a feature point of a face included in an image, and through this, may identify the presenter's expression information, the presenter's gaze information, and/or whether the presenter is speaking . According to various embodiments, the electronic device 710 may calculate an image boundary of the recognized object. The electronic device 710 may calculate an image boundary by using pixel information of an image object, extract position information of each object based on the image boundary, and extract only pixel information within the image boundary as image information or presenter image You can edit the information. For example, based on the boundary formed by the image object recognized as the presenter, only the image portion of the presenter shape in the presenter image may be cut out to edit the presenter image information. According to various embodiments, the electronic device 710 analyzes a presenter image including information on whether the presenter speaks, gesture information, gaze information, image boundary information, and presenter image information edited based on the image boundary through analysis of the presenter image. information can be created.

In operation 703, the electronic device 710 may analyze content information. According to various embodiments, the electronic device 710 may determine the type of content information. For example, the type of application required to execute or display content information in the electronic device 710 may be checked. Alternatively, the type of content information may be determined through the type of an application for displaying content information that is already being executed. The type of content information may be divided into a case of video, image, text, or a combination thereof and other applications. According to various embodiments, when the content information includes text, an image, a moving picture, or a combination thereof, the electronic device 710 may analyze the text, the image, and/or the image. For example, the content information may be in the form of including a plurality of multimedia contents (eg, images and texts) in the same output screen. In this case, the electronic device 710 may recognize each content as a content object. The electronic device 710 may identify each recognized content object, and classify and select the identified object. For example, the identified content objects may be classified according to content types, such as images, texts, and images, respectively.

According to various embodiments, when the content information corresponds to data executable as an application other than the above type, the electronic device 710 may analyze execution data of the application and input data for the corresponding application. For example, the electronic device 710 may analyze a user interface (UI) of the application. According to various embodiments, the electronic device 710 uses a simulation technique to display connection information for each UI page and/or page on a UI layout such as a pop-up window, based on a simulation of moving a UI page of a corresponding application, and for each window. Alternatively, such information may be previously stored in another component (eg, the memory 330 ) of the electronic device. According to various embodiments, the electronic device 710 may obtain mapping information between UI pages by combining each movable page and/or window for each UI page and/or window of the corresponding application. According to various embodiments, the electronic device 710 may analyze a UI component (eg, a UI object) that triggers page movement and generation of a pop-up window within the same UI page. This may be possible by reading control information included in each UI component in the same UI page. You can check the current UI page by combining all UI control information in the same page. Such control information and a combination thereof may be analyzed in real time by the electronic device 710 and may be information previously stored in the electronic device 710 . Also, the electronic device 710 may check information on an object identified in the UI page.

According to various embodiments, the electronic device 710 may be included in each content object by using image analysis, text analysis (eg, optical character recognition (OCR)) and/or image analysis for each type of content object classified. A semantic unit may be analyzed or related text information may be extracted. The electronic device 710 may extract content information of each content object using image analysis, text analysis, and/or image analysis. In addition, it is possible to check the position and size of the content object displayed on the screen, and extract information about the size and position. According to various embodiments, the electronic device 710 may generate information on a logical connection link between content objects by using semantic unit information extracted from the identified content object, related text information, and/or location information. The logical connection link may mean, for example, context information between each content object, and may mean a content path of a logically expected presentation. According to various embodiments, the context information or the logical connection link information is an order of content objects that are highly related to each other by using information extracted from each content object and using similarity of information included in each content object and proximity of distance. and/or a connection link. According to various embodiments, when the electronic device 710 determines that the presenter's presentation is related to the specified object, the object most relevant to the content of the presentation to be performed immediately after the corresponding presentation is the current object and the current object on the logical connection link information. It can be expected that the object exists in the nearest link. According to various embodiments, the electronic device 710 may generate content analysis information including at least one of location information, content information, page information, and page mapping information of a content object.

Referring to operation 705 , the electronic device 710 may transmit a presenter image and presenter image analysis information to the server 720 . According to various embodiments, operations 705 and 707 may not necessarily be sequentially performed. For example, operation 705 and operation 707 may be performed simultaneously in parallel, or the order may be changed so that operation 705 subsequent to operation 707 may be performed. According to various embodiments, the presenter image analysis information may be various data for synthesizing the presenter image information and content information. The electronic device 710 according to various embodiments provides presenter image analysis information including information on whether the presenter speaks, gesture information, gaze information, image boundary information, and presenter image information edited based on the image boundary generated through image analysis. can create The gesture information of the presenter may be information about a result of analyzing the presenter's behavior. For example, the presenter gesture information may be included in the presenter image analysis information. According to various embodiments of the present disclosure, the server 720 may receive the presenter image analysis information from the electronic device 710 and may identify a content object related to the content of the present presentation based on the presenter's gesture information. The gesture information may include information about a gesture of the presenter, the presenter's gaze, or an operation of the presenter pointing to content using a designated tool. According to various embodiments, the server 720 may receive a presenter image and presenter image analysis information from the external electronic device 730 (eg, the external electronic device 230 of FIG. 2 ).

Referring to operation 707 , the electronic device 710 may transmit content information and content analysis information to the server 720 . The content analysis information may be various data for synthesizing presenter image information and content information. The content information may include, for example, at least one of location information of a content object, content content information, page information, page mapping information, script data, and background object information. In addition, the content object may include information about the position and size displayed on the screen. The content content information may include information on a logical connection link between content objects using semantic unit information extracted from the identified content object, related text information, and/or location information. The logical connection link may mean, for example, context information between each content object, and may mean a content path of a logically expected presentation. According to various embodiments, the context information or the logical connection link information may include content with high relevance to each other using information extracted from each content object, similarity of information included in each content object, and/or proximity of distance. It may mean an order of objects and/or a connection link. According to various embodiments, the content analysis information may include content input information. For example, the content input information may be input information on the content information, such as mouse manipulation input information and keyboard manipulation input information of the electronic device 710 . When the mouse pointer is positioned at a designated location as a result of a manipulation of the mouse movement on the content, it may be data recognized by the electronic device.

According to various embodiments, operations 709, 711, and 713 may not necessarily be sequentially performed. For example, operations 709, 711, and 713 may be simultaneously performed in parallel, or the order of operations 709, 711, and 713 may be changed and performed. Referring to operation 709, the server may divide the content screen into one or more areas. The server 720 may divide the content screen into one or more regions based on the content analysis information. The content screen may be all or part of a designated screen for displaying content information. The server 720 may classify the area of the content screen based on the size and position of the content object recognized in the content information. According to various embodiments, the server 720 may classify an area for each designated UI object by using information about the layout of the content analysis information. According to various embodiments, the server 720 may classify a designated partial area including the related content object into one area based on the content information of the content object.

Referring to operation 711 , the server 720 may select an area to be placed among the divided areas. According to various embodiments, the server 720 may select at least one area among each area of the divided content screen as an area in which the presenter's image is to be arranged. According to various embodiments, the area to be arranged may be selected based on input information of a user (eg, a presenter) for content information, gesture information of the presenter, and/or voice information of the presenter.

According to various embodiments, the server 720 may receive voice information of a presenter. For example, audio data included in the presenter's video information may be received as presenter's voice information. Alternatively, voice information input by the presenter may be received from the electronic device 710 . According to various embodiments, the server 720 may analyze the presenter's voice information to determine the presenter's current presentation. For example, the server 720 may recognize the speaker's voice information and analyze it for each morpheme or semantic unit, or convert the speaker's voice information into text and perform content analysis using the text information. According to various embodiments, the server 720 may analyze the presenter's voice information and select a divided and designated area on the content screen into which the presenter's image information is to be merged by using the information on the content of the present presentation. For example, the content analysis information, which is a result of analyzing the content object included in the content information, may include the content information of the content object, and the server 720 compares the presenter voice information with the content object content information, and It is possible to detect a content object most related to the presentation content. In this case, the server 720 may arrange the presenter image in a region that does not cover the corresponding content object among the plurality of divided regions of the content screen, and/or arrange the presenter image in an area adjacent to the corresponding content object. .

According to various embodiments, the input information for the content information may be a user input received from the electronic device 710 or may be data included in the content analysis information. The content input information may be, for example, input information for content information, such as mouse manipulation input information or keyboard manipulation input information. When the mouse pointer is positioned at a designated location as a result of the manipulation of the mouse movement on the content, the server 720 receives this information from the electronic device 710 and announces the content about the content object at the location can be considered to be in progress. In this case, the server 720 may not obstruct the view of the corresponding content object or may select an area adjacent to the corresponding object. According to various embodiments, the content input information may be included in the content analysis information.

According to various embodiments, the presenter's gesture information may be information on a result of analyzing the presenter's behavior. For example, the presenter gesture information may be included in the presenter image analysis information. According to various embodiments, the server 720 may use the presenter image analysis information to identify a content object related to the content of the present presentation based on the presenter's gesture information. The gesture information may include information about the presenter's hand gesture, the presenter's gaze, and/or the presenter's operation of pointing to content using a designated tool. According to various embodiments, a region in which the presenter image information is to be arranged may be selected based on information on a logical connection link between content objects included in the content analysis information. The server 720 stores the content most relevant to the presentation to be made in the future based on the information on the logical link link between the content objects when the presentation proceeds from the content object related to the present content object to the next content object. objects can be predicted. The server 720 selects an area most suitable for the predicted content object when selecting an area to be placed in the future when the presenter image information is disposed in the designated area and the content of the presentation is predicted to proceed with the content object predicted in advance. You can choose.

Referring to operation 713 , the server 720 may edit presenter image information. The server 720 may calculate the boundary of the image by using the pixel information of the image object, extract position information of each object based on the boundary of the image, and extract only pixel information within the boundary of the image as image information or presenter image information can be edited. For example, based on the boundary formed by the image object recognized as the presenter, only the image portion of the presenter shape in the presenter image may be cut out to edit the presenter image information. According to various embodiments, the server 720 is a presenter including the presenter's utterance information, gesture information, gaze information, image boundary information, and/or presenter image information edited based on the image boundary through analysis of the presenter image. Image analysis information can be generated. According to various embodiments of the present disclosure, the server 720 transmits information about the presenter image information of the presenter currently speaking from a certain device using the presenter image analysis information received from the electronic device 710 and/or the external electronic device 730 . recognition may be determined, and the presenter's image information may be replaced or changed with the presenter's image information while speaking.

Referring to operation 715 , the server 720 may synthesize a presenter image and a content screen.

According to various embodiments, the server 720 may synthesize content information and presenter image information. The server 720 may generate a synthesized image by synthesizing the content information and the presenter image information. According to various embodiments, the content information to be synthesized may mean a screen on which the content information is executed or displayed. According to various embodiments, the server 720 may synthesize the presenter image by arranging the presenter image in at least one selected area among the divided areas of the content screen. The server 720 may generate a synthesized image by arranging the presenter image in the selected designated area. According to various embodiments, the presenter image may be the presenter image edited in operation 713 .

Referring to operation 717 , the server 720 may transmit streaming information to another electronic device (eg, the electronic device 710 and/or the external electronic device 730 ). The streaming information may include an image in which content information and presenter image information are synthesized. The electronic device 710 and/or the external electronic device 730 that has received the streaming information may output the streaming information received from the server 720 in operation 719 . According to various embodiments, the electronic device 710 may output the composite image information included in the streaming information to an output module of the electronic device (eg, the output module 340 of FIG. 3 ).

8 illustrates an embodiment of analyzing a presenter image according to various embodiments of the present disclosure.

Referring to FIG. 8 , an electronic device (eg, the electronic device 300 of FIG. 3 ) may analyze presenter image information. According to various embodiments, the electronic device 300 may analyze and edit the presenter's image. According to various embodiments, there may be a plurality of video conference participants, and a plurality of presenter image information corresponding to each participant may also be present. When there are a plurality of presenter image information ( 810 ), the electronic device (eg, the electronic device 300 of FIG. 3 ) displays a plurality of presenter images (eg, first presenter image information 811 , second presenter image information 812 ). , the third presenter image information 813 and the fourth presenter image information 814) may be received. The electronic device 300 may analyze the received plurality of presenter image information, respectively, and may generate presenter image analysis information for each presenter image information. When an object that can be classified as a person is recognized in the presenter image information, the electronic device 300 may analyze gesture information and/or facial feature points. For example, the electronic device 300 may recognize a change in the recognized image object, and analyze the gesture information using pre-stored data, data received from a server, or an algorithm. According to various embodiments, the electronic device 300 may analyze the presenter's gesture through presenter image information analysis and generate gesture information. According to various embodiments, the electronic device 300 may analyze a feature point of a face included in an image, and through this, it is possible to identify the presenter's expression information, the presenter's gaze information, and/or whether the presenter is speaking. . According to various embodiments, as a result of analyzing a plurality of presenter images, the presenter image information 815 of the presenter who is speaking may be selected. According to various embodiments, the electronic device 300 may receive voice information of a presenter. For example, audio data included in the presenter's video information may be received as presenter's voice information. Alternatively, voice information input by the presenter may be received using the microphone of the input module 320 . According to various embodiments, the electronic device 300 may analyze the presenter's voice information to identify the presenter's current presentation. For example, the electronic device 300 may recognize the speaker's voice information and analyze it for each morpheme or semantic unit, or convert the speaker's voice information into text and perform content analysis using the text information. According to various embodiments, the electronic device 300 may analyze the presenter's voice information and select the presenter's image information 815 of the presenter who is speaking.

According to various embodiments, when the presenter image information 815 or the presenter image information determined as the speaker is one 820, the electronic device 300 displays an object that can be classified as a person in the presenter image information 821 ( 822) is recognized, gesture information and/or facial feature points may be analyzed. According to various embodiments, the electronic device 300 may analyze the presenter's gesture through presenter image information analysis and generate gesture information. According to various embodiments, the electronic device 300 may analyze the feature point of the object 822 recognized as a face included in the image, and through this, facial expression information of the presenter, gaze information of the presenter, and/or the presenter's utterance It can be identified whether or not According to various embodiments, the electronic device 300 may calculate an image boundary of the recognized object. The electronic device 300 may calculate an image boundary by using pixel information of an image object, extract position information of each object based on the image boundary, and extract only pixel information within the image boundary as image information or presenter image You can edit the information. For example, based on a boundary formed by an image object recognized as a presenter, only the image portion 824 of the presenter shape in the presenter image may be cut out to edit the presenter image information. According to various embodiments, the electronic device 300 performs presenter image information 823 edited based on the presenter's utterance information, gesture information, gaze information, image boundary information, and/or image boundary through analysis of the presenter's image. It is possible to generate presenter image analysis information including

9 is a flowchart of a method for an electronic device to analyze a presenter image according to various embodiments of the present disclosure;

Referring to FIG. 9 , a series of operations for analyzing presenter image information may be performed by an electronic device (eg, the electronic device 300 of FIG. 3 or the server device 220 of FIG. 2 ).

Referring to operation 910 , the electronic device may identify, classify, and select an object in an image. According to various embodiments, the electronic device 300 may analyze image information. The image information may be, for example, image information corresponding to presenter image information and/or content information. The image information may be data captured by the electronic device 300 or image information received from an external electronic device. For example, the image information may be information received by the electronic device 300 from an external device (eg, the server device 220 of FIG. 2 ) using the communication module 310 , or the input module 320 . ) may be data captured using According to various embodiments, the electronic device 300 may use an artificial intelligence algorithm to analyze a presenter's image, for example, store an algorithm learned using a neural network (eg, ANN, CNN) in advance, or a server It is also possible to receive algorithm data from

According to various embodiments, the electronic device 300 may extract an image from image information and perform image analysis. According to various embodiments, the electronic device 300 may analyze an image included in the image information to identify at least one object included in the image. The at least one object included in the image of the video information may be, for example, an image of a person (eg, a presenter), an image designated as presentation material, and/or a visually identifiable object such as text. The electronic device 300 may classify at least one object included in the image for each category, and may perform an analysis on each object based on the classification or perform a determination to execute a specified operation. For example, the electronic device 300 may classify an object recognized in an image according to object properties such as a person, an image, a text, a screen, and a blackboard. The electronic device 300 may select the classified object. The selection operation may be an operation for reducing the complexity of image analysis. For example, the electronic device 300 may select only the objects required for analysis based on the size and/or appearance frequency of the classified designated object.

Referring to operation 920 , the electronic device 300 may analyze image information and determine whether a human object (eg, a presenter) and a content object exist simultaneously in the same image. For example, when the presenter and the content exist in the same image, the corresponding image information may be information obtained by photographing an image in which the presenter makes a presentation with the content as a background offline. The electronic device 300 may recognize an object through analysis of received or photographed image information, classify the recognized object, and determine what kind of image information the image information is based on the object classification. For example, if the main object recognized on the image information is a person, it can be recognized as a presenter image, and a designated object (eg, content object) that can be classified into a person and presentation material (eg, content information) is recognized together. In this case, it may be determined that the video is a video shot with the presenter as the background of the content.

Referring to operation 930 , when it is determined that the presenter and the content exist in the same image (operation 920 - Yes), the electronic device 300 may separate and extract the presenter image from the image information. The electronic device 300 may extract image information of the presenter based on an image boundary of a portion recognized as a presenter using image extraction technology, and may analyze the presenter image information using the extracted presenter image information . According to various embodiments, the electronic device 300 may extract content information existing in an image. For example, the content present in the image may be a designated flat object such as a screen and/or a blackboard, and the electronic device 300 may recognize it as a designated visual object through image information analysis. The electronic device 300 may separate one or more objects recognized as content through object classification, extract it as an image, and generate as one content image. The electronic device 300 may generate and extract a content image extracted from image information as content information.

Referring to operation 940 , if the electronic device 300 determines that the presenter and the content do not exist in the same image (operation 920 - NO), the electronic device 300 may determine whether a human object exists. The electronic device 300 may recognize, classify, and select an image object in the presenter's image information. When an object that can be classified as a person is recognized in the presenter image information, the electronic device 300 may analyze gesture information and/or facial feature points (operation 950).

Referring to operation 950, when an object that can be classified as a person is recognized in the presenter image information (operation 940 - Yes), the electronic device 300 may analyze the gesture information and Feature points can be analyzed. The electronic device 300 may recognize a change in the recognized image object, and analyze the gesture information using pre-stored data, data received from a server, or an algorithm. According to various embodiments, the electronic device 300 may analyze the presenter's gesture through presenter image information analysis and generate gesture information. According to various embodiments, the electronic device 300 may analyze a feature point of a face included in an image, and through this, may identify the presenter's expression information, the presenter's gaze information, and/or whether the presenter is speaking . According to various embodiments, the electronic device 300 may calculate an image boundary of the recognized object.

Referring to operation 960 , the electronic device 300 may calculate the boundary of the image by using pixel information of the image object. The electronic device 300 may extract position information of each object based on the image boundary, extract only pixel information within the image boundary as image information, or edit the presenter image information. For example, based on the boundary formed by the image object recognized as the presenter, only the image portion of the presenter shape in the presenter image may be cut out to edit the presenter image information.

Referring to operation 970 , the electronic device 300 analyzes the presenter's image to analyze the presenter's speech information, gesture information, gaze information, image boundary information, and/or presenter image information edited based on the image boundary. Image analysis information can be generated. According to various embodiments, when it is determined that the human object does not exist in operation 940, it may be a content image other than a presenter image. In this case, the operation of generating the content analysis information without generating the presenter image analysis information may be performed, which will be described later with reference to FIG. 14 .

10, 11, 12, and 13 are diagrams illustrating an example of analyzing content according to various embodiments.

According to various embodiments, the electronic device 300 may analyze content information. The content information may be multimedia data that the presenter can use as presentation material. For example, the content information includes text, image, video, and data (eg, Power Point TM) data that can be executed as an application executable in the electronic device 210 or output on the display of the electronic device 210 . ) and a combination of at least some of them. The content information may be data executed or displayed by the electronic device 300 or multimedia data information received from an external electronic device. The content information may be information received by the electronic device 300 from an external device (eg, the server device 220 of FIG. 2 ) using the communication module 310 , or executed by the electronic device 300 . or data that is executed and output by the output module 340 . Alternatively, the content information may be image data in which the electronic device 300 extracts only an image region identified as content from other image information.

Referring to FIG. 10 , content information according to various embodiments may be expressed as a screen 1010 in which an image and/or text are combined.

Referring to FIG. 10 , when content information includes text, an image, a moving picture, or a combination thereof, the electronic device 300 may analyze the text, image, and/or video. For example, the content information may be in the form of including a plurality of multimedia contents (eg, images and texts) in the same output screen. In this case, the electronic device 300 may recognize each content as a content object. The electronic device 300 may identify each recognized content object, and classify and select the identified object. For example, the identified content objects may be classified according to content types such as images, texts, and/or images, respectively. Referring to FIG. 10 , the content information 1010 classifies each recognized object to provide text 1 (Text1) 1021, image 1 (Img1) 1022, image 2 (Img2) 1023, and text 2 (Text2). ) (1024), image 3 (Img3) (1025), and text 3 (Text3) (1026).

Referring to FIG. 11 , it is an exemplary diagram of a logical connection link between the content objects classified in FIG. 10 . According to various embodiments, the electronic device 300 may be included in each content object by using image analysis, text analysis (eg, optical character recognition (OCR)) and/or image analysis for each type of content object classified. A semantic unit may be analyzed or related text information may be extracted. The electronic device 300 may extract content information of each content object using image analysis, text analysis, and/or image analysis. In addition, the position and/or size of the content object displayed on the screen may be checked, and information on the size and position may be extracted. According to various embodiments, the electronic device 300 may generate information on a logical connection link between content objects by using semantic unit information extracted from the identified content object, related text information, and/or location information. The logical connection link may mean, for example, context information between each content object, and may mean a content path of a logically expected presentation. According to various embodiments, the context information or the logical connection link information may include content that is highly related to each other using the similarity of information included in each content object and/or the proximity of a distance using information extracted from each content object. It may mean an order of objects and/or a connection link. According to various embodiments, when the electronic device 300 determines that the presenter's presentation is related to the specified object, the object most related to the content of the presentation to be performed immediately after the corresponding presentation is the current object and the current object on the logical connection link information. It can be expected that the object exists in the nearest link. According to various embodiments, the electronic device 300 may generate content analysis information including at least one of location information, content information, page information, and page mapping information of a content object. According to various embodiments, the electronic device 300 may transmit content analysis information to a server device (eg, 220 of FIG. 2 ) through the communication module 310 . Referring to FIG. 11 , an example in which logical connection link information is configured based on location information of a content object may be configured. For example, when the position of the content object is classified in a hierarchical relationship 1110, the logical link link 1130 starting from the text 1 (Text1) 1021 is the text 1 (Text1) 1021, the image 1 ( It may be a link leading to Img1) 1022 and Text2 (Text2) 1024 . In addition, the logical connection link 1140 starting from the image 2 (Img2) 1023 is a link from the image 2 (Img2) 1023 to the image 3 (Img3) 1025 or text 3 (Text3) 1026 can be As another example, if the position of the content object is classified into a left-right relationship (1120), the logical link link 1150 starting from the text 1 (Text1) 1021 is connected to the image 2 (Img2) 1023 It can be a link. In addition, the logical connection link 1160 starting from the image 1 (Img1) 1022 may be a link from the image 1 (Img1) 1022 to the image 2 (Img2) 1023, and the text 2 (Text2) A logical connection link 1170 starting at 1024 may be a link leading to an image 3 (Img3) 1025 and a text 3 (Text3) 1026 . When the electronic device 300 according to various embodiments determines a partial region of a content screen in which the presenter image information is to be arranged, the region in which the presenter image information is to be arranged includes information on a logical connection link between content objects included in the content analysis information. can be selected on the basis of The electronic device 300 determines that, when a presentation proceeds from a content object related to the present presentation content to the next content object, the electronic device 300 has the highest relevance to the presentation content to be presented later based on the information on the logical connection link between the content objects. Content objects can be predicted. When the electronic device 300 selects an area to be placed in the future, when the current presenter image information is disposed in the designated area and the content of the presentation is predicted to proceed with the content object predicted in advance, the area most suitable for the predicted content object can be selected. For example, when it is determined that the current presentation position is image 1 (Img1) 1022 , the electronic device 300 sets the content object corresponding to the next presentation to text 2 (Text2) 1024 based on the vertical relationship. It is possible to predict the image 2 (Img2) 1023 as , or based on the logical connection link information for the left-right relationship.

Referring to FIG. 12 , when content information corresponds to data executable as an application, the electronic device (eg, the electronic device 300 of FIG. 3 ) may analyze execution data of the application and input data for the application. . For example, the electronic device 300 may analyze a user interface (UI) of the application. According to various embodiments, the electronic device 300 uses a simulation technique to simulate each UI page and/or a page on a UI layout such as a pop-up window, and/or for each window based on a simulation of the movement of a UI page of the corresponding application. Connection information may be checked ( 1210 ). According to various embodiments, the electronic device 300 may obtain mapping information between UI pages by combining each movable page and/or window for each UI page and/or window of the corresponding application. Referring to FIG. 12 , a simulation is performed on UI page A (UI Page: A) 1230 and a UI page B (UI Page: B) 1250 that is in progress by clicking a clickable object 1240 can be checked, and , a relationship between UI page A 1230 and UI page B 1250 can be confirmed as a clickable object 1240 . By clicking another clickable object 1270 on the returned UI page A 1260 according to the same method, it can be confirmed that the simulation result is connected to the UI page C (UI Page C) 1280 . By repeating this simulation, the connection information 1220 between UI pages may be checked, and a result of mapping connection information for resources between UI pages may be extracted. Referring to FIG. 12 , the electronic device 300 may extract connection information 1220 between UI pages.

Referring to FIG. 13 , according to various embodiments, when content information corresponds to data executable as an application other than the above type, the electronic device 300 analyzes execution data of the application and input data for the application. can For example, the electronic device 300 may analyze a user interface (UI) of the application. According to various embodiments, the electronic device 300 uses a simulation technique to simulate each UI page and/or a page on a UI layout such as a pop-up window, and/or for each window based on a simulation of the movement of a UI page of the corresponding application. Connection information may be checked, or this information may be previously stored in another component (eg, the memory 330 ) of the electronic device 300 . According to various embodiments, the electronic device 300 may obtain mapping information between UI pages by combining each movable page and/or window for each UI page and/or window of the corresponding application. According to various embodiments, the electronic device 300 may analyze a UI component (eg, a UI object) that triggers page movement and/or creation of a pop-up window within the same UI page. This may be possible by reading control information included in each UI component in the same UI page. You can check the current UI page by combining all UI control information in the same page. Such control information and a combination thereof may be analyzed in real time by the electronic device 300 , and may be information previously stored in the electronic device 300 . Also, the electronic device 300 may check information on at least one object identified in the UI page. Referring to FIG. 13 , for example, information on each object (eg, 1310 , 1320 , 1330 , and 1340 of FIG. 3 ) constituting a designated UI page 1300 may be analyzed. For example, with respect to the specified object 1310 , resource information 1311 on the corresponding object may be extracted. In the resource information 1311, the type of the object (eg, image, layout), resource identification (resource identification), and control information (eg, isClickable/isLongClickable/isScrollable/isFocusable) (eg, clickable/long-clickable) , scrolling, or magnification), extracting the corresponding information, and including it in the content analysis information.

14 is a flowchart of a method of an electronic device analyzing content according to various embodiments of the present disclosure;

Referring to FIG. 14 , an electronic device (eg, the electronic device 300 of FIG. 3 and the server device 220 of FIG. 2 ) may analyze content information.

Referring to operation 1410 , the electronic device 300 may determine the type of content information. For example, the type of application required to execute or display content information in the electronic device 300 may be checked. Alternatively, the type of content information may be determined through the type of an application for displaying content information that is already being executed. The type of content information may be divided into a case of video, image, text, or a combination thereof and other applications. A content analysis method may be different based on the determined type of content information.

Referring to operation 1420 , when the content information includes text, an image, a moving picture, or a combination thereof, the electronic device 300 may analyze the text, the image, and/or the image. For example, the content information may be in the form of including a plurality of multimedia contents (eg, images and texts) in the same output screen. In this case, the electronic device 300 may recognize each content as a content object. Referring to operation 1430 , the electronic device 300 may identify each recognized content object, and classify and select the identified object. For example, the identified content objects may be classified according to content types such as images, texts, and/or images, respectively.

Referring to operation 1440 , when the content information corresponds to data that can be executed as an application other than the above type, the electronic device 300 may analyze execution data of the application and input data for the corresponding application. For example, the electronic device 300 may analyze a user interface (UI) of the application. According to various embodiments, the electronic device 300 uses a simulation technique to simulate each UI page and/or a page on a UI layout such as a pop-up window, and/or for each window based on a simulation of the movement of a UI page of the corresponding application. Connection information may be checked, or such information may be previously stored in another component (eg, the memory 330 ) of the electronic device.

Referring to operation 1450 , the electronic device may identify a content object and analyze layout information and/or page information. According to various embodiments, the electronic device 300 may obtain mapping information between UI pages by combining each movable page and/or window for each UI page and window of the corresponding application. According to various embodiments, the electronic device 300 may analyze a UI component (eg, a UI object) that triggers page movement and/or creation of a pop-up window within the same UI page. This may be possible by reading control information included in each UI component in the same UI page. You can check the current UI page by combining all UI control information in the same page. Such control information and a combination thereof may be analyzed in real time by the electronic device 300 , and may be information previously stored in the electronic device 300 . Also, the electronic device 300 may check information about an object identified in the UI page.

Referring to operation 1460 , the electronic device 300 may extract location information of the content object and content information of the content object. According to various embodiments, the electronic device 300 may be included in each content object by using image analysis, text analysis (eg, optical character recognition (OCR)) and/or image analysis for each type of content object classified. A semantic unit may be analyzed or related text information may be extracted. The electronic device 300 may extract content information of each content object using image analysis, text analysis, and/or image analysis. In addition, the position and/or size of the content object displayed on the screen may be checked, and information on the size and position may be extracted. According to various embodiments, the electronic device 300 may generate information on a logical connection link between content objects by using semantic unit information extracted from the identified content object, related text information, and/or location information. The logical connection link may mean, for example, context information between each content object, and may mean a content path of a logically expected presentation. According to various embodiments, the context information or the logical connection link information may include content that is highly related to each other using the similarity of information included in each content object and/or the proximity of a distance using information extracted from each content object. It may mean an order of objects and/or a connection link. According to various embodiments, when the electronic device 300 determines that the presenter's presentation is related to the specified object, the object most related to the content of the presentation to be performed immediately after the corresponding presentation is the current object and the current object on the logical connection link information. It can be expected that the object exists in the nearest link. Referring to operation 1470 , the electronic device 300 may generate content analysis information including at least one of location information, content information, page information, and page mapping information of a content object.

15 is a diagram illustrating an embodiment in which regions of a content screen are divided and a presenter image is dynamically arranged according to various embodiments of the present disclosure;

Referring to FIG. 15 , the electronic device (eg, the electronic device 300 of FIG. 3 or the server device 220 of FIG. 2 ) may divide a content screen into a plurality of areas ( 1510 ).

According to various embodiments, the electronic device 300 may divide the content screen into one or more regions based on the content analysis information. The content screen may be all or part of a designated screen for displaying content information. The electronic device 300 may classify the area of the content screen based on the size and position of the content object recognized in the content information. According to various embodiments, the electronic device 300 may classify an area for each designated UI object by using information about the layout of the content analysis information. According to various embodiments, the electronic device 300 may classify a designated partial area including a related content object into one area based on content information of the content object. For example, referring to FIG. 15 , the electronic device 300 may divide the content screen 1510 into a plurality of regions. Referring to the example of the first area 1511 and the second area 1513 among the plurality of divided areas, the first area 1511 may be any area in which an image content object exists, and the second area 1513 may be any area in which an image content object exists. may be a region in which another image content object exists.

According to various embodiments, the electronic device 300 may select a region in which the presenter image is to be placed among the divided regions. According to various embodiments, the electronic device 300 may select at least one area among each area of the divided content screen as an area in which the presenter's image is to be arranged. According to various embodiments, the area to be arranged may be selected based on input information (content input information) of a user (eg, a presenter) for content information, gesture information of the presenter, and/or voice information 1540 of the presenter. .

Referring to FIG. 15 , the electronic device 300 may dynamically arrange the presenter image information among the divided areas of the content screen as in the first arrangement result 1520 and the second arrangement result 1530 . For example, referring to the first arrangement result 1520 , content input information, presenter gesture information, and/or presenter voice information may be detected with respect to the content object 1521 of the first region 1511 . In this case, the presenter image information 1522 is disposed adjacent to the content object 1521 of the first area 1511 but on an area other than the first area 1511 that does not obstruct the view of the content object 1521 . can do. As another example, referring to the second arrangement result 1530 , it may be a case in which content input information, presenter gesture information, and/or presenter voice information is detected with respect to the content object 1531 of the second region 1513 . there is. In this case, the presenter image information 1532 may be arranged at a position related to the content object 1531 in the second area 1513 . In this case, the presenter image information 1532 is disposed on an area other than the second area 1513 adjacent to the content object 1531 of the second area 1513 but not obstructing the view of the content object 1531 . can do.

16 is a flowchart of a method for an electronic device to divide a region of a content screen and dynamically arrange a presenter image according to various embodiments of the present disclosure;

Referring to FIG. 16 , a series of operations of dividing a region of a content screen and dynamically arranging a presenter image is performed by an electronic device (eg, the electronic device 300 of FIG. 3 ) or a server (eg, the server device of FIG. 2 ) 220)).

Referring to operation 1610 , the electronic device 300 may divide the content screen into one or more regions based on the content analysis information. The content screen may be all or part of a designated screen for displaying content information. The electronic device 300 may classify the area of the content screen based on the size and position of the content object recognized in the content information. According to various embodiments, the electronic device 300 may classify an area for each designated UI object by using information about the layout of the content analysis information. According to various embodiments, the electronic device 300 may classify a designated partial area including a related content object into one area based on content information of the content object.

Referring to operation 1620 , according to various embodiments, the electronic device 300 may select at least one area among each area of the divided content screen as an area in which the presenter's image is to be placed. According to various embodiments, the area to be arranged may be selected based on input information of a user (eg, a presenter) for content information, gesture information of the presenter, and/or voice information of the presenter.

According to various embodiments, the electronic device 300 may receive voice information of a presenter. For example, audio data included in the presenter's video information may be received as presenter's voice information. Alternatively, voice information input by the presenter may be received using the microphone of the input module 320 . According to various embodiments, the electronic device 300 may analyze the presenter's voice information to identify the presenter's current presentation. For example, the electronic device 300 may recognize the speaker's voice information and analyze it for each morpheme or semantic unit, or convert the speaker's voice information into text and perform content analysis using the text information. According to various embodiments, the electronic device 300 may analyze the presenter's voice information and select a divided, designated area on the content screen into which the presenter's image information is to be merged by using the information on the currently ongoing presentation. For example, the content analysis information, which is a result of analyzing the content object included in the content information, may include content information of the content object, and the electronic device 300 compares the presenter's voice information with the content object content information to proceed with the current process. It is possible to detect a content object most related to the current presentation. In this case, the electronic device 300 may place the presenter image in a region that does not cover the corresponding content object among the plurality of divided regions of the content screen, and/or arrange the presenter image in an area adjacent to the corresponding content object. there is.

According to various embodiments, the input information for the content information may be a user input received from the input module 320 . For example, the content input information may be input information for content information, such as mouse manipulation input information and/or keyboard manipulation input information. When the mouse pointer is positioned at a specified position as a result of a manipulation of the mouse movement on the content, the electronic device 300 may recognize this and determine that the presentation of the content object at the corresponding position is in progress. In this case, the electronic device 300 may not obstruct the view of the corresponding content object or may select an area adjacent to the corresponding object. According to various embodiments, the content input information may be included in the content analysis information.

According to various embodiments, the presenter's gesture information may be information on a result of analyzing the presenter's behavior. For example, the presenter gesture information may be included in the presenter image analysis information. According to various embodiments, the electronic device 300 may use the presenter image analysis information to identify a content object related to the content of the present presentation based on the presenter's gesture information. The gesture information may include information regarding a gesture of the presenter, the presenter's gaze, and/or an operation of the presenter pointing to content using a designated tool. According to various embodiments, a region in which the presenter image information is to be arranged may be selected based on information on a logical connection link between content objects included in the content analysis information. The electronic device 300 determines that, when a presentation proceeds from a content object related to the present presentation content to the next content object, the electronic device 300 has the highest relevance to the presentation content to be presented later based on the information on the logical connection link between the content objects. Content objects can be predicted. When the electronic device 300 selects an area to be placed in the future, when the current presenter image information is disposed in the designated area and the content of the presentation is predicted to proceed with the content object predicted in advance, the area most suitable for the predicted content object can be selected.

Referring to operation 1630 , the electronic device 300 may synthesize the content screen and the presenter image by arranging the presenter image in at least one selected area among the divided areas of the content screen. The electronic device 300 may generate a synthesized image by arranging the presenter image in the selected designated area. According to various embodiments, the electronic device 300 may transmit a synthesized image to a server device (eg, the server device 220 of FIG. 2 ) through the communication module 310 , and the electronic device 300 may output A synthesized image may be output using the module 340 . According to various embodiments, when each operation 1610 , 1620 , and 1630 according to FIG. 16 is an operation of the server device 220 , the server device 220 performs the operation of the electronic device 300 and/or an external electronic device (eg, : Streaming information may be transmitted to stream the synthesized image to the external electronic device 230 of FIG. 2 .

An electronic device providing a video conferencing service according to various embodiments disclosed herein includes a communication module, an input module, a display, a memory, and operatively connected to the communication module, the input module, the display, and the memory. a processor, wherein the processor generates content analysis information by analyzing the content information being executed, generates the presenter image analysis information by analyzing the received presenter image information, and adds the content information to the content information based on the content analysis information. divides the content execution screen for the content execution screen into a plurality of regions, and selects at least one region in which the presenter image information is to be arranged among the plurality of regions of the content execution screen based on the content analysis information and the presenter image analysis information, and The presenter image information may be set to be synthesized by arranging it in a selected area among areas of the content execution screen.

Also, the processor may be configured to extract at least one content object from the content information.

In addition, the processor may be configured to check the size of the content object and the position of the content object, and to divide the region into the plurality of regions based on the checked size and position of the content object.

In addition, the processor, extracting the content information of the content object, and dividing the execution screen of the content into a plurality of regions, divide the execution screen of the content into a plurality of regions based on the content information of the content object can be set to

Also, the processor may be configured to receive an input for the content information, and to select at least one region in which the presenter image information is to be placed among the plurality of regions based on the input for the content.

In addition, the processor is configured to analyze the presenter image information to extract presenter gesture information from the image information, and to select at least one region in which to place the presenter image information among the plurality of regions based on the presenter gesture information. can be set.

In addition, the processor further receives audio information related to the presenter video information, analyzes the audio information to extract information related to presentation content from the audio information, and based on the information related to the presentation content, the plurality of It may be set to select at least one area in which the presenter image information is to be arranged among the areas.

Also, the processor may be configured to analyze the presenter image information to extract a presenter image from the presenter image information, and to edit the presenter image information based on the extracted presenter image.

In addition, the processor checks a UI change according to an input for content and a function execution of the content when the content is executed, stores UI change information for the UI change, and the plurality of areas based on the UI change information It may be set to select at least one area in which the presenter image information is to be arranged.

Also, the processor may be configured to generate synthesized image information generated by synthesizing the presenter image information into a selected region among regions of the content execution screen, and transmit the synthesized image information to an external electronic device.

A method of providing a video conferencing service according to various embodiments disclosed herein includes the steps of generating content analysis information by analyzing running content information, analyzing the received presenter image information to generate presenter image analysis information; dividing the content execution screen for the content information into a plurality of regions based on the content analysis information; the presenter image information among a plurality of regions of the content execution screen based on the content analysis information and the presenter image analysis information The method may include selecting at least one area in which to place , and synthesizing the presenter image information into a selected area among areas of the content execution screen.

The analyzing of the content information may further include extracting at least one content object from the content information, and confirming a size of the content object and a location of the content object, the identified content The method may include dividing the object into the plurality of regions based on the size and position of the object.

In addition, the analyzing of the content information may further include extracting content information of the content object, and dividing the execution screen of the content into a plurality of regions may include: based on the content information of the content object It may include dividing the execution screen of the content into a plurality of areas.

The method may further include receiving an input for the content information and selecting at least one region in which the presenter image information is to be placed among the plurality of regions based on the input for the content information.

In addition, the analyzing of the presenter image information may further include extracting presenter gesture information by analyzing the presenter image information, wherein the presenter image information is arranged in the plurality of areas based on the presenter gesture information. The method may further include selecting at least one region.

The method may further include generating synthesized image information generated by synthesizing the presenter image information to a selected region among regions of the content execution screen, and transmitting the synthesized image information to an external electronic device.

A server device according to various embodiments disclosed herein includes a communication interface, a memory, and a processor operatively connected to the communication interface and the memory, wherein the processor analyzes presenter image information and presenter image from an external electronic device Receive information, content execution screen information, and content analysis information, divide the content execution screen into a plurality of regions based on the content analysis information, and arrange the presenter image information among the plurality of regions of the content execution screen. It may be set to select one region and synthesize the presenter image information into the selected region among regions of the content execution screen.

In addition, the processor receives a plurality of presenter image information, analyzes each of the plurality of presenter image information to generate presenter image analysis information corresponding to each image, and based on the presenter image analysis information, the plurality of presenters It may be set to select presenter image information of a presenter who is currently speaking from among the image information.

Also, the processor may be configured to generate synthesized image information generated by synthesizing the presenter image information into a selected region among regions of the content execution screen, and transmit the synthesized image information to an external electronic device.

An electronic device according to various embodiments disclosed herein includes a communication module, an input module, a display, a memory, and a processor operatively connected to the communication module, the input module, the display, and the memory; The processor receives image information, extracts presenter image information and content image information by analyzing the image information, generates content analysis information by analyzing the extracted content image information, and analyzes the presenter image to obtain a presenter image generating analysis information, dividing the content image information into a plurality of regions based on the content analysis information, selecting at least one region in which the presenter image information is to be placed among the plurality of regions of the content image information, and the It may be configured to synthesize the presenter image information into a selected region among the regions of the content image information.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the element from other elements in question, and may refer to elements in other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively". When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present document, one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101) may be implemented as software (eg, the program 140) including For example, a processor (eg, processor 120 ) of a device (eg, electronic device 101 ) may call at least one command among one or more commands stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store™) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly between smartphones (eg: smartphones) and online. In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. or one or more other operations may be added.

300: electronic device 411: presenter image information
412: content information 500: function processing module

Claims (20)

An electronic device for providing a video conferencing service, comprising:
communication module;
input module;
display;
Memory; and
a processor operatively coupled to the communication module, the input module, the display, and the memory;
The processor is
Generates content analysis information by analyzing the running content information,
Analyze the received presenter image information to generate presenter image analysis information,
dividing the content execution screen for the content information into a plurality of regions based on the content analysis information;
selecting at least one region in which the presenter image information is to be arranged among a plurality of regions of the content execution screen based on the content analysis information and the presenter image analysis information;
An electronic device configured to arrange and synthesize the presenter image information in a selected area among areas of the content execution screen. According to claim 1,
The processor is
An electronic device configured to extract at least one content object from the content information. 3. The method of claim 2,
The processor is
Check the size of the content object and the location of the content object,
The electronic device is set to divide into the plurality of regions based on the identified size and location of the content object. 3. The method of claim 2,
The processor is
extracting the content information of the content object;
The operation of dividing the execution screen of the content into a plurality of areas includes:
An electronic device configured to divide the content execution screen into a plurality of regions based on content information of the content object. According to claim 1,
The processor is
receiving an input for the content information;
The electronic device is configured to select at least one area in which the presenter image information is to be placed among the plurality of areas based on the input of the content. According to claim 1,
The processor is
extracting the presenter gesture information from the image information by analyzing the presenter image information,
The electronic device is configured to select at least one area in which the presenter image information is to be arranged among the plurality of areas based on the presenter gesture information. According to claim 1,
The processor is
Receive further audio information related to the presenter video information,
Analyze the voice information to extract information related to the content of the presentation from the voice information,
The electronic device is configured to select at least one area in which the presenter image information is to be arranged among the plurality of areas based on information related to the presentation content. According to claim 1,
The processor is
An electronic device configured to analyze the presenter image information to extract a presenter image from the presenter image information, and to edit the presenter image information based on the extracted presenter image. According to claim 1,
The processor is
When executing the content, check the UI change according to the input of the content and the execution of the function of the content,
store UI change information for the UI change,
The electronic device is configured to select at least one area in which the presenter image information is to be arranged among the plurality of areas based on the UI change information. According to claim 1,
The processor is
and synthesized image information generated by synthesizing the presenter image information into a selected region among regions of the content execution screen;
An electronic device configured to transmit the composite image information to an external electronic device. A method of providing a video conferencing service, comprising:
generating content analysis information by analyzing the content information being executed;
analyzing the received presenter image information to generate presenter image analysis information;
dividing the content execution screen for the content information into a plurality of regions based on the content analysis information;
selecting at least one region in which the presenter image information is to be arranged among a plurality of regions of the content execution screen based on the content analysis information and the presenter image analysis information; and
and synthesizing the presenter image information into a selected area among areas of the content execution screen. According to claim 11,
The step of analyzing the content information includes:
extracting at least one content object from the content information; and
Further comprising the step of confirming the size of the content object and the position of the content object,
and dividing into the plurality of regions based on the size and position of the identified content object. According to claim 12,
The step of analyzing the content information includes:
Further comprising the step of extracting the content information of the content object,
The step of dividing the execution screen of the content into a plurality of areas includes:
and dividing the execution screen of the content into a plurality of regions based on the content information of the content object. 12. The method of claim 11,
receiving an input for the content information;
The method further comprising the step of selecting at least one area in which the presenter image information is to be arranged among the plurality of areas based on the input of the content information. 12. The method of claim 11,
The step of analyzing the presenter video information includes:
The method further comprising the step of analyzing the presenter image information to extract presenter gesture information,
The method further comprising the step of selecting at least one area in which the presenter image information is to be placed among the plurality of areas based on the presenter gesture information. 12. The method of claim 11,
generating synthesized image information generated by synthesizing the presenter image information with a selected region among regions of the content execution screen; and
The method further comprising transmitting the synthesized image information to an external electronic device. In the server device,
communication interface;
Memory; and
a processor operatively coupled to the communication interface and memory;
The processor is
receiving presenter image information, presenter image analysis information, content execution screen information, and content analysis information from an external electronic device;
Divide the content execution screen into a plurality of areas based on the content analysis information,
selecting at least one area in which the presenter image information is to be arranged among a plurality of areas of the content execution screen;
An electronic device configured to synthesize the presenter image information into a selected region among regions of the content execution screen. 18. The method of claim 17,
The processor is
Receive a plurality of presenter video information,
Analyze each of the plurality of presenter image information to generate presenter image analysis information corresponding to each image,
An electronic device configured to select presenter image information of a presenter who is speaking from among the plurality of presenter image information based on the presenter image analysis information. 18. The method of claim 17,
The processor is
and synthesized image information generated by synthesizing the presenter image information into a selected region among regions of the content execution screen;
An electronic device configured to transmit the composite image information to an external electronic device. In an electronic device,
communication module;
input module;
display;
Memory; and
a processor operatively coupled to the communication module, the input module, the display, and the memory;
The processor is
receive video information;
extracting presenter image information and content image information by analyzing the image information,
Analyze the extracted content image information to generate content analysis information,
Analyzing the presenter image to generate presenter image analysis information,
dividing the content image information into a plurality of regions based on the content analysis information,
selecting at least one region in which the presenter image information is to be arranged among a plurality of regions of the content image information;
An electronic device configured to synthesize the presenter image information into a selected region among regions of the content image information.

Images