KR20190136264A - Method for selecting location to execution screen of application - Google Patents

Abstract

Provided is a method for selecting a position of an execution screen of an application. According to one embodiment of the present invention, the method capable of providing an optimized execution position of an application graphical user interface (GUI) may comprise the steps of: obtaining a position of an initial execution area of an application GUI; receiving situational information for correcting the position of the initial execution area and correcting a position of an area for displaying the application GUI based on the situational information; and displaying the application GUI on the position of the area according to a correction result regardless of an operation input of a user.

Description

How to select the location of the application execution screen {METHOD FOR SELECTING LOCATION TO EXECUTION SCREEN OF APPLICATION}

The present invention relates to a method for selecting a location of an application execution screen. More specifically, the present invention relates to a method for analyzing a user's behavior on a computing device and selecting an execution screen position of an application based on the same.

In a display device connected to a computing device such as a PC device or a smart device, an execution screen of various applications may be displayed, and a display position of the application may be disposed on various locations on the screen of the display device. At this time, the position where the application is first executed and displayed is generally displayed according to a separate setting input by a user or executed again at the position where the application is terminated.

However, in general, in the display device, only one application is not executed and displayed, and another application may be additionally executed and displayed on the screen at the time when one application is executed. At this time, if the task pane of the application scheduled to be executed in the limited screen area is displayed superimposed on the task pane of the previously executed application, the user may interfere with the work in progress.

The invention described below has been made to solve the above-mentioned problems, and provides various means for displaying an application graphical user interface (GUI) in an optimal position on the screen without disturbing the user's ongoing work. I would like to.

Korean Patent Publication No. 2013-0106147

SUMMARY OF THE INVENTION The present invention has been made in an effort to provide a method and apparatus capable of providing an optimized application GUI execution position on a screen of a computing device.

Another technical problem to be solved by the present invention is to provide a method and apparatus for predicting a screen area on a computing device intended by a user in order to provide an optimized application GUI execution position to a user.

The technical problems of the present invention are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the technical problem, the method of selecting an application execution screen according to an embodiment of the present invention, the step of obtaining the position of the first execution region of the application graphical user interface (GUI), the position of the first execution region Receiving situation information for correction, correcting a position of an area for displaying the application GUI based on the situation information, and displaying the application GUI at an area position according to the correction result irrespective of a user's operation input It may include the step.

In one embodiment, correcting the position of an area for displaying the application GUI based on the context information may include determining a gaze area of a user that is a part of a screen area capable of displaying the application GUI, wherein the gaze And sensing at least one of activation of an application in an area and a user input to the application, and determining the gaze area as a location of an area for displaying the application GUI according to the detection result. .

In one embodiment, correcting a position of an area for displaying the application GUI based on the context information may include determining an execution area of at least one application GUI running in the screen area, wherein the screen area is selected from among the screen areas. And determining a non-execution area except the execution area as a location for displaying the application GUI.

The determining of the non-execution area other than the execution area as a location for displaying the application GUI may include: when a display area for executing a virtual input interface is detected in a part of the screen area, The method may include determining an area other than a display area for executing a virtual input interface as a location for displaying the application GUI.

In one embodiment, correcting the position of the area for displaying the application GUI based on the situation information, based on the position of the first execution area of the application GUI based on the operation information received from the user input device. The method may include determining a changed position as a location, and determining the changed location as a location for displaying the application GUI.

In an embodiment, the obtaining of the position of the first execution region of the application graphical user interface (GUI) may include: the screen region is divided into at least a first screen region and a second screen region, and the first screen region and the first screen region. Acquiring an initial execution position, which is one of two screen regions, and correcting the position of the region for displaying the application GUI based on the situation information comprises: acquiring an operation position received from a user input device; Confirm the changed position from the initial execution position of the application GUI, and correct according to a result of determining whether the gaze of the user matches the changed position based on face information of the user received from the image device mounted in the computing device. And the application at an area position according to the correction result. The method comprising the illustration GUI displayed regardless of the operation input of the user, may be subject to the changed position to a new position include the step of displaying the application GUI.

In an embodiment, the correcting of the position of the region for displaying the application GUI based on the context information may include: an information input / output region of an application display region that is being executed included in at least a portion of the screen region. And determining a location for displaying the execution of the application according to a result of determining the excluded area, and displaying the application GUI regardless of a user's operation input at an area location according to the correction result. And displaying the application GUI using an area in which an information input / output area is excluded as the new location.

In an embodiment, the correcting of the position of the region for displaying the application GUI based on the context information may include displaying execution of the application according to an attribute of an application which is different from the application in the screen region. And limiting the screen area in an area for the display, and displaying the application GUI at an area position according to the correction result irrespective of a user's operation input, according to the result of the restriction. The method may include assigning a priority to the application GUI display.

In one embodiment, correcting the position of the area for displaying the application GUI based on the context information may not display the application GUI when the user is not sensed from an image device mounted in the computing device. And determining that the application GUI is irrelevant to a user's operation input at an area position according to the correction result, wherein the display is limited to display for execution of the application in the screen area according to the determination. It may include a step.

In order to solve the above technical problem, the positioning learning method of the application execution screen according to an embodiment of the present invention is to identify a first display area which is a part of the screen area of the display device and which matches the direction of the user's gaze. Determining whether to execute at least one application in the first display area; and when it is determined that the at least one application is running in the first display area according to the determination result, the first display area is determined. And determining the region of interest of the user.

The determining of the first display area as the ROI of the user may include determining the first display area as the ROI of the user when a user manipulation input to the at least one application is detected. It may include.

According to another aspect of the present invention, there is provided a positioning learning method of an application execution screen, wherein the second display area is a part of a screen area of a display device and is an area that does not coincide with a user's gaze direction. Determining whether the at least one application is running in the second display area; and when it is determined that the at least one application is running in the second display area according to the determination result, the second display area Determining the area of interest of the user may include.

In an embodiment, the determining of the second display area as the ROI of the user may include determining the second display area as the ROI of the user based on operation information received from a user input device. The manipulation information may indicate a moving direction in the screen area.

According to another aspect of the present invention, there is provided a method for selecting an application execution screen according to another embodiment of the present invention to execute execution of at least one second application and a first display area for displaying execution of at least one first application. Detecting any one display area of the second display area to be displayed, the second application area being determined as the first display area according to the detection result; May be displayed on the first display area.

The displaying of the second application on the first display area may include displaying the second application on the first display area according to a result of determining that the first application is running on the first display area. It may include the step of displaying.

The displaying of the second application on the first display area may include displaying a new application on the first display area according to a result of determining that the first application is running on the first display area. It may comprise the step of limiting.

In an embodiment, the detecting of any one display area that is a gaze of a user may include: displaying the user facing the user among the first display area and the second display area when the foldable electronic device is in a folding state. Detecting an area, wherein the facing display area is a first display area.

The displaying of the second application on the first display area may include moving a display position of the second application to the second display area, wherein the second display area is out of view of the user. Which may include a step.

According to another aspect of the present invention, there is provided a computer program including: acquiring a position of an initial execution region of an application graphical user interface (GUI), and a situation for correcting the position of the first execution region; Receiving information, correcting a position of an area for displaying the application GUI based on the situation information, and displaying the application GUI at an area position according to the correction result irrespective of a user's operation input; It can be stored and implemented in a recording medium to make it.

1 is a block diagram of an apparatus in which a method for positioning an application execution screen is performed according to some embodiments of the present invention.
FIG. 2 is a diagram for describing learning data referred to in some embodiments of the present invention.
3 is a flowchart illustrating a method for selecting a location of an application execution screen according to an embodiment of the present invention.
4 to 5 are detailed flowcharts for describing a detailed process of step S300 of FIG. 3.
6 is a diagram for explaining an example of selecting a position at which execution of an application is displayed according to an embodiment of the present invention.
FIG. 7 illustrates an example of selecting a location at which execution of an application is displayed according to another exemplary embodiment of the present invention.
8 is a diagram for describing an example of selecting a position at which execution of an application is displayed according to another exemplary embodiment of the present invention.
9 is a diagram for describing an example of selecting a position at which execution of an application is displayed on a multi-screen according to an embodiment of the present invention.
FIG. 10 is a diagram for describing an example of selecting a position at which execution of an application is displayed on a mobile screen according to an embodiment of the present invention.
FIG. 11 is a diagram for describing an example of selecting a location at which execution of an application is displayed on a foldable screen according to another embodiment of the present invention.
12 is a flowchart illustrating a positioning learning method of an application execution screen according to an embodiment of the present invention.
FIG. 13 is a diagram to help explain a method of learning location selection of an application execution screen according to FIG. 13.
14 is a flowchart illustrating a positioning learning method of an application execution screen according to another exemplary embodiment of the present invention.
FIG. 15 is a diagram to help explain a method of learning location selection of an application execution screen according to FIG. 15.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms, and only the embodiments make the disclosure of the present invention complete, and the general knowledge in the art to which the present invention belongs. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used as meanings that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase.

As used herein, “comprises” and / or “comprising” refers to a component, step, operation and / or element that is mentioned in the presence of one or more other components, steps, operations and / or elements. Or does not exclude additions.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus in which a method for positioning an application execution screen is performed according to some embodiments of the present invention. Hereinafter, for convenience of description, an apparatus in which a method of selecting an application execution screen is performed is referred to as a function execution apparatus.

The function execution device 100 illustrated in FIG. 1 may be a touch-based mobile computing device. For example, the function execution device 100 according to some embodiments includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant, a portable multimedia player (PMP), Navigation, slate PC, tablet PC, ultrabook, wearable device (e.g., smart watch, smart glass, HMD (head) mounted display)).

However, it will be apparent to those skilled in the art that the configuration according to some embodiments may be applied to fixed terminals such as digital TVs, desktop computers, digital signages, and the like.

Referring to FIG. 1, the function execution device 100 may include a communication unit 110, an input unit 120, an output unit 130, a storage unit 140, and a controller 150. The components shown in FIG. 1 are not essential to implementing the function execution apparatus 100, so that the function execution apparatus 100 described in some embodiments may have more or fewer components than those listed above. Can have However, the function execution device 100 may include various input means for inputting user's context information. The situation information will be described in detail with reference to FIG. 2.

The communication unit 110 may include a mobile communication module and a wireless communication module for communicating with an external device. The communication unit 110 may receive data such as a voice signal, a video call signal, a text / multimedia message, an E-mail, and multimedia content.

The mobile communication module provided in the communication unit 110 may include technical standards for mobile communication or technical standards or communication methods (for example, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), Enhanced Voice-Data Optimized or Enhanced Voice-Data Only (EV-DO), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), A wireless signal is transmitted and received with at least one of a base station, an external terminal, and a server on a mobile communication network constructed according to Long Term Evolution (LTE) and Long Term Evolution-Advanced (LTE-A).

The wireless Internet module provided in the communication unit 110 refers to a module for wireless Internet access. The wireless Internet technologies used in the communication unit 110 include WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), and Wi-Fi (Wireless Fidelity). ) Direct, Digital Living Network Alliance (DLNA), Wireless Broadband (WiBro), World Interoperability for Microwave Access (WiMAX), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) And Long Term Evolution-Advanced (LTE-A).

The input unit 120 may include a camera or an image input unit for inputting an image signal, a microphone for inputting an audio signal, or an audio input unit, and a user input unit for receiving information from a user (for example, a touch key). , Mechanical keys, etc.). The voice data or the image data collected by the input unit 120 may be analyzed and processed as a control command of the user. In particular, the input unit 120 may generate various data regarding the situation information through various sensing devices in order to learn the state or behavior of the user of the function execution device 100. For example, it may refer to an image sensing device provided in a notebook or a smart device. Through an image sensing device provided in a notebook or a smart device, data about a face image and an iris image of a user, which can be viewed as one of the situation information, may be generated. In addition, the user may point to various devices for operating the function execution device 100. Some examples are mice or keyboards. The user may acquire data regarding behavior information of the user through various means for operating the function execution device 100. In addition, the present invention may refer to various sensors provided in a smart device or the like. Examples include gravity sensors, gyro sensors, illuminance sensors, and proximity sensors built into smart devices themselves. In the present specification, all the means are not described with respect to the input unit 120, but it may mean that all means for acquiring the situation information for analyzing the behavior of the user may be used.

The output unit 130 is to generate an output related to visual, auditory, or tactile, and may include a display unit 135. However, the present invention is not limited thereto, and may include at least one of a sound output unit, a haptic module, and a light output unit.

The display unit 135 forms a layer structure with or is integrally formed with the touch sensor, thereby implementing a touch screen. The touch screen may function as an input unit 120 providing an input interface between the function execution device 100 and the user, and at the same time, may provide an output interface between the function execution device 100 and the user.

The display unit 135 may include a touch sensor. The touch sensor applies a touch (or touch input) applied to the touch screen (or the display unit 135) using at least one of various touch methods such as a resistive film method, a capacitive method, an infrared method, an ultrasonic method, and a magnetic field method. Detect.

As such, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and then transmits the corresponding data to the controller 150. As a result, the controller 150 may determine which area of the display unit 135 is touched. Here, the touch controller may be a separate component from the controller 150 or may be the controller 150 itself.

Meanwhile, the controller 150 may perform different control or may perform the same control according to the type of the touch object that touches the display 135. Whether to perform different control or the same control according to the type of touch object may be determined according to the operation state of the current function execution device 100 or an application program being executed.

When the display unit 135 is implemented as the touch screen, the touch screen may detect a touch input using a touch pen, a stylus pen, a pointer, etc., in addition to a touch input input through a user's finger. have. In addition, the touch screen may also detect a proximity touch gesture of a predetermined distance from the touch screen, in addition to the touch input input while being in contact with the surface of the touch screen. To this end, the touch screen may further include a proximity sensor.

The storage unit 140 may store data supporting various functions of the function execution device 100. The storage 140 may store a plurality of application programs or applications that are driven by the function execution apparatus 100, data for operating the function execution apparatus 100, and instructions. At least some of these applications may be downloaded from an external server via wireless communication. In addition, at least some of these applications may be installed on the function execution device 100 from the time of shipment for the basic functions of the function execution device 100 (for example, a call forwarding, an outgoing function, a message reception, a callout function, and the like). May exist. Meanwhile, the application program may be stored in the storage 140 and installed on the function execution device 100 to be driven by the controller 150 to perform an operation (or function) of the function execution device.

The storage 140 may temporarily or non-temporarily store data transmitted from an external device, input by a user, or an operation result of the controller 150. The storage unit 140 may include a nonvolatile memory such as a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, or the like. And any well-known computer readable recording medium. Meanwhile, although the storage 140 is illustrated in a separate configuration in FIG. 1, the storage 140 may be integrated with the controller 150 to be described later and provided in the function execution device.

The controller 150 controls the overall operation of each component. The controller 150 may include a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), or any type of processor well known in the art. In addition, the controller 150 may perform an operation on at least one application or program for executing a method according to some embodiments.

Meanwhile, in the description of FIG. 1, the communication unit 110, the input unit 120, the output unit 130, the storage unit 140, and the controller 150 have been described as the configuration of the function execution device 100. According to an exemplary embodiment, the function execution device 100 may include only the input unit 120, the display unit 135, and the controller 150.

2 is a view for explaining context information referred to in some embodiments of the present invention.

As shown in FIG. 2, the function execution device 100 may include a camera 101, a multifunction sensor 103, a camera 105, a mouse 107, and a keyboard 109 provided in a laptop. Through this, situation information can be generated.

The cameras 101 and 105 provided in the laptop and the smart device may receive information about the presence, movement, and facial image of a user around the function execution device 100. In particular, a camera equipped with a 3D sensor and implementing iris recognition and face recognition may be provided in a notebook and a smart device to acquire various situation information of a user.

The multifunction sensor 103 may acquire various contextual information in addition to the information about the user itself. Examples of the multifunction sensor 103 include a gravity sensor, a gyro sensor, an illuminance sensor, a proximity sensor, and the like. The user behavior information may be supplemented by using various sensors as described above.

Examples of user input devices include a mouse 107 and a keyboard 109. The user's manipulation information may be obtained through the user input device.

In addition, although not shown in FIG. 2, context information may be obtained by referring to properties and information of an application currently being activated through various input gestures and a control unit by the touch input device.

The contextual information referred to in some embodiments of the present invention includes information about a user's face sensed from an image device mounted on a computing device, manipulation information received from a user input device, and activation information of an application running on the computing device. It may include at least one. The information on the user's face may include information using feature points of the user's eyes, and the eyes of the user may be grasped using an iris recognition technique.

Hereinafter, a method of selecting a location of an application execution screen and a method of learning location selection of an application execution screen will be described in detail.

3 is a flowchart illustrating a method for selecting a location of an application execution screen according to an embodiment of the present invention. However, this is only a preferred embodiment for achieving the object of the present invention, of course, some steps may be added or deleted as necessary.

In operation S100, a position of an initial execution region of an application graphical user interface (GUI) is obtained. The application GUI may mean a task window in which an application is executed in the screen area or a result of executing the application.

In operation S200, context information for correcting the position of the first execution region is received. The context information refers to information for analyzing and predicting a user's behavior for providing a user with a changed location of an execution region. As described above, the context information indicates context information associated with a user received through various input devices.

In operation S300, the position of the area for displaying the application GUI is corrected based on the situation information. This is to move the display position of the previously executed application GUI by analyzing the received situation information. An application that the user has performed previously may be displayed in an optimal position based on analysis of situation information to provide convenience. Detailed processes for correcting the position of the area for displaying the application GUI will be described in detail with reference to FIGS. 4 and 5 below.

In operation S400, the application GUI is displayed regardless of a user operation input at an area position according to the correction result. The location of the area where the application GUI displayed according to step S400 is displayed may be either a location where the user wishes to display the application GUI or a location where the user does not want the application GUI to be displayed.

For example, it is assumed that the screen area of the display device is divided into a left area and a right area, respectively. If the function execution device analyzes the user's situation information and the analysis result indicates that the user is looking at the right side of the screen area or performs the document editing operation using the document editing application in the right side, the user's application execution is executed. In response to the input, the document editing application task window will be automatically displayed in the right area without user's manipulation or setting. As another example, it is assumed that a user is provided with information of an application which is already running in the left area. In this case, the user may not want to block the screen of the previously running application. In this case, in response to the user's new application execution input, a new application task window is automatically displayed in the right region without the user's manipulation or setting.

In one embodiment, correcting the position of the region for displaying the application GUI based on the context information may be changed based on the position of the first execution region of the application GUI based on the operation information received from the user input device. The method may include determining a location and determining the changed location as a location for displaying the application GUI. According to the present embodiment, a new application GUI execution position may be determined at a position desired by the user based on the past behavior of the user.

4 to 5 are detailed flowcharts for describing a detailed process of step S300 of FIG. 3. However, this is only a preferred embodiment for achieving the object of the present invention, of course, some steps may be added or deleted as necessary.

Prior to explaining the detailed process of the step S300 in detail, it is noted that the detailed process can be classified in the following two ways. The first method is a positive method, in which an application GUI is displayed at a location intended by a user. Second, in a negative manner, the application GUI is displayed in an area except an area not intended by the user.

4 is a flowchart illustrating a positive position correction method referenced in some embodiments of the present invention. Hereinafter, the positive position correction method will be described in detail with reference to FIG. 4.

In operation S302, the gaze area of the user, which is a part of the screen area capable of displaying the application GUI, is determined. The gaze area corresponds to a part of the screen area displayed on the display device which the user actually watches. The determination of the gaze area may be performed through facial recognition or iris recognition through the cameras provided in the above-described notebook and smart devices.

In operation S304, at least one of activation of an application in the gaze area and a user input for the application is sensed. If the user is looking at the gaze area that is part of the screen area and an application is activated in the gaze area or if a user input for the application occurs, it will be clear that the user is actually aware of the application GUI through the gaze area. will be.

When at least one of activation of an application and a user input to the application is sensed (S306), it may be determined that the user is actually watching the gaze area (S308), and the gaze area may be determined as an area for displaying an application GUI. The position may be determined (S310). At this time, based on the determination result may be applied in step S300 of FIG.

On the contrary, if both the activation of the application and the user input for the application are not sensed (S306), it is determined that the user is not actually looking at the gaze area, no determination is made, and the correction at step S300 is not performed. will be.

In an embodiment, correcting a position of an area for displaying the application GUI based on context information includes determining a gaze area of a user that is a part of a screen area capable of displaying the application GUI, Detecting at least one of activation of an application in the gaze area and a user input to the application; and determining the gaze area as a location of an area for displaying the application GUI according to the detection result. have.

FIG. 5 is a flowchart illustrating a negative position correction method referenced in some embodiments of the present invention. Hereinafter, the negative position correction method will be described in detail with reference to FIG. 5.

In operation S322, an execution region of at least one application that is previously executed is determined in a screen region capable of displaying an application GUI. The present embodiment is for preventing the application GUI from being displayed where the user does not want to be. The important criterion may be whether or not the previously running application is activated.

In operation S324, a user's manipulation input for at least one application previously executed in the execution region is sensed. Even if a plurality of applications are running in the screen area, the application which is actually used or becomes the main application will be one application. In this step, it is to determine the activation of the application and at the same time detect an operation input of a separate user so that the execution region in which the user is working is not disturbed by another application GUI.

When the user's operation input is detected (S326), the non-execution area except the execution area of the screen area is determined as a position for displaying the application GUI (S328). As a result of clearly determining that the non-execution area is the execution area in steps S322 to S326, the non-execution area corresponds to an area that does not disturb a user's work even when a new application GUI is displayed. However, it should be noted that steps S324 to S326 may be omitted. As illustrated in FIG. 5, after determining the execution region, the execution region may be determined as a position for displaying a new application GUI in a non-execution region other than the execution region. In this case, as a result of some processes being omitted, a quick judgment can be made, and a new application GUI can be displayed immediately in an empty space of the screen area.

On the contrary, if the user's manipulation input is not detected (S326), both the execution region and the non-execution region are determined as positions for displaying the application GUI, and the correction in step S300 will not be performed.

In one embodiment, correcting the position of an area for displaying the application GUI based on context information may include determining an execution area of at least one application GUI running in the screen area, and among the screen areas. And determining the non-execution area except the execution area as a location for displaying the application GUI.

In another embodiment, determining the non-execution area other than the execution area as a location for displaying the application GUI, when a display area for executing a virtual input interface is detected in a part of the screen area, The method may include determining an area except for a display area for executing a virtual input interface as a location for displaying the application GUI. The display area for executing the virtual input interface refers to all interfaces for inputting / outputting information such as a virtual keyboard and a message input area.

Meanwhile, hereinafter, various examples in which the application GUI is displayed in the screen area and various results in various display devices will be described.

6 is a diagram illustrating an example of selecting a location where an application GUI is displayed according to an embodiment of the present invention. Although the screen region 10 of FIG. 6 is divided into four divided regions, this is for convenience of description and may be divided into at least two regions to display an application GUI by applying the example described below. Please note.

As shown in the upper part of FIG. 6, through the sensing device 101 provided in the first display device 1001, the gaze area 11, which is part of the screen area 10 that the user 1 watches, is actually When it is determined, the user input 102 in the gaze area 11 is sensed. According to the detection result, an application GUI may be displayed on the target area 12.

On the other hand, as shown in the lower portion of FIG. 6, through the sensing device 101 provided in the first display device 1001, the gaze area 11 that is actually a part of the screen area 10 that the user 1 watches. ) Is determined, and it is assumed that the application GUI that is already executed is being executed in the target area 12 (not shown). At this time, the user 1 would like the new application GUI not to be displayed in the target area 12. At the same time, when the user input 102 is detected in the gaze area 11 (eg, when a mouse manipulation input is detected), it is assumed that the user 1 is expected to work in the gaze area 11. Could be. In this case, a new application GUI may be displayed in the remaining screen areas except for the target area 12. Although not shown at the bottom of FIG. 6, a new application GUI may be displayed on the left side or the upper side adjacent to the gaze area 11, which is an area in which the target area 12 is excluded.

FIG. 7 illustrates an example of selecting a location at which execution of an application is displayed according to another exemplary embodiment of the present invention.

As illustrated in FIG. 7, it may be confirmed that an application for a document task is running in the screen area 10 of the first display device 1001. In this case, the work application screen area may be divided into a first area 13 which is an information input / output area and a second area 14 in which the information input / output area is excluded. Previously, when a user ran a new application while working on a document, the application GUI was displayed in an arbitrary location. If the executed application GUI is displayed at a position coincident with at least a portion of the first area 13, the user will interfere with the document work currently in progress. Therefore, in the present embodiment, a new application GUI can be displayed in the second area 14 in order to prevent such interference. This may be displayed in consideration of whether the previously running application is activated and the position and size of the second area 14.

In an embodiment, the correcting the position of the region for displaying the application GUI based on the context information may include an information input / output region, among application execution regions that are already included in at least a portion of the screen region. And determining a location for displaying the execution of the application according to the determined region, and displaying the application GUI regardless of a user's operation input at the location of the region according to the correction result. And displaying the application GUI using the region where the input / output region is excluded as the new position. Accordingly, a new application GUI having a size adjusted in the second area 14 in which the information input / output area is excluded may be executed and displayed.

8 is a diagram for describing an example of selecting a position at which execution of an application is displayed according to another exemplary embodiment of the present invention.

As illustrated in FIG. 8, it may be confirmed that the presentation application is running in the screen area of the first display device 1001. Although not shown in FIG. 8, when the display device is a dual monitor having two or more connected to a single computing device, the user may not want the new application GUI to be displayed on the first display device 1001 on which the presentation application is running. . Accordingly, by detecting whether the presentation application is activated or not, the position may be corrected such that a new application GUI is displayed on a display device other than the first display device 1001.

As a similar example, it may be considered that there is no user near the display device. At this time, if it is determined that the user is absent near the display device, the new application GUI does not need to be displayed.

In one embodiment, correcting the position of the area for displaying the application GUI based on context information may not display the application GUI when the user is not sensed from an image device mounted in the computing device. And determining the display of the application GUI regardless of a user's operation input at an area position according to the correction result. Restricting display for execution of the application in the screen area according to the determination. It may include.

In another embodiment, correcting the position of the region for displaying the application GUI based on the context information may include displaying execution of the application according to attributes of an application which is different from the application in the screen region. And limiting the screen area in an area for the display, and displaying the application GUI at an area position according to the correction result irrespective of a user's operation input, according to the result of the restriction. The method may include assigning a priority to the application GUI display. For example, if a high priority is given to an already running application, a new application GUI can be prevented from being displayed even if a new application is executed.

9 is a diagram for describing an example of selecting a position at which execution of an application is displayed on a multi-screen according to an embodiment of the present invention.

As shown at the top of FIG. 9, the multi-screen area is displayed on the first cover device 1001 and the second display device 1002. Through the sensing device 101 provided in the second display device 1002, it is determined that the screen area actually watched by the user 1 is the gaze area 11 that is part of the screen area of the first display device 1001. If so, the user input 102 in the gaze area 11 is sensed. According to the detection result, the application GUI may be displayed on the target area 12 that is a part of the screen area of the first display device 1001.

Meanwhile, as shown in the lower part of FIG. 9, through the sensing device 101 provided in the second display device 1002, the gaze area 11 that is a part of the screen area that the user 1 actually watches is determined. It is assumed that an application GUI that is already running is being executed in the target area 12 that is a part of the screen area of the first display device 1001 (not shown). At this time, the user 1 would like the new application GUI not to be displayed in the target area 12. At the same time, when the user input 102 is detected in the gaze area 11 (eg, when a mouse manipulation input is detected), it is assumed that the user 1 is expected to work in the gaze area 11. Can be. In this case, a new application GUI may be displayed in the remaining screen areas except for the target area 12. In this case, a new application GUI may be displayed on the second display device 1002 as well as the screen area of the first display device 1001 in which the target area 12 is excluded.

In an embodiment, the obtaining of the position of the first execution region of the application GUI may include: a screen region of the display device divided into at least a first screen region and a second screen region, wherein the first screen region and the second screen region Acquiring an initial execution position, which is any one, and correcting a position of an area for displaying the application GUI based on the situation information comprises: the application GUI according to operation information received from a user input device. Confirming the changed position from the initial execution position of and correcting according to a result of determining whether the gaze of the user matches the changed position based on face information of the user received from the image device mounted in the computing device. And using the application GUI at an area position according to the correction result. The displaying of the user GUI may include displaying the application GUI using the changed position as a new position regardless of the user's manipulation input.

The example described in FIG. 9 may be implemented in a foldable smartphone in which multiple screens are essentially implemented (not shown). Additional examples according to the characteristics of the foldable smartphone will be described with reference to FIG. 11 below.

FIG. 10 is a diagram for describing an example of selecting a position at which execution of an application is displayed on a mobile screen according to an embodiment of the present invention. Although the screen area 10 shown in FIG. 10 is divided into two divided areas, this is for convenience of description and the application GUI may be displayed by being divided into at least two areas by applying the example described below. Be careful.

As shown in the upper portion of FIG. 10, the screen area 30 of the smartphone display device 1003 may be divided into a first area 31 and a second area 32. Through the sensing device 101 provided in the smartphone display device 1003, it is determined that the gaze area gazed by the user 1 is actually the first area 31, and the user ( When it is detected that the touch input of 1) is frequently generated, a new application GUI 33 may be displayed on the first area 31 to provide convenience for the user 1 to immediately perform a task.

On the other hand, as shown at the bottom of FIG. 10, it is determined that the gaze area gazed by the user 1 is actually the first area 31, and an application currently running in the first area 31 is displayed or It is assumed that touch input occurs frequently. At this time, since the user 1 is performing the work through the first area 31, the user 1 will not want to be disturbed because the new application GUI is displayed in the first area 31. Accordingly, unlike the upper case of FIG. 10, the new application GUI 33 may be displayed in the second area 32 instead of the first area 31. Although not shown in FIG. 10, when a display area for executing a virtual input interface such as a virtual keyboard is detected in the first area 31, a new application is applied to the second area 32 so as not to disturb the input of the user 1. GUI 33 may be displayed automatically.

FIG. 11 is a diagram for describing an example of selecting a position at which execution of an application is displayed in the foldable display 1004 according to an exemplary embodiment.

As illustrated in FIG. 11, the screen area of the foldable display device 1004 may be divided into a first area 41 and a second area 42. In this case, due to the nature of the foldable, when the foldable display device 1004 is unfolded, the first area 41 and the second area 42 are simultaneously provided to the user 1 and the foldable display device 1004. When is folded, at least one of the first area 41 and the second area 42 may be out of view of the user 1. In this example, the new application GUI is displayed by reflecting the above-described characteristics of the foldable.

When the foldable display 1004 is unfolded, the example described with reference to FIG. 9 may be applied.

In an embodiment, the method performed by the foldable electronic device may include a first display area for displaying execution of at least one first application and a second display area for displaying execution of at least one second application. Detecting the display area to be the user's watched object; and if it is determined that the display area is the first display area according to the sensing result, displaying the second application on the first display area. It may include the step. This is to allow a new application GUI to be displayed on one screen area that the user watches on the display of the foldable electronic device divided into at least two screen areas.

In another embodiment, the displaying of the second application on the first display area may include displaying the second application on the first display area according to a result of determining that the first application is running on the first display area. It may include the step of displaying. This is to allow a new application GUI that can be displayed on a screen that is not watched by the user to be displayed as a screen area that the user is watching.

In another embodiment, the displaying of the second application on the first display area may include: a new application GUI in the first display area according to a result of determining that the first application is running in the first display area. Limiting the display of may include. This is to prevent a new application GUI to be executed when the user watches the first display area from covering the first display area.

In another embodiment, the detecting of any one display area that is a gaze of a user may include: facing the user among the first display area and the second display area when the foldable electronic device is in a folding state. The sensing device may include a display area. In this case, if the display area facing the user is the first display area, the second display area may correspond to the display area that is out of the user's field of view. In this case, the display position of the second application which is being executed or is scheduled to be executed may be moved to the second display area.

On the other hand, it is assumed that the foldable display device 1004 is folded. In this case, only at least one of the first area 41 and the second area 42 may be a screen area visible to the user 1. As illustrated in FIG. 11, it may be determined that the screen area gazed by the user 1 is the first area 41 through the sensing device 101 provided in the foldable display device 1004. Since the screen area visible to the user 1 is exclusive due to the characteristics of the foldable display device 1004, it is relatively simple and accurate to determine the screen area that the user 1 watches. When it is determined that the first area 41 is the gaze area, the display area of the new application GUI may be determined according to the property of the application running in the first area 41. For example, if the application to be newly executed is an application that is not used immediately by the user 1, it does not need to be placed in the first area 41. In this case, the application to be newly executed is running in the first area 41. In order to ensure the persistence of the work, a new application GUI may be displayed in the second area 42.

Although not shown in FIG. 11, the following cases may be assumed. When the folding state of the foldable display device 1004 is at right angles or is close to the right angle, the user 1 may be watching only one of the first area 41 and the second area 42. In this case, in determining the area to be watched, a multifunctional sensor (eg, a gravity sensor, a gyro sensor, an illumination sensor, a proximity sensor, etc.) included in the foldable display device 1004 may be utilized. For example, if the multi-function sensor detects that the first area 41 is closer to the ground than the second area 42, the first area 41 is actually watching the user 1. It may be determined that the screen area. Accordingly, in determining which of the first area 41 and the second area 42 to display the new application GUI, it may be a reference object.

On the other hand, in the process of selecting the position of the application GUI display described above, it is very important to determine the gaze area that the user is watching from the screen area. At this time, the user's eyes can be easily identified by analyzing image data acquired through the sensing device, but a new learning method is needed to identify not only the user's gaze but also the area of the screen. Do. In addition, the gaze area of the user may be moved even within a short time. The learning method according to some embodiments described below attempts to determine the region of interest of the user in order to solve the problems mentioned and to distinguish the region that the user actually intends or does not intend.

12 is a flowchart illustrating a positioning learning method of an application execution screen according to an embodiment of the present invention. Description of the present embodiment will be described with reference to FIG.

The first display area 110, which is a part of the screen area of the display device and which matches the visual direction of the user 1, is checked (S202), and whether or not at least one application is executed in the first display area 110 is determined. It is determined (S204).

It is determined whether at least one application is running in the first display area 110 (S206), and when a user manipulation input 102 for the application is detected (S208), the first display area 110 is user-defined. Determine as the region of interest of (S210). This may accurately determine the area intended by the user 1 based on whether the application is activated in the area that matches the gaze of the user 1 and whether there is an active manipulation of the user.

If the application is not running in the first display area 110 (S206) or is running but there is no user operation input for the application (S208), the first display area 110 is not the region of interest of the user 1. It can be determined.

In an embodiment, the determining of the first display area 10 as the ROI of the user 1 may include determining the first display area 110 when the user manipulation input for at least one application is detected. And determining the region of interest.

14 is a flowchart illustrating a positioning learning method of an application execution screen according to another exemplary embodiment of the present invention. A description of this embodiment will be described with reference to FIG. 15 together.

The second display area 120, which is a part of the screen area of the display device and does not match the direction of the eye of the user 1, is checked (S222), and at least one application is executed in the second display area 120. It is determined whether or not (S224).

It is determined whether at least one application is running in the second display area 120 (S226), and it is detected that the direction of the user manipulation input 120 is directed to the second display area 120 (S228). This is to predict the area where the user will work in the near future.

If at least one application is running in the second display area 120 and a movement according to a user's manipulation input to the second display area 120 is detected, the second display area 120 is moved to the user's interest area. Determine (S230).

If the application is not running in the second display area 120 or if the user's manipulation input 102 is not detected at all, it may be determined that the second display area 120 is not the user's area of interest. (S232).

In an exemplary embodiment, the determining of the second display area 120 as the ROI of the user 1 may include determining the second display area 120 based on the manipulation information received from the user input device. And determining, wherein the manipulation information indicates a moving direction in the screen area. For example, when it is determined that an application is running in the second display area 120 and the direction of the mouse toward the second display area 120 is directed to the second display area 120, the second display area 120 is displayed. ) May be determined as a user's region of interest.

Each of the steps described with reference to FIGS. 12 and 13 may be performed once to determine the ROI of the user. However, as a result of the repeated steps, the pattern of the ROI of the user may be confirmed. Accordingly, the ROI of the user can be accurately determined in a short time.

The concepts of the present invention described above with reference to FIGS. 1 through 15 may be implemented in computer readable code on a computer readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer equipped hard disk). Can be. The computer program recorded on the computer-readable recording medium may be transmitted to another computing device and installed in the other computing device through a network such as the Internet, thereby being used in the other computing device.

Although the operations are shown in a specific order in the drawings, it should not be understood that the operations must be performed in the specific order or sequential order shown or that all the illustrated operations must be executed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various configurations in the embodiments described above should not be understood as such separation being necessary, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. It should be understood.

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it is to be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims (19)

A method performed by a computing device,
Obtaining a location of an initial execution region of an application graphical user interface (GUI);
Receiving context information for correcting the position of the first execution region, and correcting the position of the region for displaying the application GUI based on the context information; And
Displaying the application GUI at an area position according to the correction result irrespective of a user's operation input.
How to select the location of the application execution screen. According to claim 1,
The situation information,
And at least one of information about a user's face sensed from an image device mounted in the computing device, operation information received from a user input device, and activation information of an application running on the computing device.
How to select the location of the execution screen of the application. According to claim 1,
Correcting the position of the area for displaying the application GUI based on the situation information,
Determining a gaze area of a user that is a part of a screen area capable of displaying the application GUI;
Sensing at least one of activation of an application in the gaze area and a user input to the application; And
Determining the gaze area as a location of an area for displaying the application GUI according to the detection result;
How to select the location of the application execution screen. According to claim 1,
Correcting the position of the area for displaying the application GUI based on the situation information,
Determining an execution region of at least one application GUI running in the screen region; And
Determining a non-execution area of the screen area other than the execution area as a location for displaying the application GUI;
How to select the location of the application execution screen. The method of claim 4, wherein
The determining of the non-execution area except the execution area as a location for displaying the application GUI may include:
If a display area for executing the virtual input interface is detected in a part of the screen area, determining an area except the display area for executing the virtual input interface as a location for displaying the application GUI; ,
How to select the location of the application execution screen. According to claim 1,
Correcting the position of the area for displaying the application GUI based on the situation information,
Determining the changed position based on the position of the first execution region of the application GUI based on the operation information received from the user input device; And
Determining the changed location as a location for displaying the application GUI,
How to select the location of the application execution screen. According to claim 1,
Acquiring the position of the first execution region of the application graphical user interface (GUI),
The screen area may be divided into at least a first screen area and a second screen area, and the method may include obtaining an initial execution position which is one of the first screen area and the second screen area.
Correcting the position of the area for displaying the application GUI based on the situation information,
A position changed from the initial execution position of the application GUI according to the operation information received from the user input device, and the gaze of the user is changed based on the face information of the user received from the image device mounted in the computing device And correcting according to a result of determining whether
The displaying of the application GUI at an area position according to the correction result irrespective of a user's operation input may include
Displaying the application GUI using the changed location as a new location;
How to select the location of the application execution screen. According to claim 1,
Correcting the position of the area for displaying the application GUI based on the situation information,
Determining a position for displaying the execution of the application according to a result of determining the region in which the information input / output region is excluded from the previously executed application display region included in at least a portion of the screen region;
The step of displaying the application GUI regardless of the user's operation input in the area position according to the correction result,
Displaying the application GUI using an area in which the information input / output area is excluded as the new location;
How to select the location of the application execution screen. According to claim 1,
Correcting the position of the area for displaying the application GUI based on the situation information,
Limiting the screen area in an area for displaying execution of the application according to an attribute of a previously running application different from the application in the screen area,
The displaying of the application GUI at an area position according to the correction result irrespective of a user's operation input may include:
Prioritizing the application GUI display based on the previously running application GUI according to a result of the restriction;
How to select the location of the application execution screen. According to claim 1,
Correcting the position of the area for displaying the application GUI based on the situation information,
Determining not to display the application GUI when a user is not sensed from an image device mounted in the computing device,
The displaying of the application GUI at an area position according to the correction result irrespective of a user's operation input may include
Limiting the display for execution of the application in the screen area according to the determination;
How to select the location of the application execution screen. Identifying a first display area which is a part of the screen area of the display device, and which is an area corresponding to the user's gaze direction;
Determining whether to execute at least one application in the first display area; And
If it is determined that the at least one application is running on the first display area according to the determination result, determining the first display area as the ROI of the user.
How to learn positioning of application execution screen The method of claim 11, wherein
The determining of the first display area as the ROI of the user may include:
Determining the first display area as an ROI of a user when a user manipulation input for the at least one application is detected;
How to learn positioning of application execution screen. Identifying a second display area which is a part of the screen area of the display device, and is an area which does not match the direction of the user's gaze;
Determining whether to execute at least one application in the second display area; And
If it is determined that the at least one application is running in the second display area according to the determination result, determining the second display area as the ROI of the user.
How to learn positioning of application execution screen. The method of claim 13,
The determining of the second display area as the ROI of the user may include:
Determining the second display area as a user's region of interest based on operation information received from a user input device, wherein the operation information indicates a direction of movement in the screen area;
How to learn positioning of application execution screen. In the method performed by the foldable electronic device,
Detecting any one display area of a user's gaze among a first display area for displaying execution of at least one first application and a second display area for displaying execution of at least one second application;
Displaying the second application on the first display area when it is determined that the one display area is the first display area according to the detection result.
How to select the location of the application execution screen. The method of claim 15,
The displaying of the second application on the first display area may include:
Displaying the second application on the first display area according to a result of determining that the first application is running on the first display area.
How to select the location of the application execution screen. The method of claim 15,
The displaying of the second application on the first display area may include:
Limiting display of a new application in the first display area according to a result of determining that the first application is running in the first display area;
How to select the location of the application execution screen. The method of claim 15,
Detecting any one display area that is the user's attention target,
When the foldable electronic device is in a folded state, detecting a display area facing the user among the first display area and the second display area, wherein the display area facing is a first display area;
How to select the location of the application execution screen. The method of claim 18,
The displaying of the second application on the first display area may include:
Moving the display position of the second application to the second display area, wherein the second display area is out of view of the user;
How to select the location of the application execution screen.

Images