JP2021165923A - Information processing device, computer program and information processing method - Google Patents

Abstract

To improve operability in a multi-window user interface.SOLUTION: A tablet terminal 100 arranges a code window for displaying a programming code on the front, and arranges an execution window for displaying a program execution screen on the back of the code window. When performing data input after a user instructs a focus change in a function determination area, the data input to the execution window is allowed while keeping the code window on the front of the execution window.SELECTED DRAWING: Figure 1

Description

The present invention relates to a multi-window user interface technology.

With the advent of mobile computers such as laptop PCs, tablet PCs, and smartphones, the opportunities to use computers have increased dramatically. Mobile computers are required to be lightweight and small in size. Therefore, in the user interface design of a mobile computer, how much information can be efficiently input / output to a monitor screen of a limited size is an important point.

In general, a multi-window user interface that simultaneously displays a plurality of pieces of information in a plurality of windows (areas for displaying images) is often adopted. A "layer (display layer)" is set for each window. Multiple windows are displayed as if all or part of the back (lower layer) window is hidden behind (lower) the front (upper layer) window. The user enters data in the foreground window. Hereinafter, the window to be input data is referred to as a "focus window".

JP-A-2019-168717

When the window where you want to enter information is on the back side, you need to change the display order so that this window is the front layer (hereinafter referred to as "front"). Specifically, the rear window is brought to the front by reducing (deactivating) the front window. Alternatively, after performing an operation to display reduced versions of a plurality of windows in parallel, the focus window (the window in which information is to be input) may be selected from the operations. The operation for fronting, in other words, the time and effort required for fronting, tends to cause deterioration of workability when you want to input information while changing the focus window frequently.

An object of the present invention is to provide a technique for improving operability in a multi-window type user interface.

In the information processing apparatus according to an aspect of the present invention, a display control in which a first area for displaying a first image is arranged on the front surface and a second area for displaying a second image is arranged on the back surface of the first area. It includes a unit, a first operation for designating a second area, and an operation detection unit for detecting a second operation for data input from the user.
When the second operation is detected after the detection of the first operation, the display control unit changes the second image based on the second operation while maintaining the first area in front of the second area.

According to the present invention, it becomes easy to improve the operability in the multi-window type user interface.

It is an external view of a tablet terminal. It is a schematic diagram for demonstrating the layer structure of a plurality of windows. It is a hardware configuration diagram of a tablet terminal. It is a functional block diagram of a tablet terminal. It is an external view for demonstrating the structure of the function determination area. It is a schematic diagram for demonstrating a normal input. It is a schematic diagram for demonstrating the penetration input. It is a flowchart which shows the processing process when the touch to the targeting selection area is detected. It is a flowchart which shows the processing process when the touch to other than the selection area is detected. It is a flowchart which shows the processing process when a detachment is detected. It is an external view for demonstrating the switching of a data input target. It is a 1st screen view which shows the state at the time of the penetration input to the execution window. It is a 2nd screen figure which shows the state at the time of a normal input to a code window. It is a 3rd screen view which shows the state at the time of a normal input to a code window. It is a 4th screen view which shows the state when a circle is drawn in the execution window. It is a screen view when the explanation of the function input is displayed. It is a screen diagram when inputting a gesture. It is a conceptual diagram for demonstrating the focus control of 3 or more windows.

In this embodiment, a plurality of windows are arranged and displayed on a plurality of layers. A window is an image area that imitates one sheet, and a layer is a concept that indicates the stacking position of sheets when a plurality of sheets are stacked.

First, assume physical paper sheets 1, 2, and 3. Information such as pictures and characters is described on sheets 1, 2, and 3, respectively. Let's stack these three sheets in order. It is assumed that the sheet 1 is arranged at the top, the sheet 2 is arranged in the middle, and the sheet 3 is arranged at the bottom. At this time, the user can see all of the sheet 1 at the top. In the second sheet 2 from the top, only the part that does not overlap the sheet 1 can be seen. In the third sheet 3 from the top, only the portion that does not overlap with any of the sheets 1 and 2 can be seen. A multi-window system realizes the same visibility as such stacked sheets by a computer program.

In a multi-window system, electronic windows 1, 2 and 3 are assumed in the same manner. Windows 1, 2 and 3 display information in the same way as a sheet. At this time, the "top" layer corresponding to the sheet 1 is called the "front layer". "Front" is an expression that means the front side (looks like it is) when viewed from the user who sees the monitor. Try arranging the three windows on each of the three layers. Suppose that window 1 is placed on the top layer, window 2 is placed on the middle layer, and window 3 is placed on the bottom layer. In this case, it is expressed that the window 1 is arranged in the layer "front" rather than the windows 2 and 3. Similarly, the window 2 is said to be arranged on the "back" layer of the window 1 and on the "front" layer of the window 3. It is expressed that the window 3 is arranged on the "backmost" layer.

The method of arranging on each of a plurality of layers of each of a plurality of windows, such as windows 1, 2, and 3, is called "display order". According to the above example, it is expressed as "the display order of a plurality of windows is windows 1, 2, and 3 in order from the front." Also, changing the layer of the window when the display order is changed is called "rearrangement" of the window.

In the following, "fronting" means to rearrange the layer of the specified window to the front in the multi-window type user interface. "Targeting" means setting the specified window as the target for data entry, in other words, setting the specified window as the focus window. The "data input" here means an input for adding or modifying the content displayed by the window, specifically, changing a character or an image. "Activation" means turning on a special feature layer. In the general multi-window method, "fronting" and "targeting" are substantially the same concept, but in the present embodiment, "targeting without fronting" is realized.

FIG. 1 is an external view of the tablet terminal 100.
The tablet terminal 100 (information processing device) is a thin mobile computer provided with a monitor screen 102. A touch panel is installed on the monitor screen 102. In this embodiment, a lesson that teaches programming to a user will be described. The student becomes a user of the tablet terminal 100.

Application software for programming education (hereinafter referred to as "learning software") is pre-installed on the tablet terminal 100. The purpose of the lesson is to deepen the user's understanding of the mechanism of software by having the user write a simple program code (hereinafter, simply referred to as "code") and execute the created program. The details will be described later, but the window for creating code and the window for executing the program are displayed separately as a multi-window.

Hereinafter, the description will be made on the premise of a touch panel, but the tablet terminal 100 can be connected to an input device such as a keyboard and a mouse in addition to the touch panel. The tablet terminal 100 may enable voice input.

FIG. 2 is a schematic diagram for explaining the layer structure of a plurality of windows.
The learning software includes a code window X1 (first area) and an execution window X2 (second area) as "normal layers" for data input and output. Further, a function determination area M1 and a gesture area G1 are included as "function layers" for input support for the code window X1 and the execution window X2. Both functional layers are invisible. From the frontmost (topmost), four layers of the function determination area M1, the gesture area G1, the code window X1 and the execution window X2 are arranged. The code window X1 and the execution window X2 are windows that serve as work areas for the learning software. Normally, the code window X1 is the focus window. The function determination area M1 is always enabled, and the gesture area G1 is normally disabled.

The function determination area M1 is located in the foreground. The user can change the execution window X2 to a focus window by accessing a selection area formed in a part of the function determination area M1 (details will be described later). Further, the user can activate the gesture area G1 by accessing another selection area formed in a part of the function determination area M1. In the following, the state in which the code window X1 is the focus window is the "normal mode", and the execution window X2 whose display area is blocked because it is located in the layer below the code window X1 is the focus window. The state is called "penetration mode (special mode)". Further, the state in which the gesture area G1 is enabled is called "gesture mode". There are three modes of learning software, which are exclusive to each other.

The gesture area G1 is located behind the function determination area M1. Gesture area G1 is invisible and is usually not enabled. When the gesture area G1 is activated by the method described later, the user can input various gestures using the entire monitor screen 102. The details of the gesture input will be described later in relation to FIG.

The code window X1 is displayed semi-transparently and provides a screen (first image) for writing the code. The semi-transparent display here means that the image of the code window X1 can be visually recognized, and the transparency of the code window X1 is adjusted and displayed so that the image on the back surface can also be seen through the code window X1. To say. The code window X1 is located behind the gesture area G1, but is usually the foreground in the layer. In normal mode or gesture mode (when both code window X1 and execution window X2 are displayed), code window X1 is the focus window.

The execution window X2 is opaquely displayed (normally displayed) like a general window. The execution window X2 provides an execution screen (second image) of the program created in the code window X1. The execution window X2 is located behind the code window X1. Since the code window X1 is translucent, the user can see not only the code window X1 but also the execution window X2 behind it at the same time. When the mode is changed to the penetration mode, or when the visible layer window X1 in front of the execution window X2 does not appear, the execution window X2 is targeted, that is, the focus window.

In summary, it is as follows.
1. 1. Functional layer: Invisible 1-1. Function judgment area M1: Always valid. Accepts input for mode change.
1-2. Gesture area G1: It is invalid in the normal mode and the penetration mode, and is enabled by the function determination area M1 (gesture mode). Accepts gesture input.
2. Normal layer: Visible 2-1. Code window X1: Translucently displayed and targeted in normal mode. As will be described later, it may be temporarily completely transparent (invisible).
2-2. Execution window X2: Opaque and targeted in penetration mode.

The display order of the code window X1 and the execution window X2, in other words, the positional relationship between the layers may be interchangeable, but in the following, unless otherwise specified, the function determination area M1, the gesture area G1, the code window X1 and the execution window The display order (contextual relationship) of X2 will be described as not being interchanged.
Further, although it is possible to close the code window X1 and the execution window X2, in the following, unless otherwise specified, the code window X1 and the execution window X2 are not closed and are both visible in the normal mode. It is explained as.

FIG. 3 is a hardware configuration diagram of the tablet terminal 100.
The tablet terminal 100 has a built-in storage 312 as a non-volatile memory for storing a computer program, a volatile memory 304 for expanding programs and data, a register, an arithmetic unit, an instruction decoder, and the like, and reads and executes the program from the memory 304. The processor 300 (CPU: Central Processing Unit) and the like are included. The processor 300 is connected to a relatively high speed first bus 302. In addition to the memory 304, a NIC (Network Interface Card) is connected to the first bus 302. In addition, another device such as a GPU (Graphics Processing Unit) may be connected to the first bus 302.

The first bus 302 is connected to the second bus 310, which has a relatively low speed, via a bridge 308. In addition to the storage 312, an output device 316 such as a monitor screen 102 or a speaker is connected to the second bus 310. Further, an input device 314 such as a mouse or a keyboard and a peripheral device 318 such as a printer may be connected to the second bus 310.

FIG. 4 is a functional block diagram of the tablet terminal 100.
As described above, each component of the tablet terminal 100 is hardware including a CPU and a computing unit such as various co-processors, a storage device such as a memory and a storage, and a wired or wireless communication line connecting them. It is realized by software that is stored in a storage device and supplies processing commands to a processor. A computer program may be composed of a device driver, an operating system, various application programs located on the upper layers thereof, and a library that provides common functions to these programs. Each block described below shows a block for each function, not a configuration for each hardware.

The tablet terminal 100 includes a user interface processing unit 104, a data processing unit 106, a communication unit 126, and a data storage unit 108.
The user interface processing unit 104 accepts operations from the user via the touch panel, and is in charge of processing related to the user interface such as image display and audio output. The communication unit 126 is in charge of communication processing with other devices via the Internet. The data storage unit 108 stores various data. The data processing unit 106 executes various processes based on the data acquired by the user interface processing unit 104 and the communication unit 126 and the data stored in the data storage unit 108. The data processing unit 106 also functions as an interface for the user interface processing unit 104, the communication unit 126, and the data storage unit 108.

The user interface processing unit 104 includes an operation detection unit 110 that receives input from the user and an output unit 112 that outputs various information such as images and sounds to the user. The operation detection unit 110 includes a function input unit 114, a gesture input unit 116, and a data input unit 118. The function input unit 114 detects a function selection input (first operation) which is a predetermined input to the function determination area M1. The function selection input is an input for controlling the function or state of the window itself, not the content of the window itself, such as activation and objectification, but the details will be described later in relation to FIG. 5 and the like. The gesture input unit 116 detects a gesture input which is a predetermined input to the gesture area G1. The data input unit 118 detects data input (second operation) to the code window X1 or the execution window X2.

The data storage unit 108 stores the learning software and the program created by the user.

The data processing unit 106 includes a display control unit 120 that controls the display of the code window X1 and the execution window X2.
The display control unit 120 includes a display update unit 122 and a layer management unit 124. The display update unit 122 manages image generation and update in the code window X1 and the execution window X2. The layer management unit 124 manages the layer, position, and size of each window. The layer management unit 124 also manages modes related to the normal mode, the special mode, and the gesture mode.

FIG. 5 is an external view for explaining the configuration of the function determination region M1.
In the invisible function determination area M1, the selection area 128L1, the selection area 128R1, the selection area 128L2, and the selection area 128R2 (hereinafter, collectively referred to as "selection area 166") are set at the four corners of the monitor screen 102.

When the user touches the selected area 128L1 or the selected area 128R1, in other words, when the lower left corner or the lower right corner of the monitor screen 102 is touched, the layer management unit 124 shifts from the normal mode to the penetration mode (special mode). At this time, the execution window X2 on the back side is targeted. Hereinafter, of the selection area 166, when the selection area 128L1 and the selection area 128R1 for targeting the execution window X2 are collectively referred to as "targeting selection area 162".

When the user stops touching the target selection area 162 (detach), the layer management unit 124 returns from the penetration mode to the normal mode. At this time, the code window X1 on the front side is targeted again.

When the user touches the selected area 128L2 or the selected area 128R2, in other words, when the upper left corner or the upper right corner of the monitor screen 102 is touched, the layer management unit 124 shifts from the normal mode to the gesture mode. The layer management unit 124 activates the gesture area G1 and enables gesture input. Hereinafter, among the selection areas 166, the selection area 128L2 and the selection area 128R2 for activating the gesture area G1 are collectively referred to as "gesture activation area 164".

When the user stops touching the gesture activation area 164 (detach), the layer management unit 124 shifts from the gesture mode to the normal mode. At this time, the gesture area G1 is invalidated. Touching any of the four selection areas 166 is called "function selection input".

There are two types of data input in the present embodiment: "normal input" in the normal mode and "penetration input (special input)" in the penetration mode. Data input (mainly text or image input) is possible by touch input with a finger or stylus into the window area, but other input devices such as a mouse can also be used. In the normal mode, the code window X1 in front of the execution window X2 is the focus window, and the user inputs data to the code window X1. This is called "normal input".

When the user touches the targeting selection area 162, the penetration mode is set, the execution window X2 on the back side becomes the focus window, and the user inputs data to the execution window X2. This is called "penetration input".

At the time of penetration input, the layer management unit 124 does not change the context of the code window X1 and the execution window X2. In other words, in the penetrating input, even though the code window X1 is on the front side, a unique operation feeling is provided as if the code window X1 is penetrated and the execution window X2 on the back side is directly touched. .. This feeling of operation will be further described in relation to FIGS. 6 and 7 below.

FIG. 6 is a schematic diagram for explaining a normal input.
As described above, the layers visible to the user are two normal layers, the code window X1 and the execution window X2. The code window X1 is on the front side, and the execution window X2 is on the back side. Since the code window X1 is translucent, the user can see not only the code window X1 but also the execution window X2 on the back side at the same time. Therefore, the user can check the execution status of the program while writing or checking the program code.

In the normal mode, the code window X1 is the focus window. When the user touches the monitor screen 102, the data input unit 118 determines this as a data input (normal input) for the code window X1 (focus window). The user inputs data to the code window X1 by touch operation. The display update unit 122 displays the data input in the code window X1 according to the input result, or moves the cursor. You can also move the code window or scroll the character area.

FIG. 7 is a schematic diagram for explaining the penetration input.
When the user touches the targeting selection area 162 (selection area 128L1 or selection area 128R1), the mode shifts from the normal mode to the penetration mode. The layer management unit 124 sets the execution window X2 as the focus window instead of the code window X1. When the user touches the monitor screen 102 in this state, the data input unit 118 determines that this is a data input (penetration input) to the execution window X2. The user inputs data to the execution window X2 by touch operation. The display update unit 122 draws an image in the execution window X2 according to the input result. Further, the display update unit 122 updates the code window X1 in response to the result of image input to the execution window X2.

In the penetration mode, the user feels as if his or her finger penetrates the code window X1 on the front and operates the execution window X2 on the back. In the penetration mode, the display update unit 122 may temporarily hide (completely make) the code window X1. In this case, the user can input data while looking only at the execution window X2.
In the following, it will be described as assuming that the front code window X1 is hidden in the penetration mode.

FIG. 8 is a flowchart showing a processing process when a touch to the targeting selection area 162 is detected.
The process shown in FIG. 8 is executed when a touch to the targeting selection area 162 is detected in the monitor screen 102. First, when the touch to the monitor screen 102 is detected, the function input unit 114 is a touch to the targeting selection area 162 (selection area 128L1, selection area 128R1) (lower left corner or lower right corner of the monitor screen 102). Judge whether or not. If the targeting selection area 162 is touched, the layer management unit 124 sets the penetration mode (S10). At this time, the execution window X2 becomes the focus window. The display update unit 122 sets the code window X1 to be hidden (S12). That is, when the targeting selection area 162 is touched to shift to the penetration mode, the execution window X2 becomes the focus window, and the code window X1 on the front side of the execution window X2 is completely transparent (hidden). The code window X1 is invisible, but not closed. Also, the code window X1 is not layered on the back of the execution window X2.
The process when a touch to the gesture activation area 164 is detected will be described later in relation to FIG.

FIG. 9 is a flowchart showing a processing process when a touch other than the selection area 166 is detected.
When a touch other than the selection area 166 is detected, if it is in the normal mode (S20: normal mode), the data input unit 118 has a touch point in the area where the code window X1 and the execution window X2 overlap (hereinafter, "superimposition area"). It is determined whether or not it is (S22). If it is a touch to the superposed area (Y in S22), the data input unit 118 accepts data input for the code window X1 (S24). That is, when the overlapping area of the code window X1 and the execution window X2 is touched, and the code window X1 (front side) is the focus window, normal input is made.

When the touched point is not the superposed area (N in S22), the data input unit 118 accepts data input for the code window X1 or the execution window X2 (S26). At this time, the layer management unit 124 temporarily treats the window that accepts the touch operation as the focus window. For example, when the touch point has only the code window X1, it is treated as a normal input to the code window X1, and when there is only the execution window X2, it is treated as a normal input to the execution window X2. In either case, the context of the layers with the code window X1 on the front and the execution window X2 on the back is maintained.

When the touch is detected, if it is in the penetration mode (S20: penetration mode), the data input unit 118 determines whether or not the touch point is a superposition region (S28). If it is a touch to the superposed area (Y in S28), the data input unit 118 accepts data input for the execution window X2 (S30). That is, when the overlapping area of the code window X1 and the execution window X2 is touched, and the execution window X2 (back side) is the focus window, the penetration input is performed.

When the touched point is not the superposed region (N in S28), the data input unit 118 accepts data input for the touched window of the code window X1 or the execution window X2 (S32). Also at this time, the layer management unit 124 temporarily treats the window that accepts the touch operation as the focus window.

FIG. 10 is a flowchart showing a processing process when detachment is detected.
The process shown in FIG. 10 is executed when a detach (touch-off) from the monitor screen 102 is detected. First, in the case of detachment in the penetration mode (Y in S40), the display update unit 122 redisplays the code window X1 that was hidden in S12 (FIG. 8) (S42). At this time, the code window X1 becomes a normal semi-transparent display. The layer management unit 124 sets the normal mode (S44). The code window X1 is targeted.

The display update unit 122 updates the data of the non-focus window according to the data input. For example, when data is input to the code window X1, the display update unit 122 changes the image of the execution window X2 according to the input contents to the code window X1. When data is input to the execution window X2, the display update unit 122 changes the drawn image of the code window X1 according to the input contents to the execution window X2. The data update will be described in detail in connection with FIG. 12 and after.

In the case of detaching in the normal mode (N in S40), the process shifts to S46.

FIG. 11 is an external view for explaining switching of data input targets.
The user can freely change the position and size of the code window X1 and the like as in the case of a general multi-window. In FIG. 11, the code window X1 on the front side is moved to the lower right. On the other hand, the execution window X2 on the back side is displayed on the entire surface of the monitor screen 102.

The superimposed area 132 indicates an area in which the execution window X2 (back surface) is hidden by the code window X1 (front surface). The exposed area 134 indicates an area where the execution window X2 on the back surface is exposed by moving the code window X1 on the front surface to the lower right. When the user touches the point P1 of the exposed area 134, the exposed execution window X2 on the back becomes the focus window (the window to be input data) regardless of whether or not it is in the penetration mode. .. The layer management unit 124 temporarily treats the execution window X2 as a focus window when the exposed area 134 is touched.

When the user touches the point P2 of the superimposed area 132, as described above, either the code window X1 or the execution window X2 is the focus window (data entry), depending on whether it is in the penetration mode or the normal mode. The target window). When the execution window X2 is moved and the area where only the code window X1 exists is touched, the code window X1 becomes the target of data input.

Next, as a specific example of the user interface, the process of creating a code for drawing a circle while properly using the penetration input and the normal input will be described with reference to FIGS. 12 to 15.

FIG. 12 is a first screen view showing a state at the time of penetrating input to the execution window X2.
The user is touching the selection area 128L1 (targeting selection area 162) with the thumb of the right hand and touching the monitor screen 102 with the middle finger of the right hand. By touching the selection area 128L1, the penetration mode is set. When the selection area 128L1 is touched, the display update unit 122 hides (completely makes transparent) the code window X1. The data input unit 118 detects a touch on the coordinates (849,581) as a penetrating input to the code window X1. The data input unit 118 detects the touch on the coordinates P3 as a dot operation on the execution window X2.

FIG. 13 is a second screen view showing a state at the time of normal input to the code window X1.
When the user releases the right thumb from the selection area 128L1 (detach), the display update unit 122 redisplays the code window X1. Since the code window X1 is displayed semi-transparently, the user can visually recognize the dot P3 of the execution window X2 on the back through the code window X1. The display update unit 122 displays the code menu 136 in the vicinity of the coordinates P3 (849,581) in response to the immediately preceding dot operation. The code menu 136 is a part of the code window X1 and includes a plurality of buttons for creating code.

Various codes can be created with the buttons on the code menu 136. The user wants to execute the code that draws a circle here. First, the user touches the start point button 135 to temporarily register the coordinates P3. Similarly, in the execution window X2, the coordinate P4 of the second point is touched by the penetration input. After inputting the above, touch the circle drawing button 138. At this time, the coordinates P3 become the center coordinates of the circle, and the distance from the coordinates P3 to the coordinates P4 becomes the radius of the circle.

FIG. 14 is a third screen view showing a state at the time of normal input to the code window X1.
The drawing color of the program code input by the button of the circle drawing 138 may be selected in advance from the color selection palette (not shown) by the user, or may be a color specified in advance as an initial setting. Here, it is assumed that "radius: 100" and "color: red" are specified. The display update unit 122 displays the code "draw circle (849,581,100, color.red)" in the code window X1. This "draw circle" is a code for instructing the drawing of a circle, and is associated with the circle drawing button 138 in advance. The first argument shows the X coordinate of the center point, and the second argument shows the Y coordinate of the center point. In the present embodiment, the coordinates (849,581) of the hitting point P3 in FIG. 12 are substituted as they are as the coordinates of the center point. The third argument "100" indicates the radius. The fourth argument "color.red" indicates that the circle is drawn in "red". In this way, the user can create a code by selecting a button from the code menu 136. The user can also directly enter the code "draw circle" into the code window X1.

FIG. 15 is a fourth screen view showing a state when a circle is drawn in the execution window X2.
When the code "draw circle" is input, the display control unit 120 draws the circle 140 specified in the execution window X2. Circle 140 is a red circle with a radius of 100 centered at point P3 (849,581), as indicated by the code "drawcircle (849,581,100, color.red)". In FIG. 15, the code window X1 is hidden, but the code window X1 may remain semi-transparent. In this way, the user visually recognizes the code window X1 and the execution window X2, sets the striking point P3 (penetrating input to the execution window X2), and inputs a code based on the striking point P3 (normal input to the code window X1). After doing so, the code execution result can be confirmed in the execution window X2. According to such a control method, it is possible to provide a user interface that makes it easy to confirm the connection between the code and the execution result.

Suppose the user wants to create a program that draws a circle. At this time, the center coordinates of the circle may be directly input as an argument of the code "draw circle". However, for beginners in programming, the task of finding out what kind of arguments should be entered for each code is burdensome. Therefore, in the present embodiment, the "place to be the center of the circle (point P)" is directly specified by touching the execution window X2 showing the execution result of the program. After the points P3 and P4 are struck in the execution window X2, the user is made to select the circle drawing button 138 while providing the support tool such as the circle drawing button 138. As a result, the code "draw circle (849,581,100, color.red)" is displayed in the code window X1.

The user can enter the code (instruction) "drawcircle" to draw a circle by semi-automatically generating the code "drawcircle (849,581,100, color.red)", and therefore the center point as an argument. You can intuitively understand the common programming common sense that you only have to specify the coordinates, radius, and color of. After entering the code "draw circle", a circle is actually drawn in the execution window X2 according to this code, so the user can confirm that the circle is drawn exactly as the code imagined. By maintaining the visibility and input ease of both the code window X1 and the execution window X2, it becomes easy for the user to be aware of the interlocking of both windows. By such a control method, the user can be made to recognize the relationship between the "code" and the "execution result", in other words, "how the code is written and what kind of output is obtained".

FIG. 16 is a screen view when the explanation of the function input is displayed.
The user sets the penetration mode by touching the targeting selection area 162 (selection area 128L1, 128R1), and sets the gesture mode by touching the gesture activation area 164 (selection area 128L2, 128R2). The display update unit 122 is for explaining the penetration input function activated by the selection area 128L1 when a predetermined time, for example, 5 seconds elapses without inputting data while the user keeps touching the selection area 128L1. Display the explanation area 142. The same applies to the selection area 128R1. When the touch to the gesture activation area 164 (selection area 128L2, 128R2) is continued, the display update unit 122 displays the explanation area 142 related to the gesture input.

When the number or types of selection areas 128 are large, the user recognizes that invisible selection areas 128 exist at the four corners of the monitor screen 102, but forgets which selection area 128 provides what function. It may be difficult to understand or it may be difficult to understand. Since the explanation area 142 is displayed by continuously touching the invisible selection area 128 at the end of the monitor screen 102 for a while, the user can easily check the functions such as the selection area 128.

When the selection area 166 is touched, the display update unit 122 may immediately display the explanation area 142. However, in this case, since the explanation area 142 is displayed every time the user touches the selection area 166 of the monitor screen 102, the explanation area 142 is displayed on condition that the selection area 166 is continuously touched for a predetermined time. Is considered to be more preferable. Further, the explanation area 142 may be displayed on condition that the user makes another input while touching the selection area 166. For example, the explanation area 142 may be displayed when the user touches the selection area 128L1 and then speaks a predetermined word, for example, "explanation".

FIG. 17 is a screen view when performing gesture input.
The user can activate the gesture area G1 by touching the gesture activation area 164 (selection area 128L2, 128R2). In FIG. 17, the user is touching the selection area 128L2 with the thumb of the left hand. At this time, the layer management unit 124 activates the gesture area G1 and shifts from the normal mode to the gesture mode. Gesture area G1 remains invisible when enabled. The display update unit 122 may change the display mode such as changing the color of the entire screen so that the user can easily understand that the mode is the gesture mode. Alternatively, an icon indicating that the gesture mode is in progress may be displayed in a predetermined area of the monitor screen 102. The gesture input unit 116 detects all inputs in the gesture mode as gesture inputs.

The gesture input unit 116 determines whether or not the movement of the user's finger on the monitor screen 102 corresponds to any of the pre-registered gesture patterns. When it corresponds to the gesture pattern, the process corresponding to the gesture pattern is executed for the code window X1 (focus window). For example, as shown in FIG. 17, when the user slides two fingers up and down at the same time, the display update unit 122 scrolls the code window X1 up and down. The display update unit 122 may be able to scroll the code window X1 up and down at a higher speed than usual by inputting a gesture.

In addition to this, the following gesture patterns are registered.
-When the user taps the monitor screen 102 with one finger, the context menu can be displayed at the tapped portion of the code window X1. The context menu is generally a menu that is displayed when the mouse is right-clicked, and displays a list of various operation options. The user can select the menu he wants to execute from the context menu.
When the user double-tap the monitor screen 102 with two fingers, the display update unit 122 enlarges or reduces the code window X1.
-Even when the user pinches the monitor screen 102 with two fingers, the display update unit 122 enlarges or reduces the code window X1.
-When the user slides two fingers left and right, the display update unit 122 changes the page of the code window X1.
When the user double-tap the monitor screen 102 with three fingers, the display update unit 122 maximizes the code window X1.
The above is an example, and various gesture patterns may be registered in addition to the above. For example, various gesture patterns can be defined according to the number or combination of fingers used, or through circular movement, linear movement, polygonal movement (eg N-shaped movement, W-shaped movement), etc. be.

If the gesture mode is set, various gesture inputs can be performed using the entire monitor screen 102. The user can perform various operations on the code window X1 by touching the selection area 128L2 with the left hand supporting the tablet terminal 100 and drawing a large gesture on the entire monitor screen 102 with the right hand.

The tablet terminal 100 has been described above based on the embodiment.
According to the tablet terminal 100 of the present embodiment, when the code window X1 and the execution window X2 are superimposed and displayed, the back side of the code window X1 and the execution window X2 does not need to be fronted to switch the context. Data can be entered in the execution window X2.

Conventionally, when the code window X1 is displayed hierarchically on the front side and the execution window X2 is displayed on the back side, it is necessary to bring the execution window X2 to the front side in order to input data to the execution window X2. By the frontal operation, the execution window X2 is moved to the front side, and the code window X1 is moved to the back side. In order to bring it to the front, it was necessary to "push and release" devices such as touch panels and mice. If it is a mouse, it is necessary to click the "close button" or the "reduce button" of the code window X1. In this case, it is necessary to move the mouse pointer to a position such as a "close button", press the left mouse button, and release the hand. At the time of the front operation, the user is aware of the layer replacement work of "moving the execution window X2 on the back to the front by eliminating the obstructive code window X1 on the front". The same applies to the work using shortcut keys in that the layers of the window are replaced by the fronting operation.

On the other hand, in the tablet terminal 100 of the present embodiment, the user can target the execution window X2 simply by touching the targeting selection area 162. For example, the user can input data into the execution window X2 on the back surface with the right hand while touching the targeting selection area 162 with the left hand holding the tablet terminal 100. When you want to input data to the code window X1, you only have to take your finger off the targeting selection area 162. Since the data is input with the right hand while selecting the focus window with the left hand, it is easy for the user to continue the work without feeling interruption even when the focus window is changed. In particular, when working while frequently going back and forth between the code window X1 and the execution window X2, the user only has to repeatedly touch and detach the target selection area 162 with his / her free hand, so that the operability associated with the change of the focus window is improved. It will improve dramatically.

Further, when the focus window is changed, the context of the code window X1 and the execution window X2 is maintained, so that the user does not need to be aware of the change in the layer structure. In other words, you don't have to worry about "which window is moving to which layer" even if you switch the focus windows one after another when there are many windows. In the present embodiment, the visibility of the execution window X2 (focus window) is further enhanced by hiding the code window X1 when the targeting selection area 162 is touched.

In the normal mode, since the code window X1 is displayed semi-transparently, the code window X1 and the execution window X2 can be visually recognized at the same time. There is no need to juxtapose the code window X1 and the execution window X2, move or close the code window X1 to see the execution window X2.

When the user keeps touching the selected area 166 and the like, the explanation area 142 is displayed. Even if the function determination area M1 is not visible, the user understands that the function selection input can be performed by touching the vicinity of the end of the monitor screen 102. Since the explanation area 142 is displayed by continuously touching the end of the monitor screen 102, the user can easily understand what kind of function selection input can be performed by touching where.

As another accessibility, the display update unit 122 may display information indicating available selection areas or the role of each selection area. In this information display, each of the selected areas may be illuminated and displayed together with the function name thereof. For example, when a certain time elapses without inputting the next function after the user inputs a certain function, in other words, when a predetermined time elapses from the appearance of the explanation area 142, the display update unit 122 has such a function. Auxiliary functions may be enabled.

The user can activate the gesture area G1 by touching the gesture activation area 164 (selection area 128L2, 128R2). When the gesture area G1 is enabled, the user can input various gestures while moving his / her finger to fill the monitor screen 102. Since the monitor screen 102 can be used extensively for gesture input, high operability can be realized even when the monitor screen 102 is small. Further, since the gesture input and the data input are switched by the touch / detach of the gesture activation area 164, the data input is not erroneously performed during the gesture input. Similarly, the operation for data input is not mistakenly recognized as gesture input.

As described above, according to the present embodiment, for example, a right-handed user can perform data input work by right-handedness and appropriately change the mode by using his / her free left hand.

The present invention is not limited to the above-described embodiment or modification, and the components can be modified and embodied within a range that does not deviate from the gist. Various inventions may be formed by appropriately combining a plurality of components disclosed in the above embodiments and modifications. In addition, some components may be deleted from all the components shown in the above embodiments and modifications.

Although it has been described that the learning software is executed by one tablet terminal 100, a part of the functions of the learning software or the tablet terminal 100 may be realized by another device such as a server.

[Modification example]
FIG. 18 is a conceptual diagram for explaining the focus control of three or more windows.
In FIG. 18, six layers are formed by four windows W1 to W4 in addition to the function determination area M1 and the gesture area G1. A gesture activation area 164 is set in the upper left part of the function determination area M1, and four targeting selection areas 162 of W1 selection area 150, W2 selection area 152, W3 selection area 154 and selection W4 selection area 156 are set in the lower side part. Is set. When the gesture activation area 164 is touched, the gesture area G1 is activated.

The W1 selection area 150 is associated with the window W1, the W2 selection area 152 is associated with the window W2, the W3 selection area 154 is associated with the window W3, and the W4 selection area 156 is associated with the window W4. The user can set the window W1 as the focus window when the W1 selection area 150 is touched. Similarly, the user can set the window W2 as the focus window when the W2 selection area 152 is touched. The same applies to the W3 selection area 154 and the W4 selection area 156.

The layer management unit 124 does not change the layer even if the target window is changed. Therefore, the context of windows W1 to W4 is maintained. For example, it is assumed that window W1 is a web screen, window W2 is an editing screen by text creation software, window W3 is an editing screen of graphic editing software, and window W4 is an editing screen of table calculation software. When the user works while switching the four softwares at any time, the focus window may be switched at any time by the targeting selection area 162. Since the context of layers is not changed, you don't have to worry about which layer the desired window is on, even when there are multiple windows.

Even if the selected window is set as the focus window on the condition that a part or all of the selected window is shielded by any of the windows on the front side among the windows W1 to W4. good. In other words, when the front surface is not shielded, the focus change by the function selection input may be treated as invalid. For example, suppose window W3 is selected by the W3 selection area 154. When a part of the window W3 is shielded by the window W1 or the window W2 on the front side, the layer management unit 124 sets the window W3 as the focus window. On the other hand, when the front surface of the window W3 is not shielded because both the window W1 and the window W2 are closed, the setting of the window W3 may not be changed to the focus window even if the W3 selection area 154 is touched. ..

In the present embodiment, the code window X1 is displayed semi-transparently, and the code window X1 and the execution window X2 can be visually recognized at the same time. As a modification, the code window X1 may be displayed normally (opaquely displayed).

In the present embodiment, it has been described that the selection area 166 is arranged at the four corners of the function determination area M1 (monitor screen 102). As a modification, a plurality of selection areas 166 may be arranged on the left side portion of the function determination region M1 or may be arranged on the bottom side portion. In addition, the arrangement of the selection area 166 such as the upper side and the right side can be arbitrarily set.

In the present embodiment, the learning software has been described as being executed on the tablet terminal 100, but the learning software may be executed on another mobile computer such as a smartphone or a laptop PC. Similarly, the learning software may be executed on a stationary computer such as a desktop PC.

In the present embodiment, it has been described that the code window X1 is set as the focus window in the normal mode. As a modification, the execution window X2 on the back side may be set as the focus window in the normal mode, and the code window X1 on the front side may be set as the focus window in the special mode.

The change to the penetration mode may be a method other than touching the targeting selection area 162. For example, you may change to penetration mode when you double-click the mouse or press a certain key on the keyboard. In addition, the penetrating mode may be changed by voice input, or may be changed to the penetrating mode when the user focuses his / her line of sight on a predetermined area of the monitor screen 102. A change to penetration mode may be made using a foot device such as a pedal. Similarly, in addition to touch, data may be input by a mouse, keyboard, voice, or the like.

In the present embodiment, it has been described that the penetration mode is set only for the period during which the touch of the targeting selection area 162 continues. As a modification, the layer management unit 124 may continue the penetration mode for a predetermined valid period, for example, 3 seconds after the targeting selection area 162 is touched. If the penetration input is made during the validity period, the validity period may be further extended for 3 seconds. When no input continues for 3 seconds, the layer management unit 124 may return from the penetration mode to the normal mode.

In the present embodiment, it has been described that the code window X1 on the front surface is displayed semi-transparently. As a modification, the display update unit 122 may display the code window X1 as a normal display (opaque display). The display update unit 122 may change the code window X1 to a semi-transparent display or a transparent display on condition that the code window X1 and the execution window X2 overlap. Alternatively, when the code window X1 and the execution window X2 overlap, the display update unit 122 may set only the superposed area of the code window X1 to semi-transparent display or transparent display.

In the present embodiment, the layer management unit 124 has described that the contexts of the code window X1 and the execution window X2 are not interchanged. As a modification, the layer management unit 124 may conditionally replace the context of the code window X1 and the execution window X2. When a data input operation is performed on the exposed area of the execution window X2, in other words, the area of the execution window X2 that is not blocked by the code window X1, the layer management unit 124 has a context of the execution window X2 and the code window X1. May be replaced. For example, when the point P1 (exposed area 134) in FIG. 11 is touched, the layer management unit 124 may display the execution window X2 in front of the code window X1 and set the execution window X2 as the focus window.

In the penetration mode, when the user touches the point P1 of the exposed area 134, the layer management unit 124 may maintain the context of the code window X1 and the execution window X2 as they are. When the exposed area 134 of the execution window X2 (rear surface) is touched, the context may be maintained in the penetration mode, and the execution window X2 may be moved to the front in the normal mode.

In the present embodiment, it has been described that the translucent code window X1 is hidden (completely transparent) when the penetration mode is set. As a modification, both the code window X1 and the execution window X2 may be normally displayed. Then, when the penetration mode is set, the display update unit 122 may increase the visibility of the execution window X2 (focus window) by increasing the transparency of the code window X1.

Further, when the penetration mode is set, the transparency of the execution window X2 may be lowered. For example, assume that both the code window X1 and the execution window X2 are translucently displayed. When the penetration mode is selected, the visibility of the execution window X2 (focus window) may be improved by reducing the transparency of the execution window X2 which is the focus window.

The same applies to the case where three or more windows are provided as shown in FIG. As an example, windows W1 to W4 are all displayed semi-transparently. When the user targets the window W2, if the transparency of the window W2 is reduced, the user can easily recognize that the window W2 is the focus window. At this time, the transparency of the window W1 on the front surface may be increased.

The window on the back side may be a window having an input interface such as a button, for example, a dialog box or a software keyboard. When the user wants to operate the dialog box, he / she may touch the targeting selection area 162 (selection area 128L1, 128R1). At this time, the window as the operation console on the back side may be operated by penetrating input, and then the main window on the front side may be returned and operated by normal input.

In the present embodiment, it has been described that the targeting selection area 162 is touched to shift to the penetration mode, and the operation for data input is performed in that state. As a modification, it is possible to dynamically select which of the code window X1 and the execution window X2 is to be the focus window according to the setting when performing the operation for data input.

As an example, the touch panel may have a function of detecting the touch pressure. In this case, the layer management unit 124 may set the code window X1 as the focus window when it is lightly touched, and may set the execution window X2 as the focus window when it is strongly touched. Further, the operation detection unit 110 may detect which finger is touched by the area of the touched finger. For example, the code window X1 may be set as the focus window when touched with the index finger, and the execution window X2 may be set as the focus window when touched with the little finger.

The operation detection unit 110 may distinguish between the touch of the finger and the touch of the stylus pen depending on the difference in capacitance. In this case, the code window X1 may be set as the focus window when touched with a finger, and the execution window X2 may be set as the focus window when touched with the stylus pen.

A plurality of types of stylus pens may be prepared, and the operation detection unit 110 may identify each of them. Associate a window with each of multiple stylus pens. In this case, data can be input to a plurality of windows by using a plurality of stylus pens properly. For example, software that draws a picture like an oil painting by overlaying an undercoat window, a drawing window, a rough drawing window, and a finishing window can be considered. In this case, the user may operate the four translucent windows at the same time while exchanging the stylus pen (type of brush). Then, the picture may be completed by superimposing four translucent windows.

Gesture input may be possible during the penetration mode. For example, when the user is touching only the targeting selection area 162, the code window X1 is the target of the gesture input. On the other hand, when the user is touching the targeting selection area 162 and the gesture activation area 164 at the same time, the execution window X2 may be operated by the gesture input by the user.

In addition to the learning software, the penetration input shown in this embodiment can be used. For example, in a computer game, two types of windows are prepared. One is the game field window, and a menu window for selecting equipment is prepared on the back of the window. In this case, the user may operate the menu window by penetrating input as necessary while targeting the window on the front side. Further, a plurality of windows may be prepared corresponding to a plurality of game fields. The game may be played simultaneously in the front game field and the back game field, and the user may play the game while switching the focus of the two windows.

In SNS (Social Networking Service) software or the like, a pop-up screen is displayed while inputting a chat screen, and data input on the chat screen may be recognized as an erroneous input on the pop-up screen. As a countermeasure for this, the chat screen may be used as the focus window in the normal mode. Even if the pop-up screen is displayed in front of the chat screen, the chat screen on the back side remains the focus window, so it is possible to prevent input to the pop-up screen during chat. When the user wants to change the input to the pop-up screen, he / she may touch the selection area 166 to set the pop-up screen as the focus window. In this case as well, the context of the pop-up screen and the chat screen is maintained. According to such a control method, it is possible to prevent the user from being interpreted as input to the pop-up screen that suddenly appears even though the user intends to input to the chat screen.

100 tablet terminal, 102 monitor screen, 104 user interface processing unit, 106 data processing unit, 108 data storage unit, 110 operation detection unit, 112 output unit, 114 function input unit, 116 gesture input unit, 118 data input unit, 120 display Control unit, 122 display update unit, 124 layer management unit, 126 communication unit, 128L1 selection area, 128L2 selection area, 128R1 selection area, 128R2 selection area, 132 superimposition area, 134 exposed area, 135 start point button, 136 code menu, 138 Circle drawing button, 140 yen, 142 description area, 150 W1 selection area, 152 W2 selection area, 154 W3 selection area, 156 W4 selection area, 162 target selection area, 164 gesture activation area, 166 selection area, 300 processors, 302 1st bus, 304 memory, 308 bridge, 310 2nd bus, 312 storage, 314 input device, 316 output device, 318 peripheral device, G1 gesture area, M1 function judgment area, X1 code window, X2 execution window

Claims (14)

A display control unit that arranges the first area for displaying the first image on the front side and arranges the second area for displaying the second image on the back side of the first area.
A first operation for designating the second area and an operation detection unit for detecting a second operation for data input from the user are provided.
When the second operation is detected after the detection of the first operation, the display control unit keeps the first region in front of the second region and based on the second operation, the second operation. An information processing device characterized by changing an image. After the detection of the first operation, when the second operation is detected in the overlapping region of the first region and the second region, the display control unit inputs the second operation to the second region. The information processing apparatus according to claim 1, wherein the information processing apparatus is determined to be. When the second operation is detected after the detection of the first operation, the display control unit changes the second image based on the second operation, and further, the display control unit changes the second image according to the change of the second image. The information processing apparatus according to claim 1 or 2, wherein changes in both the first image and the second image are visually displayed by changing the first image. When the second operation is detected in the exposed region where the first region is not superimposed on the second region, the display control unit displays the second region in front of the first region. The information processing apparatus according to any one of claims 1 to 3, wherein the information processing apparatus is changed. 4. The display control unit maintains the context of the first region and the second region when the second operation is detected in the exposed region after the detection of the first operation. The information processing device described in. The display control unit is characterized in that a plurality of areas are set for each of the plurality of display layers, and any area can be selected as a data input target according to a selection instruction as the first operation from the user. The information processing apparatus according to any one of claims 1 to 5. When a touch from the user is detected in any of a plurality of selection areas formed on the screen, the operation detection unit detects the touch as the first operation.
The information processing device according to claim 6, wherein the display control unit sets an area associated with the selection area to be the target of the first operation as an area to be the target of data input. The display control unit sets the selected area as a target area for data input on the condition that all or a part of the selected area is shielded by the front area. The information processing apparatus according to claim 6 or 7. The information processing device according to claim 1, wherein the display control unit further increases the transparency of the first region when the first operation is detected. The information processing device according to claim 1 or 9, wherein the display control unit further reduces the transparency of the second region when the first operation is detected. When the second operation is detected during the continuation of the first operation, the display control unit changes the second image according to the second operation, and when the first operation is completed, the second operation The information processing apparatus according to claim 1, wherein the first image is changed according to the above. A display control unit that arranges the first area and arranges the second area on the back surface of the first area.
It is equipped with an operation detection unit that detects a second operation for data input from the user.
When the second operation is detected in the overlapping region of the first region and the second region, the display control unit at the time of detection while maintaining the display state of each of the first region and the second region. An information processing apparatus characterized in that either the first region or the second region is selected as an image region to be input by the second operation according to a setting. The function of arranging the first area for displaying the first image and
A function of arranging a second area for displaying a second image on the back surface of the first area, and
A function to detect the first operation that specifies the second area, and
A function to detect the second operation for data entry from the user,
When the second operation is detected after the detection of the first operation, the function of changing the second image based on the second operation while maintaining the first region in front of the second region. , A computer program characterized by exerting on a computer. The step of arranging the first area for displaying the first image and
A step of arranging a second area for displaying the second image on the back surface of the first area, and
A step of detecting the first operation of designating the second region, and
The step of detecting the second operation for data entry from the user,
When the second operation is detected after the detection of the first operation, the step of changing the second image based on the second operation while maintaining the first region in front of the second region. Information processing method including.

Images