JP2007250015A - Graphical user interface - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To control deformation of a window and image irrespective of a position of a cursor by combining information of a plurality of pointing devices. <P>SOLUTION: The graphical user interface is executed by a system provided with the plurality of the pointing devices 63 for inputting a plurality of pointing information relating to a display screen, a personal computer 21 for processing the pointing information from the plurality of the pointing device 63 and outputting it, a display 11 for displaying information output from the personal computer 21 on the screen. The personal computer 21 is provided with a holding part 31 for holding first pointing information input after a certain unit time from the plurality of the pointing devices 63, a CPU 30 for deforming the display screen in accordance with the first pointing information and second pointing information when the second pointing information is input after another unit time. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a graphical user interface for deforming an image displayed on a display, for example, resizing a window or deforming an image being edited.

  In general, an operating system (hereinafter referred to as an OS) that is mainstream in current computers is operated by a graphical user interface (GUI) in order to improve user operability. In this graphical user interface, applications and data are displayed as appropriate images on the screen, and the user operates a pointing device such as a mouse or a device such as a keyboard to give instructions and data to the application displayed on the screen. Edit.

  A graphical user interface is an interface designed to be interactively and intuitively operated using a graphical screen. The current graphical user interface is designed so that applications and data on a personal computer can be operated with the same feeling as a human work on a desk in both Windows (registered trademark) and Mac OS. Specifically, applications and files are displayed with easy-to-associate visual icons, and double-clicking on the icon starts the application or opens the file. The application can be moved for each rectangular area called a window, or multiple windows can be displayed at the same time for easy switching.

  That is, in a general graphical user interface, each application displays a work area called one or a plurality of windows on the screen, and each data is displayed in the window. The user can send an instruction to the application through the window displayed on the screen to view or edit the data, and the application displays the result on the screen.

  The user's operation is performed using a device such as a mouse called a pointing device. The pointing device normally includes one or more pointers for operating a cursor on the screen and switches called buttons. The user operates the pointer to move the cursor, and presses a button to transmit appropriate information as an electrical signal.

  An electrical signal sent from the pointing device is understood by the OS as an event. The type of event differs depending on the electrical signal sent from the pointing device. For example, if the mouse button is pressed and released once in a relatively short time, the event is “click”. If the button is pressed and released twice in a relatively short time, the event is “double click”. If the cursor is moved with the pointer while holding down the button for a relatively long time, the event may be “drag”.

  The event understood by the OS is appropriately processed by the OS according to the additional data such as its type and coordinates, or is sent to the application when the event occurs in the application area. When an event is sent to an application, an appropriate action according to the event is performed by the received application or discarded. Thus, a system in which an operation is performed by an event inside the OS is called an event-driven system.

  Details of event generation, transmission / reception method, and the like differ depending on each system. However, there are already available systems having a graphical user interface called event-driven type as described above. For example, the Mac OS of Apple, the Microsoft Windows (registered trademark) of Microsoft, the OS / 2 of IBM, or the X Windows System operating on UNIX (registered trademark) is known.

  In the above-mentioned system, generally when resizing a window, place the cursor near the "border" that is the outermost frame of the window, perform a drag operation that moves the cursor while pressing the button, and then the desired size. When it becomes, perform the operation of releasing the button. When moving to the window position, move the cursor to the part called the “title bar” that is positioned horizontally at the top of the window. Release the button when the position is reached. Here, the user performs the same operations for resizing and moving. Therefore, the action to the window, in this case resizing and moving, is based on the cursor position when the button is first pressed.

  By the way, there are times when it is difficult for the user to use the conventional method of deciding how to instruct by matching the position of the cursor. For example, when resizing, it is difficult to align the cursor as described above. If you press the button while it is out of alignment, you have given another instruction. Therefore, as a means for solving such a problem, a “display monitoring device” of Patent Document 1 is disclosed.

JP-A-9-128194

  However, in the device disclosed in Patent Document 1, although the area for recognizing an event when the window is reduced can be kept small, the device is originally narrow like the outer frame of the window at the time of resizing. There was a problem that it was not possible to make it easy to align the cursor.

  The present invention has been made in view of the above, and an object of the present invention is to enable control of windows and image deformation without depending on the position of a cursor by combining information of a plurality of pointing devices.

  To achieve the above object, a graphical user interface according to claim 1 processes a plurality of pointing devices for inputting a plurality of pointing information relating to a display screen, and processing the pointing information from the plurality of pointing devices. In a graphical interface executed in a system comprising: a computer for outputting; and a display for displaying information output from the computer on a screen, the computer receives a first input after a unit time from the plurality of pointing devices. An information holding means for holding one pointing information, and when the second pointing information is input after another unit time, the display screen according to the first pointing information and the second pointing information A display screen deforming means deforming, those having a.

  According to the present invention, it is possible to move a window regardless of the position of the cursor in the window, for example, by combining the pointing information sent from a plurality of pointing devices in the system and determining the contents of the user's instruction. .

  In the graphical user interface according to claim 2, the plurality of pointing information each include a current coordinate position, and the display screen deformation means is formed of a plurality of pointing information within a certain unit time. The first minimum rectangular area is calculated, and when the second pointing information is input after another unit time and the coordinate position is moved, the second minimum rectangular area formed by the second pointing information is calculated. Then, the display screen is deformed according to the first minimum rectangular area and the second minimum rectangular area.

  According to the present invention, by combining the pointing information sent from a plurality of pointing devices in the system and judging the user's instruction content, for example, it is formed by the movement of the coordinate position regardless of the position of the cursor in the window. The rectangular area allows the window to be resized.

  Further, in the graphical user interface according to claim 3, the display screen deformation means obtains a conversion ratio in the x and y directions between the first minimum rectangular area and the second minimum rectangular area, and The resizing operation is performed according to the conversion ratio.

  According to this invention, in claim 2, the conversion ratio in the x and y directions between the first minimum rectangular area formed at the first coordinate position and the second minimum rectangular area formed at the coordinate position after movement. Thus, it is possible to perform a resizing (window size changing) operation according to the conversion ratio.

  Further, in the graphical user interface according to claim 4, the plurality of pointing information each include a current coordinate position, and the display screen deformation means is formed by a plurality of pointing information within a certain unit time. A first straight line is calculated, and when the second pointing information is input after another unit time and the coordinate position is moved, the second straight line formed by the second pointing information is calculated, The display screen is deformed according to the first straight line and the second straight line.

  According to the present invention, by combining the pointing information sent from a plurality of pointing devices in the system and judging the user's instruction content, for example, it is formed by the movement of the coordinate position regardless of the position of the cursor in the window. The image can be rotated according to the inclination angle of the straight line.

  In the graphical user interface according to the fifth aspect, the display screen to be deformed is processed on the application.

  According to the present invention, it is possible to move application data, for example, an image being edited, by performing not only the movement of the window but also the processing by the display screen deformation means by the application.

  As described above, according to the graphical user interface according to the present invention (claim 1), the cursor is used for combining the pointing information sent from a plurality of pointing devices in the system and determining and processing the user's instruction content. The window and image deformation can be controlled without depending on the position of the image.

  Further, according to the graphical user interface (claim 2) of the present invention, by combining the pointing information sent from a plurality of pointing devices in the system and judging the content of the user's instruction, for example, the position of the cursor in the window Regardless, it is possible to perform an operation of resizing the window to a rectangular area formed by the movement of the next coordinate position from the initial coordinate position.

  According to the graphical user interface (claim 3) of the present invention, in claim 2, the first minimum rectangular area formed at the first coordinate position and the second coordinate position formed after the movement are provided. Since the conversion ratio in the x and y directions with respect to the minimum rectangular area is obtained and resized (the size of the window is changed), the resize operation according to the conversion ratio can be reliably performed.

  Further, according to the graphical user interface (claim 4) of the present invention, by combining the pointing information sent from a plurality of pointing devices in the system and judging the content of the user's instruction, for example, the position of the cursor in the window Regardless of the determination according to (dragging of the corner portion of the image), the image is rotated according to the inclination angle of the straight line formed by the movement of the coordinate position, so that a reliable image rotation operation can be performed.

  According to the graphical user interface of the present invention (Claim 5), not the movement of the window but the processing by the display image deformation means is similarly performed on the application to move the application data, for example, the image being edited. Or zoom.

  DESCRIPTION OF EMBODIMENTS Hereinafter, a preferred embodiment of a graphical user interface according to the present invention will be described in detail with reference to the accompanying drawings. The present invention is not limited to this embodiment.

  FIG. 1 is a block diagram of a system configuration example of a graphical user interface according to the embodiment. In the figure, reference numeral 11 is a display for displaying image information sent from a personal computer, reference numeral 21 is a personal computer connected to the display 11, and reference numerals 60a and 60b are connected to the personal computer 21 by wire, wireless or via a network. Connected pointing device.

  The display 11 is composed of a CRT display, a liquid crystal display, a plasma display, or the like, and displays a resolution of, for example, 640 (horizontal) × 480 (vertical) dots, that is, a screen of 640 × 480 = 307200 fine dots. Have the ability to

  The personal computer 21 includes a CPU 30, a holding unit 31 using a hard disk or the like used as a secondary storage device, and an output unit 40 having a VRAM (video random access memory) 41. And an input unit 50 connected to the pointing device 60a. The VRAM 41 is a video memory that stores the color of each dot of an image to be displayed in the form of binary RGB data.

  The pointing devices 60 a and 60 b are input devices such as a joystick or a mouse, and include an output unit 61, an input unit 62, and three switches 63.

Any connection method or standard (protocol) may be used between the pointing devices 60a and 60b and the personal computer 21. For example, RS232C and MIDI (musical instrument digital)
interface), USB (Universal Serial Bus), IEEE 1394, or other serial communication or parallel communication, or wireless communication using optical, FM, AM modulation / demodulation, etc., and a general-purpose network such as Ethernet (registered trademark) You may implement | achieve by the connection through.

  That is, the personal computer 21 includes a CPU 30 that performs calculation and control related to display output described later, a VRAM 41 for drawing an image to be displayed, and an output unit 40 that outputs the contents of the VRAM 41 to the display 11 as graphics. The output unit 40 includes a device for physical connection, a VRAM 41, and software for driving the device. Furthermore, the personal computer 21 has an input unit 50 that is input from the pointing devices 60a and 60b. The input unit 50 includes a device for physical connection and software for processing a signal from the device.

  Conventionally, one pointing device is usually connected, but this embodiment has one feature in connecting a plurality of pointing devices. Conventionally, a plurality of pointing devices may be connected. For example, a mouse is connected to an external terminal of a portable computer system such as a notebook PC. However, in this case, for example, even when a plurality of pointing devices are connected, the information input from the pointing devices is processed independently from the present invention. In the present invention, information input from a plurality of pointing devices is combined and processed.

  Here, the meaning of “plurality of” in a plurality of pointing devices is not limited to a physically different pointing device, for example, when two mice are connected, but a plurality of pointers physically from one pointing device. This includes cases where information can be output. For example, in a touch panel, a pen tablet, and the like, a device that can recognize and output a plurality of pointer information when a finger or a pen is simultaneously brought into contact with a plurality of locations is also included. An apparatus capable of outputting a plurality of pointer information even if it is physically one pointing device is synonymous with a plurality of pointing devices in the present invention.

  FIG. 2 is an explanatory diagram showing a typical display screen of the graphical user interface according to the present invention. In the figure, reference numeral 1 is a title bar, reference numeral 2 is a window (application area), reference numeral 3 is a border, reference numeral 4 is an iconized button, reference numeral 5 is an icon, and reference numeral 6 is a resize box.

  That is, the rectangle located in the center of FIG. In the currently mainstream OS, instructions to the OS and applications are given through the window 2. The screen background is also a special window, usually called the root window. The window 2 includes areas to which the contents of instructions to the OS, such as a normal title bar 1, a border 3, an iconize button 4, and a resize box 5, are assigned. This position is almost common between windows in the same OS, but may differ depending on the type of OS.

  In many OSs, as shown in FIG. 3, the window 2 is moved on the display by moving the cursor 6 to the title bar 1 and moving the cursor while pressing the button (switch 63). can do. Thus, in the conventional method, the pointing device instruction is determined by recognizing the position of the cursor 6. Therefore, in order to move the window 2, the cursor 6 must be positioned on the title bar 1 as described above. For this reason, if the same thing is done in window 2, the OS will give an instruction to the application to send the event to the application that is the owner of the window.

  On the other hand, in the present invention, the contents designated by the user are determined by combining information sent from a plurality of pointing devices 60a and 60b in the system. Here, the pointing devices 60a and 60b in FIG. 1 are referred to as Dev1 and Dev2. Dev1 and Dev2 are each provided with a switch (reference numeral 63 in FIG. 1) called a button. As described above, Dev1 and Dev2 may be physically different devices, or may be one device having a plurality of pointers physically, such as a certain kind of pen tablet.

  Next, an operation example in the system configured and defined as described above will be described. FIG. 4 is a flowchart showing a first operation example according to the embodiment of the present invention. FIG. 5 is a display screen showing a first operation example according to the embodiment of the present invention. Consider the case where the button is pressed while the Dev1 cursor is at the position p1 in FIG. Since this position is within the window 2, it is normally sent from the OS to the application as usual, but in this embodiment, the operation of FIG. 4 is performed.

  First, it is determined whether or not the button has been pressed (step S11). Here, if it is determined that the Dev1 button 1 is pressed, the information sent from Dev1 for a certain unit time t1 (for example, several 100 ms) after being pressed is held in the holding unit 31 (step S12). Then, it is determined whether there is next information (step S13). Here, if there is the following information, it is further determined whether or not the Dev2 button 2 is pressed (step S14). Here, it is assumed that the button is pressed in a state where the cursor of Dev2 is at the position p2 shown in FIG. Then, the held Dev1 information is not sent to the application, and thereafter, it is combined with the Dev2 information (step S15) and interpreted as a predetermined instruction to the OS.

  It is determined whether or not another unit time t2 has elapsed (step S16), and another unit time t2 has elapsed. For example, after 1 second, the cursor positions Dev1 and Dev2 have moved to p1 ′ and p2 ′ in FIG. And The OS calculates the shift amounts of p1 → p1 ′ and p2 → p2 ′ (step S17), determines the window movement amount by a predetermined method (step S18), and performs the window movement operation (step S19). ). Here, several predetermined methods can be considered. For example, the amount of change of p1 → p1 ′, the amount of change of p2 → p2 ′, or an average value thereof may be calculated. As a result of this operation, the window moves as shown in FIG.

  In this example, two devices Dev1 and Dev2 are used. However, the system can connect three or more pointing devices. The same operation is performed even if the input order of Dev1 and Dev2 is switched.

  As described above, according to the first operation example, by combining information sent from the plurality of pointing devices Dev1, Dev2 (60a, 60b), the window can be moved regardless of the position of the cursor in the window. . Further, it is possible to move application data, for example, an image being edited, by performing the same steps with the application instead of moving the window.

  By the way, when resizing a conventional window, an operation of clicking an outer frame (border 3) of the window or an area called a resize box 5 if present and dragging to a desired size is performed. However, since the outer frame of the window is thin, it is difficult to align the cursor even though there is some margin. Therefore, in this embodiment, this is solved by the second operation example.

  FIG. 6 is a flowchart showing a second operation example according to the embodiment of the present invention. FIG. 7 is a display screen showing a second operation example according to the embodiment of the present invention. As in the first operation example described above, consider the case where the button is pressed while the Dev1 cursor is at the position p1 in FIG. Since this position is within the window 2, it is normally sent from the OS to the application as usual, but in this embodiment, the operation of FIG. 6 is performed.

  First, it is determined whether or not the button has been pressed (step S21). Here, if it is determined that the Dev1 button 1 is pressed, the information sent from Dev1 for a certain unit time t1 (for example, several 100 ms) after being pressed is held in the holding unit 31 (step S22). Then, it is determined whether there is next information (step S23). Here, if there is the following information, it is further determined whether or not the Dev2 button 2 has been pressed (step S24). Here, it is assumed that the button is pressed in a state where the cursor of Dev2 is at the position p2 shown in FIG. Then, the held Dev1 information is not sent to the application, and thereafter, it is combined with Dev2 information (step S25) and interpreted as a predetermined instruction to the OS.

  First, a rectangle that is the minimum range including the coordinate values of Dev1 and Dev cursor positions p1 and p2 is calculated (step S26). In this example, since there are only two pointing devices in the system, this is a rectangle having p1 and p2 as diagonal points. When there are three or more pointing devices in the system, the minimum rectangle is obtained from the minimum value and the maximum value of each coordinate.

  It is determined whether another unit time t2 has elapsed (step S27). If the unit time t2 has elapsed, it is determined whether the cursor has further moved (step S28). Here, it is assumed that another unit time t2 has elapsed, for example, after 1 second, the positions of the cursors Dev1 and Dev2 have moved to p1 'and p2' in FIG. If it is determined that the cursor has moved, a rectangle that is the minimum range including p1 'and p2' is obtained (step S29). Further, the conversion ratios in the x and y directions between the rectangle composed of p1 and p2 and the rectangle composed of p1 'and p2' are obtained (step S30). The resize operation to the window is performed with the conversion ratio (step S31). As a result, the window moves as shown in FIG.

  In this example, two devices Dev1 and Dev2 are used. However, the system can connect three or more pointing devices. The same operation is performed even if the input order of Dev1 and Dev2 is switched.

  As described above, according to the second operation example, by combining information sent from the plurality of pointing devices Dev1, Dev2 (60a, 60b), the window can be resized regardless of the position of the cursor in the window. . Further, by performing the same steps with the application, it is also possible to move the application data, for example, the image being edited.

  Next, a method for rotating an image will be described. By the way, there is no current mainstream OS that can rotate the window. However, rotation of a display image is generally performed in a certain type of application, for example, a tool for editing an image. In this case, the corner (corner part) of the rotating image is usually dragged. Therefore, as in the case of window movement and resizing, since the instruction is determined according to the cursor position, there is the same problem as described above. For this reason, in this embodiment, this is solved by a third operation example described below.

  FIG. 8 is a flowchart showing a third operation example according to the embodiment of the present invention. FIG. 9 is a display screen showing a third operation example according to the embodiment of the present invention. As in the first operation example described above, consider the case where the button is pressed while the Dev1 cursor is at the position p1 in FIG. Since this position is within the window 2, it is normally sent from the OS to the application as usual, but in this embodiment, the operation of FIG. 8 is performed.

  First, it is determined whether or not the button has been pressed (step S41). Here, if it is determined that the Dev1 button 1 has been pressed, the information sent from Dev1 for a certain unit time t1 (for example, several 100 ms) after being pressed is held in the holding unit 31 (step S42). Then, it is determined whether there is next information (step S43). If there is the following information, it is further determined whether or not the Dev2 button 2 has been pressed (step S44). Here, it is assumed that the button is pressed in a state where the cursor of Dev2 is at the position p2 shown in FIG. Then, the held Dev1 information is not sent to the application, and thereafter, it is combined with Dev2 information (step S45) and interpreted as a predetermined instruction to the OS.

  It is determined whether another unit time t2 has elapsed (step S46). If the unit time t2 has elapsed, it is determined whether the cursor has further moved (step S47). Here, it is assumed that another unit time t2 has elapsed, for example, after 1 second, the position of the Dev2 cursor has moved to p2 'in FIG. If it is determined that the cursor has moved, the angle formed by the straight line p1p2 and the straight line p1p2 'is calculated (step S48). Thereafter, the image is rotated with the calculated angle as a rotation angle (step S49). That is, finally, the coordinates of the angle and the center of p1 are sent to the application, and an appropriate operation is performed by the application. In this example, the rectangle that is the image being edited is rotated by an angle formed by the straight line p1p2 and the straight line p1p2 '.

  As described above, according to the third operation example, by combining information sent from a plurality of pointing devices, an image can be rotated regardless of the position of the cursor in the window.

It is a block diagram which shows the system configuration example of the graphical user interface concerning embodiment. It is explanatory drawing which shows the typical display screen of the graphical user interface concerning this invention. It is explanatory drawing which shows the example of a movement display screen of a window. It is a flowchart which shows the 1st operation example concerning embodiment of this invention. It is a display screen which shows the 1st operation example concerning embodiment of this invention. It is a flowchart which shows the 2nd operation example concerning embodiment of this invention. It is a display screen which shows the 2nd operation example concerning embodiment of this invention. It is a flowchart which shows the 3rd operation example concerning embodiment of this invention. It is a display screen which shows the 3rd operation example concerning embodiment of this invention.

Explanation of symbols

1 Title Bar 2 Window 3 Border 5 Cursor 6 Resize Box 11 Display 21 Personal Computer 30 CPU
31 Holding unit 40 Output unit 41 VRAM
50 Input unit 60a, 60b Pointing device 63 Switch

Claims (5)

A plurality of pointing devices for inputting a plurality of pointing information about the display screen;
A computer that processes and outputs pointing information from the plurality of pointing devices;
A display for displaying information output from the computer on a screen;
In a graphical interface that runs on a system with
The computer
Information holding means for holding first pointing information input after a unit time from the plurality of pointing devices;
Display screen deformation means for deforming the display screen according to the first pointing information and the second pointing information when the second pointing information is input after another unit time;
A graphical user interface comprising: Each of the plurality of pointing information includes a current coordinate position;
The display screen deforming means calculates a first minimum rectangular area formed by a plurality of pointing information within a unit time, and the second pointing information is input after another unit time, and the coordinate position is moved. A second minimum rectangular area formed by the second pointing information is calculated, and the display screen is deformed according to the first minimum rectangular area and the second minimum rectangular area. The graphical user interface of claim 1.   The display screen deforming unit obtains a conversion ratio in the x and y directions between the first minimum rectangular area and the second minimum rectangular area, and performs a resizing operation according to the conversion ratio. The graphical user interface according to 2. Each of the plurality of pointing information includes a current coordinate position;
The display screen deformation means calculates a first straight line formed by a plurality of pointing information within a unit time, and when the second pointing information is input after another unit time and the coordinate position is moved, 2. The graphical user according to claim 1, wherein a second straight line formed by the second pointing information is calculated, and the display screen is deformed according to the first straight line and the second straight line. interface.   The graphical user interface according to claim 1, wherein the display screen to be deformed is processed on an application.

Images