JP2005301669A - Information processor and information processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the operability of a graphical user interface. <P>SOLUTION: In the information processor 1, when pressing of a plurality of buttons displayed on a graphical user interface is determined, or when overlap of a pointer with a plurality of contact determination areas prevents clear determination of which of buttons is pressed, a probably pressed button is rearranged in a clearly separated position on a display screen. Accordingly, when buttons of relatively small areas are closely displayed in the graphical user interface, a user can easily press down an intended button, and the operability of the graphical user interface can be improved. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an information processing apparatus having a graphical user interface and an information processing program for realizing the information processing apparatus.

Currently, a graphical user interface (GUI) is used as a user interface in an information processing apparatus such as a PC (Personal Computer), a personal digital assistant, or a mobile phone.
The GUI is a user interface that inputs and outputs information between the apparatus and the user using a lot of graphics. The user can perform input operations on the screen with a mouse or the like, and is highly convenient.

By the way, it is practically difficult to use an input device such as a keyboard, a mouse, or a trackball in a portable device such as a mobile phone among information processing apparatuses. In addition, portable devices are small in size, and it is difficult to provide a large number of hardware buttons and the like on the main body.
For this reason, in a portable device, it is conceivable to use a touch panel that can be input by an operation on a display screen.

However, in recent years, portable devices have been reduced in size, and when such a device is equipped with a GUI, a large number of buttons must be displayed in a limited-size display screen, resulting in a decrease in operability. Resulting in. In addition, the display screen is directly touched or pressed, which may cause the display screen to become dirty or cause equipment failure.
Therefore, as a GUI applicable to a portable device, a predetermined object or a user's body, for example, a pen tip or a hand (hereinafter referred to as “object” as appropriate) is photographed by an imaging device, and an input operation is performed according to the movement. A technique for performing the above has been proposed.

For example, Japanese Patent Application Laid-Open No. 10-207619 discloses a technology that functions as a GUI pointer by photographing a user's hand, a pen tip, and the like, and recognizing those operations through image processing (Patent Document). 1).
According to the technique described in Patent Document 1, since an input operation can be performed by virtually pressing a button displayed on the display screen, there is no need to include an input device such as a mouse, Since the touch panel is not actually touched, a GUI that does not cause the above-described problem can be realized.
Japanese Patent Laid-Open No. 10-207619

  However, in the conventional technique including the technique described in Patent Document 1, the size of the display screen is limited even when the object is photographed and used as a GUI pointer as described above. Therefore, the area of buttons and the like displayed in the display screen is relatively small. When a plurality of buttons are displayed on such a display screen and the user tries to press any button, the image of the object touches the images of the adjacent buttons, and the user does not intend to use the button. A situation where the button is pushed down may occur.

Therefore, the user must take measures to avoid touching a plurality of buttons by moving the object away from the imaging device and reducing the image of the object, and the operability of the graphical user interface is reduced. There was a problem to do.
Such a problem may generally occur when buttons having a relatively small area are displayed in a dense manner, but becomes a particularly serious problem in a portable device having a small display screen.

  An object of the present invention is to improve the operability of a graphical user interface.

In order to solve the above problems, the present invention provides:
An information processing apparatus including an image pickup unit (for example, the image pickup apparatus 10 in FIGS. 1 and 2) that captures an image of a subject, and an interface display unit (for example, the control in FIG. 2) that displays a graphical user interface on a display screen. Unit 30, display control unit 50 and display 60) and an image of a preset object (for example, the indicator 2 in FIG. 1 or a user's finger) is captured by the imaging unit, Pointer recognition means for recognizing the image of the object as a pointer in the graphical user interface (for example, the control unit 30 in FIG. 2) and a pointer recognized by the pointer recognition means in the graphical user interface A plurality of input areas displayed for operation (for example, the connection shown in FIG. 6). It is determined by the overlap determination means (for example, the control unit 30 in FIG. 2) which determines whether or not the pointer overlaps the plurality of input areas. In this case, a plurality of areas overlapping with the pointer are rearranged at positions farther apart on the display screen (for example, the four corners of the display screen or the upper side, the lower side, and the center of the left and right sides). And rearrangement means (for example, the control unit 30 in FIG. 2).
As a result, when it is determined that the input operation is performed on the plurality of input areas displayed on the graphical user interface, that is, the pointer overlaps with the plurality of input areas, and which of the input operations is performed. When it is not possible to clearly discriminate, the input area that may have been subjected to the input operation is rearranged at a position that is clearly isolated on the display screen.

Therefore, when an input area (button or the like) having a relatively small area is densely displayed in a graphical user interface, it is easy to perform an input operation intended by the user, and the operability of the graphical user interface is improved. Is possible.
In addition, the rearrangement unit arranges the plurality of input areas determined to overlap with the pointer at at least one of the four corners of the display screen.

Thus, the input area can be rearranged at a position that is clearly isolated on the display screen.
Further, the rearrangement unit arranges the plurality of input areas determined to overlap with the pointer in at least one of the central portions of the sides surrounding the display screen.

Thus, the input area can be rearranged at a position that is clearly isolated on the display screen.
In addition, the interface display unit sets an input area displayed other than the input area rearranged by the rearrangement unit to an invalid state in which an input operation is not accepted.

As a result, the rearranged input area is located in the vicinity of another input area, so that it is possible to prevent a situation in which an input operation is performed again on a plurality of input areas. Can be further improved.
Further, the interface display means displays an annotation area suggesting the operation content of the input area for the input area rearranged by the rearrangement means.

Thereby, it is possible to surely perform an input operation intended by the user, and it is possible to further improve the operability of the graphical user interface.
In addition, the interface display means, after an input operation is performed on any of the input areas rearranged by the rearrangement means, a plurality of rearranged input areas other than the rearranged positions It is characterized by being displayed at a position on the display screen.

Thereby, it is possible to prevent the input area from being biased to a specific position on the display screen in the process of performing various input operations.
The present invention also provides:
An information processing program for controlling an information processing apparatus including an imaging unit that captures an image of a subject, wherein an interface display function for displaying a graphical user interface on a display screen, and an image of a preset object are A pointer recognition function for recognizing an image of the object as a pointer in the graphical user interface and a pointer recognized by the pointer recognition function in the graphical user interface when the image is taken by the imaging unit. When it is determined by the overlap determination function that the pointer is overlapped with the plurality of input areas by the overlap determination function for determining whether or not the input areas are overlapped with each other displayed for operation And a plurality of overlapping pointers The pass each other, is characterized in that to realize the relocation function of relocating to a more distant position on the display screen.

  According to the present invention, when an input area having a relatively small area is displayed densely in a graphical user interface, it is easy to perform an input operation intended by the user, and the operability of the graphical user interface is improved. Is possible.

Embodiments of an information processing apparatus according to the present invention will be described below with reference to the drawings.
First, the configuration will be described.
FIG. 1 is a schematic diagram showing an external configuration of an information processing apparatus 1 to which the present invention is applied. FIG. 1A is a front view and FIG. 1B is a rear view. FIG. 2 is a block diagram illustrating a functional configuration of the information processing apparatus 1. In addition, in FIG. 1, the indicator 2 used at the time of the instruction | indication input with respect to the information processing apparatus 1 is shown collectively.

1 and 2, the information processing apparatus 1 includes an imaging device 10, a captured image processing unit 20, a control unit 30, a storage unit 40, a display control unit 50, and a display 60. .
The imaging apparatus 10 converts an image formed by a lens into an electrical signal by a photoelectrically convertible device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. Then, the imaging device 10 outputs image data (hereinafter referred to as “captured image data”) indicated by the electrical signal to the captured image processing unit 20.

The captured image processing unit 20 further includes an image processing unit 21 and an image processing memory 22.
The image processing unit 21 performs processing for converting the captured image data input by the imaging device 10 into image data of a predetermined format such as color interpolation or size change, and stores the processing result in the image processing memory 22. Let

The image processing memory 22 is configured by a semiconductor memory such as SRAM (Static Random Access Memory), and stores captured image data input by the imaging device 10 and image data of a predetermined format that is a processing result of the image processing unit 21. To do.
The control unit 30 further includes a processor 31 and a memory 32.
The processor 31 controls the entire information processing apparatus 1 and performs various processes stored in the storage unit 40 in accordance with various instruction signals input by a power button (not shown) or a graphical user interface (GUI) described later. Read and execute the program related to. For example, when the power button is pressed and the information processing apparatus 1 is turned on, the processor 31 reads a button input control processing program stored in the storage unit 40 and executes a button input control process described later.

  In this button input control process, the processor 31 displays a button image on the GUI and sets a “contact determination area” in the button image. The contact determination area is an area for determining whether or not the button is pressed. If the pointer partially overlaps the contact determination area, it is determined that the button is pressed. The contact determination area is basically set to coincide with the button image, but is set to be slightly wider than the button image (for example, extending from the outline of the button image to a predetermined pixel outside). Or slightly narrower than the button image (for example, reduced by a predetermined pixel inward from the outline of the button image).

Then, the processor 31 stores various processing results in predetermined areas of the memory 32 and the storage unit 40 or outputs them to the display control unit 50.
The memory 32 is composed of a semiconductor memory such as a DRAM (Dynamic Random Access Memory), and has a work area for temporarily storing various data generated in various processes executed by the processor 31.

  The storage unit 40 is configured by a nonvolatile storage device such as a flash ROM, and stores data held in a nonvolatile manner such as an operating system program, an application program, a system data file, or a button input control processing program of the information processing device 1. ing. The storage unit 40 stores various parameters set by the user and various data acquired during operation of the information processing apparatus 1.

The display control unit 50 includes a display controller and a display driver, and generates a control signal for causing the display 60 to display the image data input by the processor 31.
The display 60 is a display device such as an LCD (Liquid Crystal Display), and displays various images according to control signals generated by the display control unit 50.

Subsequently, the indicator 2 used in the present embodiment will be described.
The indicator 2 shown in FIG. 1 is composed of a rod-shaped holding portion 2a that is a portion held by a user, and a spherical recognition portion 2b attached to the tip of the holding portion 2a.
The recognized portion 2b is colored red, and the information processing apparatus 1 identifies the recognized portion 2b from the shape and color in the captured image. The image of the recognized portion 2b is used as a pointer (cursor) in the GUI, and the user can input various instructions by moving the pointing tool 2 in each direction with the holding portion 2a. .

The specific form (shape, color, etc.) of the pointing tool 2 can be determined according to the design. For example, the holding portion 2a is a cylindrical shape having a diameter of 5 millimeters and a length of 10 centimeters. The recognition unit 2b can be a sphere having a diameter of 2 centimeters.
Next, the operation will be described.
The information processing apparatus 1 starts button input control processing upon power-on, and thereafter, when an input operation is performed on the GUI, the input operation on the button is controlled based on the button input control processing.

In this button input control process, when the GUI button is pressed by the image of the pointing tool 2 preset as the GUI pointer, the button that is in contact with the image of the pointing tool 2, that is, the button can be pressed. The appropriate buttons can be input by rearranging the buttons having the characteristic in the display screen.
FIG. 3 is a flowchart showing button input control processing executed by the processor 31 of the information processing apparatus 1.

  In FIG. 3, when the button input control process is activated, the processor 31 acquires image data in a predetermined format by the imaging device 10 and the captured image processing unit 20 (step S1), and in the acquired image data, a GUI pointer is acquired. A process for extracting the image area of the pointing tool 2 set as (hereinafter referred to as “object extraction process”) is executed (step S2).

Next, the processor 31 rearranges the buttons and displays a new input screen according to the positional relationship between the image area of the pointing tool 2 extracted in step S2 and the area of the displayed button. Processing (hereinafter referred to as “button rearrangement processing”) is executed (step S3), and thereafter steps S1 to S3 are repeated.
Next, the object extraction process executed in step S2 of the button input control process will be described.

FIG. 4 is a flowchart showing the object extraction process executed in the button input control process.
When the object extraction process is started, the processor 31 extracts a red area (hereinafter referred to as “target area”) from the image data acquired in step S1 of the button input control process (step S101).

Here, the reason why the red region is extracted is that the recognized portion 2b of the pointing tool 2 is colored red.
Next, the processor 31 converts the extracted red region (target region) into black and the region other than red into white, and binarizes the image data (step S102).
Next, the processor 31 performs image reduction processing by thinning out pixels included in the image data (step S103).

By performing the reduction process, an area that can be regarded as noise, such as a target area consisting of one pixel, can be erased.
Then, the processor 31 returns the image reduced in step S103 to the original size again (step S104), and assigns (labels) an identification number to each of the regions remaining as target regions at this time (step S105). .

Furthermore, the processor 31 derives the circularity for each target region to which the identification number is assigned in step S105 (step S106).
Note that the circularity of each target region is derived by matching the contour with a circle.
Then, the processor 31 acquires the position of each pixel included in the target area of the identification number derived that has the highest circularity (step S107), and returns to the button input control process.

Next, the button rearrangement process executed in step S3 of the button input control process will be described.
FIG. 5 is a flowchart showing a button rearrangement process executed in the button input control process.
When the button rearrangement process is started, the processor 31 determines whether or not the pointer overlaps the plurality of contact determination areas in the GUI (step S201), and the pointer overlaps the plurality of contact determination areas. If it is determined that the button is present, the buttons determined to have overlapping pointers are rearranged at positions clearly separated on the display screen (for example, four corners of the display screen) (step S202).

Then, if there is a button other than the button rearranged in step S202, the processor 31 invalidates the button (a state in which an input is not accepted and the user cannot operate) (step S203).
FIG. 6 is a schematic diagram illustrating processing in the case where it is determined in step S201 that the pointers overlap in a plurality of contact determination areas.

In FIG. 6A, the first to fourth contact determination areas overlap with the pointer, and it is determined that the pointer is in contact with the pointer in step S201.
In FIG. 6B, the first to fourth contact determination areas are rearranged at the four corners of the display screen, and the fifth contact determination area is invalidated.
Returning to FIG. 5, when it is determined in step S201 that the pointers do not overlap in the plurality of contact determination areas, and after step S203, the processor 31 displays the operation content corresponding to the contact determination area in which the input operation has been performed. A subroutine to be executed is started (step S204), and the process returns to the button input control process.

In step S204, an area in which an annotation related to the operation content of each contact determination area is described (hereinafter referred to as “annotation area”) is displayed as an input operable area, and each contact determination area is invalidated. It is also possible to accept an input operation for.
FIG. 7 is a diagram illustrating an example of a display screen when an annotation area is displayed for each contact determination area. An annotation area indicating instruction input contents (zoom-in, capture, and zoom-out) for controlling the imaging device 10 is illustrated. It is displayed.

When the annotation area is displayed in this way, the processor 31 determines whether or not the pointer overlaps any one of the annotation areas (that is, whether or not an input operation has been performed on any one of the annotation areas). If it is determined that the pointer overlaps with any annotation area, a subroutine for executing the operation content corresponding to the annotation area is started.
By displaying the annotation area and accepting the input operation in this way, when it is determined that a plurality of buttons have been pressed, the user can clearly select the button to be pressed again.

Note that the above-described annotation area may not be an area where input operation can be performed, but may be an area that simply describes an annotation related to the operation content.
Here, in the button rearrangement process described above, the example in which the pointer overlaps the first to fourth contact determination areas has been described, but when the pointer overlaps five or more contact determination areas, It is possible to select four contact determination regions in descending order of overlapping area and rearrange them at the four corners of the display screen.

FIG. 8 is a schematic diagram showing processing when the pointer overlaps five or more contact determination areas.
In FIG. 8A, the pointer overlaps with the first to fifth contact determination areas, and the overlap area between the fourth contact determination area and the pointer is the smallest.
In FIG. 8B, four contact determination areas other than the fourth contact determination area are selected and rearranged at the four corners of the display screen, and the fourth contact determination area is invalidated.

Further, in addition to the case where the contact determination areas are rearranged at the four corners of the display screen, it is only necessary to be able to be clearly separated on the display screen. Therefore, as shown in FIG. It is also possible to rearrange the contact determination areas at a total of six locations in the center of the lower side. Furthermore, the contact determination area may be rearranged in the center of the left and right sides on the display screen.
In the button rearrangement process, after an input operation is performed on any of the contact determination areas, the buttons rearranged at the four corners of the display screen are arbitrary other than these as shown in FIG. Moved to the position.

Thereby, it is possible to prevent the buttons displayed on the GUI from being biased and displayed at the four corners of the display screen.
As described above, the information processing apparatus 1 according to the present embodiment determines that a plurality of buttons displayed on the graphical user interface are pressed, that is, the pointer overlaps with a plurality of contact determination areas. If it is not possible to clearly determine which button is pressed, the buttons that may be pressed are rearranged at positions clearly isolated on the display screen.

Therefore, when buttons having a relatively small area are densely displayed in the graphical user interface, the user can easily press the intended button, and the operability of the graphical user interface can be improved.
In addition, the information processing apparatus 1 according to the present embodiment determines that a plurality of buttons have been pressed, and when the buttons are rearranged at the four corners of the display screen, the annotation area for the buttons is displayed and the buttons Instead, an input operation can be performed by pressing the annotation area.

Therefore, it is possible to reliably perform an input operation intended by the user, and it is possible to further improve the operability of the graphical user interface.
In addition, the information processing apparatus 1 according to the present embodiment disables buttons other than the rearranged buttons.
Therefore, when the rearranged buttons are located in the vicinity of other buttons, a situation where a plurality of buttons are pressed again can be prevented, and the operability of the graphical user interface can be further improved. It becomes.

Further, in the information processing apparatus 1 according to the present embodiment, after an input operation is performed on the rearranged buttons, the rearranged buttons are rearranged at arbitrary positions other than the rearranged positions. The
Accordingly, it is possible to prevent the buttons from being biased and displayed at the four corners of the display screen in the process of performing various input operations.

It is the schematic which shows the external appearance structure of the information processing apparatus 1 to which this invention is applied. 2 is a block diagram illustrating a functional configuration of the information processing apparatus 1. FIG. 4 is a flowchart illustrating button input control processing executed by a processor 31 of the information processing apparatus 1. It is a flowchart which shows the target object extraction process performed by button input control processing. It is a flowchart which shows the button rearrangement process performed in a button input control process. It is a schematic diagram which shows a process when it determines with the pointer overlapping in the some contact determination area | region. It is a figure which shows the example of a display screen in the case of displaying an annotation area | region about each contact determination area | region. It is a schematic diagram which shows a process in case the pointer overlaps with five or more contact determination areas. It is a figure which shows the example of a display screen in the case of rearranging a contact determination area | region in a total of six places of the four corners of a display screen and the center of the upper side and lower side of a display screen. It is a figure which shows the state by which a contact determination area | region is moved after input operation.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Information processing apparatus, 2 Indicator, 2a Holding part, 2b Recognized part, 10 Imaging apparatus, 20 Captured image processing unit, 21 Image processing part, 22 Image processing memory, 30 Control part, 31 Processor, 32 Memory, 40 Storage unit, 50 display control unit, 60 display

Claims (7)

An information processing apparatus including an imaging unit that captures an image of a subject,
Interface display means for displaying a graphical user interface on a display screen;
Pointer recognition means for recognizing the image of the object as a pointer in the graphical user interface when an image of the object set in advance is taken by the imaging means;
Overlap determination means for determining whether or not the pointer recognized by the pointer recognition means overlaps with a plurality of input areas displayed for performing a predetermined input operation in the graphical user interface;
When the overlap determining unit determines that the pointer overlaps with the plurality of input areas, the plurality of areas overlapping with the pointer are rearranged at positions farther apart on the display screen. Relocation means to
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the rearrangement unit arranges the plurality of input areas determined to overlap the pointer at at least one of four corners of the display screen. .   2. The rearrangement unit arranges the plurality of input areas determined to overlap with the pointer in at least one of the central portions of the sides surrounding the display screen. 2. The information processing apparatus according to 2.   4. The interface display unit according to claim 1, wherein the input area displayed other than the input area rearranged by the rearrangement unit is set to an invalid state in which an input operation is not accepted. The information processing apparatus according to item.   5. The interface display unit according to claim 1, wherein an annotation region suggesting an operation content of the input region is displayed for the input region rearranged by the rearrangement unit. 6. Information processing device.   The interface display means displays a plurality of the rearranged input areas other than the rearranged positions after an input operation is performed on any of the input areas rearranged by the rearrangement means. The information processing apparatus according to claim 1, wherein the information processing apparatus is displayed at a position on the screen. An information processing program for controlling an information processing apparatus including an imaging unit that captures an image of a subject,
An interface display function for displaying a graphical user interface on a display screen;
A pointer recognition function for recognizing an image of the object as a pointer in the graphical user interface when an image of the object set in advance is taken by the imaging unit;
An overlap determination function for determining whether or not the pointer recognized by the pointer recognition function overlaps a plurality of input areas displayed for performing a predetermined input operation in the graphical user interface;
When it is determined by the overlap determination function that the pointer overlaps with the plurality of input areas, the plurality of areas overlapping with the pointer are rearranged at positions farther apart on the display screen. Relocation function to
An information processing program characterized by realizing the above.

Images