JP2012226691A - Display control device and display control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To control a display mode in response to a movement of a pointer approaching a display screen when three-dimensionally detecting a user's operation against the display screen.SOLUTION: A display device detects a contact operation which touches a display screen and a proximity operation which does not touch the display screen but is close to the display screen. The display device determines whether a retention time, for which a pointer detected when performing the proximity operation is included in a position determination area, is equal to or more than a predetermined threshold (S2), when the retention time is equal to or more than the threshold, performs enlarged display of an image according to a position of the pointer (S3). On the other hand, when the retention time is less than the predetermined threshold, the display device does not perform enlarged display of the image.

Description

  The present invention relates to display control when an operation on a display surface is performed using an indicator such as a finger.

  A technique for three-dimensionally detecting an operation for detecting an operation on a display surface of an image is known. For example, Patent Document 1 discloses an input device that enlarges and displays the currently displayed content when a finger approaches the touch panel. Patent Documents 2 and 3 also disclose similar display control.

JP 2007-280316 A JP 2008-117371 A JP 2009-116583 A

Since the user's operation may vary depending on the situation at that time, for example, there are cases where enlarged display is suitable and cases where it is not. Therefore, if it is uniformly determined whether or not to perform a specific display control, the display control is not desired by the user.
Therefore, an object of the present invention is to control a display mode according to an operation when an indicator approaches the display surface when a user operation on the display surface is detected three-dimensionally.

  The display control apparatus according to an aspect of the present invention is an operation unit that repeatedly detects the position of the indicator with respect to the display surface and receives an operation, and a contact operation that causes the indicator to contact the display surface; An operation unit that detects a proximity operation for bringing the indicator close to the display surface without contacting the display surface, and an operation mode related to a display mode of the display surface based on a transition of the proximity operation detected by the operation unit. Determining means for determining; and specifying means for specifying a display mode of an image displayed on the display surface in accordance with the operation mode determined by the determining means and the position of the indicator detected by the operating means; And display control means for displaying an image on the display surface in the display mode specified by the specifying means.

In a preferred aspect, the discrimination means discriminates the operation mode based on a residence time in which a three-dimensional position of the indicator indicated by the proximity operation detected by the operation means is included in a predetermined area set in advance.
In another preferable aspect, the determination unit determines that the operation mode is a predetermined mode when the residence time is longer than a predetermined threshold, and the specifying unit is determined by the determination unit. When the operation mode is the predetermined mode, a display mode for highlighting a part of the image displayed on the display surface is specified.
In still another preferred embodiment, the specifying means includes a range of highlighting on the display surface, an orthogonal projection coordinate of the position of the indicator with respect to the display surface, and a distance from the display surface at the position. And at least one of
In still another preferred aspect, the specifying unit specifies a magnification when an image displayed on the display surface is enlarged.
In still another preferred aspect, the specifying unit specifies a pop-out amount when the image displayed on the display surface is stereoscopically displayed.
In still another preferred aspect, the display control means cancels the highlight display when a contact operation is detected by the operation means after the image is highlighted, and the position detected by the operation unit is Highlighting is not performed again until it is outside the predetermined area.
In still another preferred embodiment, the determination unit is configured such that the position detected by the operation unit enters the determination region until the position is detected as the contact operation, or the position leaves the determination region. Is calculated as the residence time.
In still another preferred aspect, the determination unit determines the operation mode based on a moving speed of the indicator indicated by the proximity operation detected by the operation unit.
In still another preferred aspect, the specifying unit uses the control data stored in the storage unit according to the operation mode determined by the determination unit and the position of the indicator detected by the operation unit. Specify the display mode.
In still another preferred aspect, the storage means stores the control data according to an application, and the specifying means uses the control data according to an application corresponding to an image displayed on the display surface. Specify the display mode.
In still another preferred aspect, the display control device includes a filter unit that executes a filter process for reducing noise with respect to coordinate information representing a position detected by the operation unit, and the determination unit includes the filter unit. The operation mode is discriminated based on the coordinate information on which the filtering process is executed.
In still another preferred aspect, the operation means includes a first sensor that detects the contact operation and a second sensor that detects the proximity operation.
In still another preferred aspect, the operation means includes a single sensor that detects the contact operation and the proximity operation.

  The display control method according to another aspect of the present invention is a step of repeatedly detecting a position of a pointer with respect to a display surface and receiving an operation, the contact operation for bringing the pointer into contact with the display surface, A display mode of the display surface based on a first step of detecting a proximity operation for bringing the indicator close to the display surface without contacting the display surface, and a transition of the proximity operation detected in the first step Display on the display surface in accordance with a second step of determining an operation mode relating to, an operation mode determined in the second step, and a position of the indicator detected in the first step. A third step of specifying an image display mode; and a fourth step of displaying an image on the display surface in the display mode specified in the third step.

  ADVANTAGE OF THE INVENTION According to this invention, when detecting a user's operation with respect to a display surface three-dimensionally, it is possible to control a display mode according to operation | movement when a pointer approaches a display surface.

Figure showing the appearance of the display device Block diagram showing the hardware configuration of the display device Functional block diagram showing the functional configuration of the control unit The figure which showed an example of the display control table A flowchart showing display control realized by the control unit Diagram showing the relationship between the detectable area and the discrimination area The figure which showed the example of a display for every operation mode by a display device Flow chart showing display control realized by control unit (second embodiment) Block diagram showing configuration of display device (third embodiment) The flowchart which showed the display control which a control part implement | achieves (4th Embodiment). Flow chart showing display control realized by control unit (fifth embodiment)

[Summary of the Invention]
The present invention is characterized by display control based on the characteristics of an operation when a pointer approaches a display surface without touching the display surface in a display device that three-dimensionally detects a user operation on the display surface. Here, the indicator refers to a body part or an instrument for the user to perform an operation pointing to the display surface, and is, for example, a user's finger or a stylus (stylus pen).

  In general, when a user tries to perform a fine operation, the movement of the indicator tends to be slow. For example, Fitts's law indicates that the smaller the object of operation (buttons, etc.), the more time it takes for the user to operate, but it is known that it closely matches the actual user's action. Yes. In general, it is recognized that the user, such as a beginner or an elderly person, is more unfamiliar with the operation by the indicator, and the operation by the user becomes slower. Therefore, when the feature that the user's operation is relatively slow is indicated, it can be said that the display mode that assists or supports the user's operation is likely to contribute to the user.

  Therefore, the present invention focuses on such a feature of the user's operation and switches the display mode between the case where the operation of the feature is performed and the case where the operation is not performed. Specifically, the present invention determines the movement of the indicator based on the time or speed at which the indicator is detected, and makes the display mode of the display surface different from normal when the user's operation satisfies a predetermined condition. This makes it easy to operate the display surface. Such a display mode (a display mode different from the normal mode) is to highlight an image, for example, to display an image that the user intends to indicate in an enlarged manner or to display it in a three-dimensional manner. By realizing such highlighting, the user can more easily specify (ie, select) an object such as a button.

[First Embodiment]
FIG. 1 is a diagram illustrating an appearance of a display device 100 according to an embodiment of the present invention. The display device 100 is an information processing device that allows a user to perform operations on the display surface 101. For example, a mobile phone, a smartphone, a PDA (Personal Digital Assistant), a tablet PC (Personal Computer) or a slate PC, a game Machine, electronic book reader, etc. The display surface 101 is a surface for displaying an image and a surface for receiving a user operation. The display device 100 is configured to detect the position of the fingertip of the user's hand. That is, the indicator that is the detection target of the present embodiment is the user's finger. For convenience of explanation, the surface shown in FIG. 1, that is, the surface having the display surface 101 will be referred to as “front” hereinafter.

  FIG. 2 is a block diagram illustrating a hardware configuration of the display device 100. As illustrated in FIG. 1, the display device 100 includes a control unit 110, a storage unit 120, a display unit 130, an operation unit 140, and a communication unit 150. The display unit 130 and the operation unit 140 may be configured integrally or may be separated. The communication unit 150 is not an essential component for the present invention.

  The control unit 110 is a unit that controls the operation of each unit of the display device 100. The control unit 110 includes an arithmetic processing device such as a CPU (Central Processing Unit) and a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and executes a program stored in the ROM or the storage unit 120. Thus, display control of the display unit 130 and communication control of the communication unit 150 are realized. The control unit 110 has a function of recognizing the current time based on the system clock and measuring the time.

  The storage unit 120 is a unit that stores data used by the control unit 110 for control. The storage unit 120 is configured by a hard disk or a flash memory. The storage unit 120 may include so-called removable media, that is, removable storage means. The storage unit 120 stores a display control table to be described later.

  The display unit 130 is a means for displaying an image on the display surface 101. The display unit 130 includes a display panel that displays an image using a liquid crystal element or an organic EL (electroluminescence) element, a drive circuit that drives the display panel, and the like. Here, it is assumed that the display surface 101 is a rectangle, and pixels are arranged in a matrix along the short side and the long side of the rectangle. As shown in FIG. 1, it is assumed that a three-dimensional orthogonal coordinate system having an appropriate position (here, the upper left corner) as the origin O is defined for the display surface 101. Here, the X axis is defined in the long side direction of the display surface 101, the Y axis is defined in the short side direction of the display surface 101, and the Z axis is defined in a direction orthogonal to the X axis and the Y axis. The Z-axis is a coordinate axis with the surface of the display surface 101 as the origin and the direction toward the user (upward in FIG. 1) as the positive direction.

  The operation unit 140 is a unit that receives a user operation. The operation unit 140 includes a sensor for detecting the position of the user's fingertip with respect to the display surface 101 and supplying coordinate information representing the position to the control unit 110. More specifically, the operation unit 140 includes a contact sensor 141 and a proximity sensor 142. The contact sensor 141 is a sensor for detecting a state in which the user's fingertip is in contact with the display surface 101. The contact sensor 141 can be realized by, for example, a known touch screen (also referred to as a touch panel). On the other hand, the proximity sensor 142 is a sensor for detecting a state in which the user's fingertip is close to the display surface 101. The proximity sensor 142 may also be a sensor using a well-known technique. For example, the proximity sensor 142 is realized by detecting the capacitance of the fingertip or optically detecting the position of the fingertip.

  The contact sensor 141 two-dimensionally detects the position of the surface of the display surface 101 where the user's fingertip has contacted. Therefore, the coordinate information supplied by the contact sensor 141 is coordinate information representing coordinates in the X-axis direction (X coordinate) and coordinates in the Y-axis direction (Y coordinate). On the other hand, the proximity sensor 142 three-dimensionally detects the position of the user's fingertip that is in close proximity without touching the display surface 101. Therefore, the coordinate information supplied by the proximity sensor 142 is coordinate information representing the coordinates in the respective directions of the X axis, the Y axis, and the Z axis. Hereinafter, the operation specified by the coordinate information supplied by the contact sensor 141 is referred to as “contact operation”, and the operation specified by the coordinate information supplied by the proximity sensor 142 is referred to as “proximity operation”.

  The proximity sensor 142 has a limited range in which the user's fingertip can be detected (hereinafter referred to as “detectable region”). The detectable area is determined by the hardware performance of the proximity sensor 142. “Proximity” as used in the present embodiment refers to a state where the position of the fingertip (particularly the position in the Z-axis direction) is in this detectable region, and more specifically, a state where the position of the fingertip is included in the determination region described later It points to.

  The contact sensor 141 and the proximity sensor 142 detect the position of the indicator at a predetermined sampling rate. That is, the contact sensor 141 and the proximity sensor 142 repeatedly and continuously detect the position of the indicator. The proximity sensor 142 may temporarily lower the sampling rate when the user's finger is not detected for a predetermined time or longer, and return to the original sampling rate when the finger is detected again. The contact sensor 141 may function after the proximity sensor 142 detects a finger. This is because the finger always comes close to the display surface 101 in advance for the finger to contact the display surface 101.

  The operation unit 140 may be a unit that accepts not only operations on the display surface 101 but also other operations. For example, the display device 100 may include a physical key such as a button or a switch on a portion other than the display surface 101 (including a side surface and a back surface), and the operation unit 140 may be configured to accept an operation using the physical key. . In this case, the operation unit 140 supplies operation information representing an operation with a physical key to the control unit 110. The physical key here may be a so-called QWERTY keyboard or numeric keypad.

  The communication unit 150 is a means for communicating with an external device. The communication unit 150 includes, for example, an antenna and a network adapter. The communication unit 150 may communicate with an external device via a network such as the Internet or a mobile communication network, but may communicate directly with an external device without using a network, such as near field communication. The communication by the communication unit 150 is wireless communication here, but may be wired communication.

  The hardware configuration of the display device 100 is as described above. Based on this configuration, the display device 100 communicates with an external device and displays an image corresponding to the process being executed on the display surface 101. At this time, the display device 100 detects the proximity operation and the contact operation, and detects the position of the user's fingertip. When detecting the proximity operation, the display device 100 changes the display mode of the image as necessary according to the transition of the detected position of the fingertip.

  In the present embodiment, the display device 100 enlarges and displays an image when the fingertip stays in the detectable region for a predetermined time or more and the contact operation is still not detected. The display device 100 may enlarge and display only a part of the image displayed on the display surface 101 (for example, an image in the vicinity of the fingertip position), but the entire image displayed on the display surface 101 is displayed. An enlarged display may be used. However, in the latter case, it is desirable to control the display position of the image so that the position that the user intends to indicate is not hidden by the enlarged display.

  In the present embodiment, there are a plurality of enlarged image display modes. In addition, the image display mode may include an image that does not perform enlarged display even when the user's operation satisfies a predetermined condition. These display modes are defined in advance and can be set by the user according to his / her preference. Or the display mode of an image may differ for every application (application corresponding to the image currently displayed on the display surface 101) in execution.

  FIG. 3 is a functional block diagram showing a part related to image display control in the functional configuration of the control unit 110. The control unit 110 executes functions corresponding to the data acquisition unit 111, the determination unit 112, the specifying unit 113, and the display control unit 114 illustrated in FIG. These functions may be realized by either an OS (Operating System) or specific software that controls image display, or may be realized by cooperation of a plurality of software.

  The data acquisition unit 111 is means for acquiring data necessary for image display. Specifically, the data acquisition unit 111 is image data indicating an image to be displayed on the display surface 101 and coordinate information supplied by the operation unit 140. The image data may be stored in advance in the storage unit 120 (for example, image data used when executing a specific application) or received by the communication unit 150 from an external device (for example, by a browser). Web page data to be displayed).

  The determination unit 112 is means for determining an operation mode related to the display mode of the display surface 101 based on the coordinate information supplied from the proximity sensor 142, that is, based on the proximity operation detected by the proximity sensor 142. Here, the operation mode is to determine the image display method and to determine whether or not to highlight the image. There are two types of operation modes according to the present embodiment: “proximity operation effective mode” for enabling enlarged display by proximity operation and “proximity operation invalid mode” for invalidating enlarged display by proximity operation. Which of these operation modes the display device 100 is operated in is set in advance. Further, the operation mode is determined in more detail according to the coordinate (Z coordinate) in the Z-axis direction of the coordinate information supplied from the proximity sensor 142, that is, the distance from the display surface 101 of the position of the fingertip. In the present embodiment, these operation modes are described in the display control table. Therefore, the determination unit 112 performs the determination by referring to the display control table stored in the storage unit 120.

  In the present embodiment, the determination unit 112 determines the operation mode based on the time during which the position of the fingertip detected by the proximity sensor 142 is included in the predetermined area, that is, the time during which the fingertip remains in the predetermined area. Here, the predetermined area is an area included in the above-described detectable area, and may be an area in the same range as the detectable area. However, in the present embodiment, the predetermined area is an area smaller than the detectable area. Suppose there is. Such an area is hereinafter referred to as a “discrimination area”. The detectable area is an area defined in hardware, while the discrimination area is an area defined in software. The time measured by the determination unit 112 to determine whether or not to perform enlarged display is hereinafter referred to as “residence time”.

  The dwell time is preferably set such that, after the position of the fingertip detected by the proximity sensor 142 enters the determination area, the Z coordinate of the position becomes 0 (that is, the fingertip touches the display surface 101), or the position Is the time until the finger leaves the discrimination area, and is the time during which the position of the fingertip continues to be included in the discrimination area. However, the staying time may continue to be accumulated while ignoring such time as long as the time when the fingertip leaves the determination area is very short.

  FIG. 4 is a diagram illustrating an example of the display control table. The determination unit 112 refers to a table corresponding to the time during which the position of the fingertip detected by the proximity sensor 142 continues to be included in the determination region (that is, the residence time). The determination unit 112 refers to the table 4 when the residence time is less than the predetermined threshold, and selects an operation mode according to the table 4, while when the residence time is equal to or greater than the predetermined threshold, One of the tables 1 to 3 is referred to, and an operation mode according to the referenced table is selected. Note that which of the tables 1 to 3 the determination unit 112 refers to may be determined in advance by the user, or may be determined by an application corresponding to an image to be enlarged and displayed.

  In FIG. 4, Tables 1 to 3 are examples of tables used in the proximity operation effective mode. On the other hand, the table 4 is an example of a table used in the proximity operation invalid mode. Here, “Z” represents the Z coordinate, and Th1 to Th4 represent threshold values set for the Z coordinate. Although details will be described later, if the detectable area is larger than the range of the discrimination area, the position satisfying Z = Th4 is set to the boundary of the discrimination area (the side closer to the user of the two boundaries in the Z-axis direction). Or a boundary of a detectable region (a boundary closer to the user of the two boundaries in the Z-axis direction). In the former case, the proximity operation in the determination area is possible, but in the latter case, the proximity operation in the detectable area is possible. However, for the sake of simplification of explanation, it is assumed that the position satisfying Z = Th4 below is the boundary of the discrimination region. In other words, only the position of the fingertip within the discrimination area is used for the enlarged display control.

  Table 1 is a table showing a display mode in which enlarged display is not performed when Z> Th4, and enlarged display is performed at a constant magnification (m times) when 0 <Z ≦ Th4. When the display device 100 performs an enlarged display using the table 1, the image displayed on the display surface 101 is displayed at a constant magnification at a certain time interval, and the position of the fingertip changes slightly. However, the magnification does not change as long as the position is within the discrimination area. In the case of Table 1, the magnification m is an appropriate value that satisfies m> 1.

  On the other hand, Table 2 is the same as Table 1 in that the enlarged display is not performed when Z> Th4, but the magnification of the enlarged display changes according to the magnitude of the value of Z when 0 <Z ≦ Th4. It is a table which shows an aspect. When the display device 100 performs an enlarged display using the table 2, an image displayed on the display surface 101 is enlarged and displayed with a certain time as a boundary, and the magnification is increased as the finger is brought closer to the display surface 101. Increases gradually.

  The table 3 is the same as the table 2 in that the enlarged display is not displayed when Z> Th4 and the magnification of the enlarged display changes depending on the value of Z when 0 <Z ≦ Th4. The aspect of change in magnification is different from that in Table 2. Specifically, when the display device 100 performs an enlarged display using the table 3, the image displayed on the display surface 101 is displayed larger as Z is closer to Th4, and as Z is closer to 0 (that is, the fingertip). The display mode is such that the magnification approaches 1 (equal magnification) and approaches the original size (approaching the display surface 101). Such a magnification is represented by the product of Z and n. Here, the value of n is an appropriate value satisfying n> 0. Further, the change in magnification at this time is not continuous as in the case of Table 2, but is more continuous.

  Note that Table 4 corresponds to the proximity operation invalid mode, and the magnification remains at 1 (equal magnification) regardless of the value of Z. Therefore, when referring to the table 4, the display device 100 may omit detecting the proximity operation by the proximity sensor 142, and may detect only the contact operation by the contact sensor 141.

  The specifying unit 113 is means for specifying the display mode of the image displayed on the display surface 101 according to the operation mode determined by the determining unit 112 and the position of the fingertip detected by the operation unit 140. The specifying unit 113 specifies the magnification of the image based on the Z coordinate of the position of the fingertip when performing display control in which the magnification of the enlarged display is variable. Further, when displaying a part of the image displayed on the display surface 101 in an enlarged manner, the specifying unit 113 displays the X and Y coordinates of the fingertip position (that is, the coordinates of the orthogonal projection of the fingertip position on the display surface 101). ) To specify the position to enlarge the image.

  The display control unit 114 is a unit that displays an image on the display surface 101 in the display mode specified by the specifying unit 113. In the present embodiment, the display control unit 114 enlarges and displays the image in the display mode specified by the specifying unit 113 when the image is to be highlighted. Therefore, the specific display mode of the image differs depending on the position of the user's fingertip and the operation mode.

  FIG. 5 is a flowchart showing display control realized by the control unit 110 with the above-described functional configuration. When it is necessary to determine whether or not to enlarge the image, the control unit 110 executes the display control process shown in FIG. That is, the control unit 110 does not always need to execute the display control process illustrated in FIG. 5. For example, when a specific application that does not require enlarged display is being executed, the control unit 110 does not execute the process. May be.

  The display control by the control unit 110 is started when the user's fingertip enters the detectable region and a predetermined condition is satisfied. Therefore, first, the control unit 110 specifies the position of the proximity operation based on the coordinate information supplied from the proximity sensor 142, and determines whether or not the position is included in the detectable region (step S1). Control unit 110 repeats the same processing until this determination becomes affirmative (YES).

  When the position of the fingertip is included in the detectable area, control unit 110 determines whether or not the position is further included in the determination area, and whether the dwell time in the determination area is equal to or greater than a predetermined threshold value. It is determined whether or not (step S2). The determination in step S2 is negative (NO) when the fingertip approaches the display surface 101 and the user's operation shifts from the proximity operation to the contact operation or when the fingertip moves away from the display surface 101 and exits the determination area. Become. This determination is also negative when the residence time is less than a predetermined threshold.

  FIG. 6 is a diagram showing the relationship between the detectable area and the discrimination area. In the figure, the vertical coordinate axis represents the Z axis, but the horizontal coordinate axis may be either the X axis or the Y axis. For the convenience of illustration, the discrimination area is shown as a smaller area than the detectable area not only in the Z-axis direction but also in the X-axis direction (or Y-axis direction). The size of the direction (area when viewed from the front by the user) may be the same as that of the detectable region.

  When the dwell time in the fingertip discrimination area is equal to or greater than a predetermined threshold, the control unit 110 controls the display unit 130 to display an enlarged image in a display mode according to the position of the fingertip according to the display control table. (Step S3). In addition, the control unit 110 controls the display mode of the image as described above and determines a change in the position of the fingertip. Specifically, control unit 110 determines whether the fingertip is within the detectable region or whether the user operation has shifted from the proximity operation to the contact operation (step S4). At this time, once the user's fingertip comes out of the detectable region, the determination in step S4 becomes negative. Then, the control part 110 repeats a process from step S1.

  On the other hand, when the fingertip is within the detectable region or when the user's operation is a contact operation, the control unit 110 further determines which of these corresponds to the user's operation. Specifically, the control unit 110 determines whether or not the user operation is a contact operation (step S5). The contact operation and the proximity operation are distinguished by the Z coordinate of the coordinate information. Alternatively, the control unit 110 may distinguish between the contact operation and the proximity operation depending on whether the coordinate information is supplied from the contact sensor 141 or the proximity sensor 142.

  If the user operation is a contact operation, the control unit 110 changes the display of the image according to the position where the fingertip touches (step S7). At this time, the control unit 110 may execute a process corresponding to the position touched by the fingertip, that is, the object selected by the user. For example, if the user selects a hyperlink (a character string or an icon) of the web page, the control unit 110 receives the data described in the hyperlink and renders the data to switch the display of the page. , And so on. On the other hand, if the user's operation is not a contact operation (that is, if it is a proximity operation), control unit 110 repeats the processes after step S3.

  The control unit 110 also determines whether or not the user's operation is a contact operation even when the staying time in the determination region is less than the predetermined threshold in step S2 (step S6). Also in this case, if the user's operation is a contact operation, the control unit 110 executes the process of step S7. On the other hand, if the user's operation is not a contact operation, control unit 110 repeats the processing from step S1 onward in this case.

  If the process of step S7 is performed, the control part 110 will reset the value of the accumulated residence time, and will return to 0 (step S8). In addition, the control part 110 may reverse the execution order of the process of step S7 and the process of step S8. In addition to executing the process of step S7, the control unit 110 resets the dwell time value when the position indicated by the coordinate information is outside the detectable area (or determination area). May be.

  FIG. 7 is a diagram illustrating a display example for each operation mode by the display device 100. Each of these display examples shows a case where the user gradually approaches his / her finger to select the icon Ic2 in a state where the icons Ic1 to Ic6 are displayed. It should be noted that the display mode of the proximity operation effective mode is that in accordance with the table 3 shown in FIG. In the proximity operation valid mode, the display device 100 changes the image display mode in the flow Im1 → Im2 → Im3 → Im4, while in the proximity operation invalid mode, the image display mode is changed in the flow Im1 → Im4. Let

  The image Im1 is an image in an initial state, and is an image when the user's finger is outside the detectable region. On the other hand, the image Im2 is an image when the user performs a proximity operation and the staying time becomes a predetermined threshold value or more, and the image Im3 is displayed on the display surface 101 by the user more than the case of the image Im2. It is an image when it is brought close to. In these cases, the icon Ic2 is enlarged and displayed so that it is more conspicuous than the other icons Ic1, Ic3 to Ic6, and its confirmation and selection are easy. The image Im4 is an image when the icon Ic2 is selected.

  On the other hand, in the proximity operation invalid mode, even if the position of the user's fingertip is equivalent to the images Im2 and Im3, the display mode of the icon Ic2 does not change. Such a display mode is realized when the finger moves relatively quickly and the finger reaches the display surface 101 in a relatively short time. That is, in this case, the display device 100 does not change the image display mode by the proximity operation. Such an operation may be realized by the display device 100 not accepting the proximity operation itself, but may be realized by not reflecting the accepted proximity operation in the display mode.

  Note that the display mode of the enlarged display is not limited to the example shown in FIG. For example, the display device 100 does not enlarge only a specific object (in the case of FIG. 7, an icon), but may enlarge and display a predetermined range centered on the position indicated by the user. Good. Further, the display device 100 displays an enlarged image as a semi-transparent image (an image in a display mode in which the background can be seen through) so as not to hide and hide other parts by enlarging a part of the image. It may be. Further, the display device 100 may change the color of the object or blink the display in accordance with the enlarged display.

  As described above, the display device 100 according to the present embodiment is configured to determine whether or not to perform an enlarged display based on the residence time in the fingertip determination area. According to such a display device 100, it is possible to determine the operation mode according to the user's operation without the user having previously designated the operation mode (by an operation other than the proximity operation). Therefore, even when the display device 100 displays the same image, whether or not to enlarge the display according to the situation depends on the degree of proficiency with respect to the user's operation and the user's current operation speed. Can be changed.

  Such display control is also suitable, for example, when displaying an image in which a plurality of objects (buttons, icons, hyperlinks, etc.) are mixed as a user's selection target. According to the display device 100, when such an image is displayed, if the user tries to select a relatively small object by a slow operation, the object is enlarged and displayed. When a relatively large object is selected by a quick operation, it is possible to prevent the object from being enlarged and displayed.

  The display control of this embodiment is also suitable when the user is an elderly person or a disabled person. According to the display device 100, the user's hand shakes and the position where the user wants to touch is not fixed, or the finger is close to the display surface 101 because the image on the display surface 101 cannot be seen well. For example, when the user is not allowed to perform the operation, the image near the finger can be enlarged and displayed to assist or assist the operation.

[Second Embodiment]
In the present embodiment, a part of the configuration and operation of the first embodiment described above is modified. This embodiment is an example in the case where the display unit 130 enables three-dimensional display of an image. The three-dimensional display here is a display mode in which an image is perceived as if it is three-dimensional. The display unit 130 may realize what is called autostereoscopic viewing, but may also realize stereoscopic viewing using auxiliary equipment such as dedicated glasses. As a specific method for realizing stereoscopic viewing by the display unit 130, a known appropriate technique may be used.

  In addition, in embodiment after this embodiment, description of the part which is common in embodiment already demonstrated is abbreviate | omitted suitably. In addition, in the embodiments after this embodiment, the same reference numerals as those already described are attached to the components common to the already described embodiments and the processes in the flowchart.

  FIG. 8 is a flowchart showing the display control of this embodiment. This flowchart is the same as the display control of the first embodiment (see FIG. 5) except that the process of step S3 is replaced with the process of step S3a.

  In step S3a, the control unit 110 controls the amount of pop-up when displaying an image in three dimensions. Here, the pop-out amount represents a sense of visual distance between the user and the image, and the larger the pop-out amount, the closer the image is to the user (that is, the more popping out than the display surface 101). Represents being close to The control unit 110 may locally change the pop-out amount in accordance with the position of the fingertip, but may make the entire display surface 101 appear to pop out. In the display control table, the “magnification” portion in the first embodiment may be read as “the amount of protrusion”.

  Further, the control unit 110 may execute a combination of the display control of the present embodiment and the display control of the first embodiment. That is, the control unit 110 may control the display so that the image displayed on the display surface 101 is enlarged and displayed in a stereoscopic manner. In this case, the control unit 110 may make the part of the image displayed on the display surface 101 to be enlarged and the part to be stereoscopically displayed different.

[Third Embodiment]
FIG. 9 is a block diagram showing the configuration of the display device 100a of the present embodiment. As shown in the figure, the display device 100a is different from the display device 100 (see FIG. 2) in that a filter unit 160 is provided. The filter unit 160 is means for executing a filter process for reducing noise on the coordinate information. Note that the filter unit 160 may be hardware independent of other components, but may be realized as a function of the control unit 110 or the operation unit 140.

  Specifically, the filter used by the filter unit 160 is a smoothing filter, a median filter, a Gaussian filter, a moving average filter, or the like. That is, when the coordinates represented by the coordinate information fluctuate finely in a short time, the filter unit 160 converts the coordinate information so as to suppress the fluctuation. Therefore, the noise referred to here is a high-frequency component when coordinate information repeatedly and continuously detected is expressed in time series.

  According to the display device 100a of the present embodiment, it is possible to make the finger trajectory smoother than the actual trajectory when the control unit 110 is used for display control (that is, the positional variation is gentle). As a result, the control unit 110 may prevent the magnification (or the amount of pop-up when the image is displayed in 3D) from being abruptly (or frequently) changed when the image on the display surface 101 is enlarged and displayed. This makes it possible to prevent the image from blurring and becoming difficult for the user to see. Such display control is particularly effective when the magnification (or pop-out amount) changes continuously as in the case of the table 3 in FIG.

  Note that the filter unit 160 may perform the filtering process on each of the X coordinate, the Y coordinate, and the Z coordinate, but may perform the filtering process only on the Z coordinate. In addition, the filter unit 160 may change the mode of filter processing (more specifically, the degree of noise reduction) to be performed for each coordinate for each coordinate.

[Fourth Embodiment]
This embodiment is characterized by display control after enlarged display (or stereoscopic display). Specifically, in the present embodiment, when a contact operation is detected after the image on the display surface 101 is highlighted, the display device 100 cancels the highlight and returns to the normal display, and then the user Until the fingertip is once out of the discrimination area, the highlighting is limited not to be performed again. Such display control is particularly effective when screen transition such as page switching occurs due to a touch operation.

  FIG. 10 is a flowchart showing the display control of this embodiment. This flowchart is different from the display control of the first embodiment (see FIG. 5) in that the processes of steps S9 and S10 are added. In the present embodiment, the control unit 110 temporarily cancels the enlarged display in accordance with the user's contact operation in step S7.

  In step S9, the control unit 110 determines whether or not the fingertip position indicated by the coordinate information of the proximity operation is included in the determination area. When the determination is negative, that is, when the position of the fingertip goes out of the determination area, the control unit 110 repeats the processes after step S1. At this time, the control unit 110 can shift to the proximity operation effective mode if the dwell time becomes equal to or greater than the threshold value again.

  On the other hand, when the position of the fingertip is within the determination area, the control unit 110 operates in the proximity operation invalid mode as shown in step S10, and repeats the determination in step S9 until the position of the fingertip is outside the determination area. Note that the control unit 110 may invalidate the enlarged display based on the proximity operation instead of invalidating the proximity operation in Step S10.

  According to the display device 100 of the present embodiment, it is possible to prevent highlighting contrary to the user's intention from being performed after the contact operation. For example, when the display device 100 displays a web page by a browser, when the user selects a hyperlink in the web page by a contact operation, the page displayed by the browser is displayed as the hyperlink from the web page before selection. It is common to change to other web pages shown. Since such page transitions involve communication and rendering by the communication unit 150, it may take time until the pages are completely switched. At this time, the user may wait without moving the finger position much until the next page is displayed. In such a case, if the user's fingertip continues to stay in the determination area and the staying time exceeds a predetermined threshold, the next page is suddenly displayed in a highlighted state. . Such a display mode is highly likely to be a display mode not intended by the user. On the other hand, according to the display control of the present embodiment, since the highlighting is not performed unless the user removes his / her finger once, the highlighting is not intended by the user when there is little movement of the user's finger as described above. In contrast, it can be prevented from being performed.

  Note that the determination in step S9 may be performed based on a part of the determination area instead of the determination area itself. For example, the determination in step S9 is based on a region having a narrower range in the Z-axis direction than the determination region (that is, a region smaller than the determination region), and determines whether or not the fingertip position is included in the region. May be.

  In addition, as a modification of the determination in step S9, a contact operation may be used instead of the proximity operation. For example, when a contact operation is further detected after the contact operation in step S5 or S6, the control unit 110 calculates the distance between the coordinates of these two contact operations, and the distance and a predetermined threshold value are calculated. It may be determined whether to enable or disable highlighting by comparing.

[Fifth Embodiment]
In the present embodiment, the feature of the user's operation is discriminated based on the moving speed of the finger instead of the residence time described above. More specifically, the moving speed here is a moving speed when the fingertip of the user approaches the display surface 101. Further, the moving speed may be a speed of displacement of the fingertip in the Z-axis direction (that is, a speed not considering the displacement in the X-axis direction and the Y-axis direction), or considering the displacement in each of these three axes directions. The calculated speed may be used.

  FIG. 11 is a flowchart showing the display control of this embodiment. This flowchart is the same as the display control of the first embodiment (see FIG. 5) except that the process of step S2 is replaced with the process of step S2a. In step S2a, the control unit 110 calculates a moving speed based on a plurality of coordinate information indicating the proximity operation, and determines whether the moving speed is equal to or less than a predetermined threshold. The moving speed can be calculated by dividing the displacement between coordinates indicated by a plurality of (at least two points) coordinate information by the difference between the detection times.

  When the moving speed is equal to or lower than the threshold value, that is, when moving relatively slowly, the control unit 110 determines that the operation mode is the proximity operation effective mode, and executes the process of step S3. On the other hand, when the moving speed exceeds the threshold value, that is, when moving relatively quickly, the control unit 110 determines that the operation mode is the proximity operation invalid mode, and executes the process of step S6.

  Note that the control unit 110 can calculate the acceleration and jerk (jerk) of the fingertip position instead of the moving speed of the fingertip position, and use this for the determination. That is, the control unit 110 may determine the operation mode based on the moving speed per unit time or its acceleration instead of the fingertip displacement per unit time.

[Modification]
The present invention is not limited to the aspect of each embodiment described above, and can be implemented in other aspects. The present invention can also be implemented, for example, by the modes shown in the following modifications. In addition, this invention may be implemented in the aspect which combined these some modified examples, and may be implemented in the aspect which combined multiple characteristics of each embodiment mentioned above.

(1) As described above, the indicator of the present invention may be an instruction device such as a stylus that the user holds and moves. The operating means in the case of using such an indicator may detect the position of the indicator with infrared rays or ultrasonic waves. In addition, when an indicator having a magnetic body at the tip is used, the position of the indicator can be detected magnetically.

(2) The operation means of the present invention is not configured to separately include a sensor for detecting the contact operation and a sensor for detecting the proximity operation, but may be configured to detect the contact operation and the proximity operation with a single sensor. Good.

(3) The display control device of the present invention may be a part of the configuration of the display device 100 as in the above-described embodiment, but the display device and other devices provided independently of the display device It may be realized by cooperation. For example, in the case of a configuration in which the main body and the display device are separated as in a so-called desktop PC, the present invention is provided with operation means on the display device side and other means (discriminating means, specifying means, display control means, etc.) May be provided on the main body side.

  Or the display control apparatus of this invention may be provided with the means (data acquisition part 111) which replaces with an operation means and acquires the coordinate information supplied from the operation means. That is, the display control apparatus of the present invention can be configured only by the control unit 110 described above. Further, such a display control apparatus can be implemented in the form of a program for causing a computer to realize this and a recording medium on which such a program is recorded.

DESCRIPTION OF SYMBOLS 100, 100a ... Display apparatus, 101 ... Display surface, 110 ... Control part, 111 ... Data acquisition part, 112 ... Discrimination part, 113 ... Identification part, 114 ... Display control part, 120 ... Storage part, 130 ... Display part, 140 ... Operation part 141 ... Contact sensor 142 ... Proximity sensor 150 ... Communication part 160 ... Filter part

Claims (15)

An operation means for repeatedly detecting the position of the indicator with respect to the display surface and receiving an operation, the contact operation for contacting the indicator with the display surface, and without bringing the indicator into contact with the display surface An operation means for detecting a proximity operation to be close to,
A discriminating unit for discriminating an operation mode related to a display mode of the display surface based on a transition of the proximity operation detected by the operating unit;
Specifying means for specifying a display mode of an image displayed on the display surface according to the operation mode determined by the determining means and the position of the indicator detected by the operation means;
Display control means for displaying an image on the display surface in the display mode specified by the specifying means. The discrimination means includes
The operation mode is determined based on a dwell time included in a predetermined area in which a three-dimensional position of the indicator indicated by the proximity operation detected by the operation means is set in advance. Display control device. The discrimination means includes
When the residence time is longer than a predetermined threshold, it is determined that the operation mode is a predetermined mode,
The specifying means is:
The display mode for highlighting a part of an image displayed on the display surface when the operation mode determined by the determination unit is the predetermined mode. Display control device. The specifying means is:
The range of the display surface to be highlighted is specified using at least one of the coordinates of the orthogonal projection of the position of the indicator with respect to the display surface and the distance of the position from the display surface. The display control apparatus according to claim 3, wherein The specifying means is:
The display control apparatus according to claim 3 or 4, wherein a magnification for enlarging an image displayed on the display surface is specified. The specifying means is:
The display control apparatus according to claim 3 or 4, wherein an amount of pop-up when the image displayed on the display surface is stereoscopically displayed is specified. The display control means includes
When a touch operation is detected by the operation means after the image is highlighted, the highlight display is canceled and the highlight display is not performed again until the position detected by the operation unit is outside the predetermined area. The display control apparatus according to claim 3, wherein: The discrimination means includes
The dwell time is calculated as the time from when the position detected by the operation means enters the determination area until the position is detected as the contact operation or until the position leaves the determination area. A display control apparatus according to any one of claims 2 to 7. The discrimination means includes
The display control apparatus according to claim 1, wherein the operation mode is determined based on a moving speed of the indicator indicated by the proximity operation detected by the operation unit. The specifying means is:
The display mode according to the operation mode discriminated by the discriminating unit and the position of the indicator detected by the operation unit is specified using the control data stored in the storage unit. The display control apparatus according to any one of 1 to 9. The storage means
Storing the control data according to the application;
The specifying means is:
The display control apparatus according to claim 10, wherein a display mode is specified using the control data corresponding to an application corresponding to an image displayed on the display surface. Filter means for performing a filter process for reducing noise on the coordinate information representing the position detected by the operation means,
The discrimination means includes
The display control apparatus according to any one of claims 1 to 11, wherein an operation mode is determined based on coordinate information on which filter processing is performed by the filter means. The operation means includes
The display control apparatus according to claim 1, further comprising a first sensor that detects the contact operation and a second sensor that detects the proximity operation. The operation means includes
The display control apparatus according to claim 1, further comprising a single sensor that detects the contact operation and the proximity operation. A step of repeatedly detecting the position of the indicator with respect to the display surface and receiving an operation, the contact operation of bringing the indicator into contact with the display surface, and without bringing the indicator into contact with the display surface A first step of detecting a proximity operation to be brought close;
A second step of determining an operation mode related to the display mode of the display surface based on the transition of the proximity operation detected in the first step;
A third step of specifying a display mode of an image displayed on the display surface in accordance with the operation mode determined in the second step and the position of the indicator detected in the first step; ,
A display control method comprising: a fourth step of displaying an image on the display surface in the display mode specified in the third step.

Images