JP6794763B2 - Display system - Google Patents

Description

The present invention relates to a display system . More specifically, the present invention relates to a display system that can improve convenience.

Electrophotographic image forming devices include an MFP (Multi Function Peripheral) equipped with a scanner function, a facsimile function, a copying function, a printer function, a data communication function, and a server function, a facsimile machine, a copying machine, a printer, and the like. is there.

The MFP includes a storage device such as an HDD (Hard Disk Drive). The storage device includes a storage area (box area) that can be used individually by the user of the MFP. The user of the MFP can save the file saved in his / her mobile terminal or the like in the box area, or send the file saved in the box area to his / her mobile terminal or the like.

By the way, in recent years, a force sense presentation device has been proposed. The force sense presentation device is a device that gives the wearer a force sense, which is a reaction force sense when the object comes into contact with the object. It is expected that small force-sensitive devices that can be worn will become common in the future. The force sense presentation device is expected to be applied to various vocational skill training, remote control, health, medical and welfare equipment, and the like.

The technique relating to the force sense presentation device is disclosed in, for example, Patent Document 1 below. The following Patent Document 1 discloses an information processing device capable of appropriately operating a plurality of types of force-tactile presentation devices by using a single piece of information. The information processing device calculates the force to be presented to the force-tactile presentation device connected to the IF (interface) unit based on the feature amount of the image object. The information processing device corrects the calculated force to a force suitable for the force-tactile presenting device connected to the IF unit according to the type of the force-tactile presenting device. The information processing device outputs a signal indicating the corrected presentation force to the force-tactile presentation device via the IF unit.

Further, a technique related to object processing is disclosed in, for example, Patent Document 2 below. The information processing device of Patent Document 2 below includes an input operation receiving unit, a distance specifying unit, and a control unit. The input operation receiving unit receives the movement of an object substantially parallel to the display surface (two-dimensional plane) of the display unit on which the touch panels are stacked as a touch operation on the touch panel. The distance specifying unit detects the distance from the display surface (two-dimensional plane) of the display unit to the object when the touch operation is performed. The control unit executes processing related to the displayed object based on the type of touch operation received by the input operation reception unit (different type according to the movement trajectory of the object) and the distance detected by the distance identification unit. Is variably controlled.

Japanese Unexamined Patent Publication No. 2008-305109 Japanese Unexamined Patent Publication No. 2016-095885

In a display device equipped with a touch panel UI (User Interface), when moving the storage location of a file (for example, when moving a file from the display device to the MFP), the user can use the object displayed on the display of the display device (to the file). Select the related object) and perform a drag operation on that object. After receiving the operation from the user, the display device starts the process of moving the storage location of the file.

At first glance, the user cannot see the attributes (size, permissions, etc.) of the file associated with the object. For this reason, the user may find that the operation is not appropriate after performing a drag operation to move the storage location of the file. In other words, the user could not easily determine whether the operation was appropriate before the drag operation to move the file storage location was completed. As a result, the convenience of the display device has been reduced.

As an example, when the size of the file to be moved is large, it takes time to move the file, which causes a decrease in the response speed of the display device while moving the file. Also, if copying of the file is prohibited, the file cannot be moved even if the drag operation is performed. In these cases, the user is likely to determine that the operation was inappropriate after performing a drag operation to move the file storage location.

The problem of not being able to easily determine whether an operation related to a file is appropriate before the operation of processing the file is completed is a common problem when operating a file on a display device. Met.

The present invention is for solving the above problems, and an object of the present invention is to provide a display system capable of improving convenience.

A display system according to one aspect of the present invention is a display system including a display device and a force sensory device that is separate from the display device and is worn on the user's wrist, and the display device is associated with a file. A display means for displaying the displayed object, a selection reception means for accepting an operation for selecting an object displayed by the display means from the user, and a selection reception means associated with the selected object when the operation is received by the selection reception means. The force sense device includes an extraction means for extracting the attributes of the file and a notification means for notifying the force sense device of the content of perception determined according to the attributes extracted by the extraction means. The receiving means for receiving the notification of the above, the operation detecting means for detecting the operation on the selected object after receiving the content of perception by the receiving means, and the operation detecting means for detecting the operation on the selected object. In some cases, it includes a giving means for giving the user a sense of force, which is a perception according to the content of the perception received by the receiving means .
In the above display system, preferably, the display device uses a storage device that stores a table in which the relationship between the attributes of the file and the perception is described, and the content of the perception using the table based on the attributes extracted by the extraction means. Further including a deciding means for determining, the notifying means notifies the haptic device of the content of the perception determined by the deciding means.
In the above display system, preferably, the storage device stores a plurality of tables, the types of the attributes of the files described in each of the plurality of tables are different from each other, and the display device accepts the setting of the priority of the plurality of tables. Includes priority setting reception means.
In the above display system, preferably, the type of file attribute described in each of the plurality of tables is the file size, the number of files, the file permissions, or the file creator.
Preferably, in the display system, the display device further includes a table setting receiving means that accepts the setting of the relationship between the attribute of the file described in the table and the perception.
In the above display system, preferably, the force sense device further includes a relaxation means for relaxing the force sense given by the giving means when the elapsed time from receiving the operation by the selection receiving means exceeds the threshold value.
In the above display system, preferably, when the elapsed time from receiving the operation by the selection receiving means exceeds the threshold value, the relaxation means stops the giving of the force sense by the giving means.
In the above display system, preferably, the operation detecting means detects the operation of dragging the selected object, and the giving means detects the drag operation by the operation detecting means, and the content of the perception received by the receiving means. Start giving the sense of force, which is the perception according to.
In the above display system, preferably, the notification means notifies the haptic device of the content of perception in which at least one of the intensity, the direction, and the timing is different, which is determined according to the attribute extracted by the extraction means.

According to the present invention, it is possible to provide a display system capable of improving convenience.

It is a conceptual diagram which shows the conceptual structure of the display system in one Embodiment of this invention. It is a block diagram which shows the hardware structure of the display system in one Embodiment of this invention. FIG. 5 is a diagram schematically showing a basic screen displayed on the operation display unit 110 of the mobile terminal 100 in one embodiment of the present invention. It is a figure which shows typically the attribute table 152 stored in the mobile terminal 100 in one Embodiment of this invention. It is a figure which shows typically the perception conversion table 153a in one Embodiment of this invention. It is a figure which shows typically the perception conversion table 153b in one Embodiment of this invention. It is a figure which shows typically the perceptual conversion table 153c in one Embodiment of this invention. It is a figure which shows typically the priority table 154 in one Embodiment of this invention. It is the first half part of the sequence diagram which shows the operation of the display system in one Embodiment of this invention. It is the latter half part of the sequence diagram which shows the operation of the display system in one Embodiment of this invention. FIG. 5 is a diagram schematically showing a setting screen of a perception conversion table displayed on the operation display unit 110 of the mobile terminal 100 in step S7 of FIG. It is a figure which shows typically the state which the force sense device 200 applied the load corresponding to the force sense value to the user UR in step S32 of FIG. It is a flowchart which shows the operation of the mobile terminal 100 in one Embodiment of this invention. It is a figure which shows typically the perception conversion table 153 which a mobile terminal 100 stores in the 1st modification of one Embodiment of this invention. It is a figure which shows typically the operation of the mobile terminal 100 in the 2nd modification of one Embodiment of this invention. It is a figure which shows typically the visual conversion table 153 in the 2nd modification of one Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

In the following embodiment, a case where the display device is a mobile terminal will be described. The display device may be a mobile terminal, a mobile phone, a tablet PC (Personal Computer), an operation panel that accepts operations of a device such as an image forming device, or the like.

[Display system configuration]

First, the configuration of the display system according to the present embodiment will be described.

FIG. 1 is a conceptual diagram showing a conceptual configuration of a display system according to an embodiment of the present invention.

With reference to FIG. 1, the display system according to the present embodiment includes a mobile terminal 100 (an example of a display device) and a force sense device (force sense presentation device) 200. The mobile terminal 100 and the force sensor 200 communicate with each other by wireless communication such as short-range wireless communication.

The user UR is a user who can use the mobile terminal 100 and the force sensory device 200. While holding the mobile terminal 100 with one hand, the user UR operates the operation display unit 110 of the mobile terminal 100 with the other hand wearing the force sensor 200 on the wrist.

The mobile terminal 100 displays an object associated with the file on the operation display unit 110, and receives an operation of selecting the displayed object from the user UR. When the mobile terminal 100 accepts the operation of selecting an object, the mobile terminal 100 extracts the attributes of the file associated with the selected object. The mobile terminal 100 gives the user UR a perception according to the extracted attributes.

In the present embodiment, the mobile terminal 100 gives an instruction to the force sense device 200 worn by the user UR to give a force sense in which at least one of the intensity, the direction, and the timing is different depending on the extracted attributes. And send. The force sense device 200 gives the force sense to the user UR according to the instruction.

FIG. 2 is a block diagram showing a hardware configuration of a display system according to an embodiment of the present invention.

With reference to FIG. 2, the mobile terminal 100 is an operated terminal that receives an operation from the user UR, and is a CPU 101, a ROM 102, a RAM 103, a storage device 104, a wireless communication unit 105, and a short-range wireless communication unit. It includes 106, a coordinate management unit 107, an attribute determination unit 108, a perceptual conversion unit 109, and an operation display unit 110. In the CPU 101, each of the ROM 102, the RAM 103, the storage device 104, the wireless communication unit 105, the short-range wireless communication unit 106, the coordinate management unit 107, the attribute determination unit 108, the perceptual conversion unit 109, and the operation display unit 110 via a bus. Is connected.

The CPU 101 controls the entire mobile terminal 100 by executing the control program stored in the ROM 102.

The ROM 102 is, for example, a flash ROM. Various programs executed by the CPU 101 and various fixed data are stored in the ROM 102. The ROM 102 may be non-rewritable.

The RAM 103 is the main memory of the CPU 101. The RAM 103 is used for temporarily storing data and image data necessary for the CPU 101 to execute various programs.

The storage device 104 is composed of, for example, an HDD (Hard Disk Drive), and includes a program of a perception notification application 151, which is an application for cooperation with a force sense device 200, an attribute table 152, a perception conversion table 153, and a priority. Various information such as table 154 is stored.

The wireless communication unit 105 performs wireless communication with another device such as the force sensor 200 by a communication protocol such as TCP / IP according to an instruction from the CPU 101.

The short-range wireless communication unit 106 performs short-range wireless communication with other devices such as the force sensor 200 based on a short-range wireless communication standard such as Bluetooth (registered trademark).

The coordinate management unit 107 identifies an object existing at the coordinates of the position where the operation from the user UR is received by referring to the attribute table 152.

The attribute determination unit 108 extracts the attributes of the file associated with the object specified by the coordinate management unit 107 by referring to the attribute table 152.

By referring to the perception conversion table 153, the perception conversion unit 109 determines the perception given to the user UR, which is the perception according to the attribute determined by the attribute determination unit 108.

The operation display unit 110 includes a display unit including a touch panel display and the like, and an operation unit including software keys and the like displayed on the touch panel display. The operation display unit 110 receives various operations from the user and displays various information such as a screen including an object to the user.

The force sense device 200 gives a force sense to the wearing user UR according to an instruction from the mobile terminal 100. The force sense device 200 includes a CPU 201, a ROM 202, a RAM 203, a storage device 204, a wireless communication unit 205, a short-range wireless communication unit 206, a force sense presentation unit 207, an operation display unit 208, and a position acquisition unit. 209 and is included. ROM 202, RAM 203, storage device 204, wireless communication unit 205, short-range wireless communication unit 206, force sense presentation unit 207, operation display unit 208, and position acquisition unit 209 are each connected to the CPU 201 via a bus. There is.

The CPU 201 controls the operation of the entire force sensory device 200. The CPU 201 performs processing based on the control program.

The ROM 202 stores a control program or the like executed by the CPU 201.

The RAM 203 is a working memory of the CPU 201, and temporarily stores various information.

The storage device 204 stores various information such as a program of the force sense presentation application 251 which is an application for giving a force sense to the user UR wearing the force sense device 200.

The wireless communication unit 205 performs wireless communication with another device such as the mobile terminal 100 by a communication protocol such as TCP / IP according to an instruction from the CPU 201.

The short-range wireless communication unit 206 performs short-range wireless communication with other devices such as the mobile terminal 100 based on a short-range wireless communication standard such as Bluetooth (registered trademark).

The force sense presentation unit 207 gives (presents) a force sense to the user UR wearing the force sense device 200.

The operation display unit 208 receives various inputs from the user UR and displays various information to the user UR.

The position acquisition unit 209 includes an acceleration sensor, a geomagnetic sensor, a GPS (Global Positioning System) receiver, and the like. The position acquisition unit 209 acquires the position information of the force sense device 200. Further, the position acquisition unit 206 detects the direction and acceleration of movement of the own machine by the wearing user UR.

FIG. 3 is a diagram schematically showing a basic screen displayed on the operation display unit 110 of the mobile terminal 100 in one embodiment of the present invention.

With reference to FIG. 3, the basic screen includes a plurality of objects OB1, OB2, OB3, and OB4 which are items to be operated by the user. The object OB1 is an object of "file A1". Object OB2 is an object of "file A2". Object OB3 is an object of "folder B", and "file B1" and "file B2" are stored in the folder B. Object OB4 is an object of "App C". Objects OB1, OB2, and OB3 are objects associated with the file. The object OB4 is an object for starting an application, and is not an object related to a file.

FIG. 4 is a diagram schematically showing an attribute table 152 stored in the mobile terminal 100 in one embodiment of the present invention.

With reference to FIG. 4, the attribute table 152 describes the relationship between the coordinates (x, y) of the object included in the screen displayed on the operation display unit 110, the object name, and the attributes of the file related to the object. It is a table. The attribute table 152 is created in advance for each screen displayed on the operation display unit 110. The attribute table 152 shown in FIG. 4 is an attribute table 152 related to the basic screen of FIG. The notation of the coordinates of the object is arbitrary, but here, the upper left corner of the operation display unit 110 is the origin, the right direction in FIG. 3 of the operation display unit 110 is the x direction, and the lower direction in FIG. Is the y direction.

The attribute table 152 shown in FIG. 4 contains information about each of the objects OB1, OB2, OB3, and OB4. For example, according to the information at the top, the area surrounded by (10,30), (30,30), (10,70), and (30,70) has an object named file A1 (FIG. 3). It can be seen that the object OB1) exists, and the file associated with this object has a size of 6MB, is created by user UR1, and is allowed to be copied. Since the information in the third row from the top is the information of the object OB3 in which the two files are associated, the attributes of each file are described. Since the information at the bottom is information about the object OB4 that is not associated with the file, the file attribute column is blank.

5 to 7 are diagrams schematically showing a perceptual conversion table 153 according to an embodiment of the present invention.

With reference to FIGS. 5 to 7, the perception conversion table 153 is a table in which the relationship between the attributes of the file and the perception given to the user UR is described. File attributes include, for example, file size, number of files, file permissions, or file creator.

In this embodiment, the user UR is given a sense of force. In the following description, the specific content (direction, intensity, timing, etc.) of the force sense given to the user UR may be referred to as a force sense value.

The mobile terminal 100 stores a plurality of perceptual conversion tables 153a, 153b, and 153c as the perceptual conversion table 153. The types of file attributes listed in each of the plurality of perceptual transformation tables 153a, 153b, and 153c are different from each other. The mobile terminal 200 may store only one perceptual conversion table 153.

The perceptual conversion table 153a shown in FIG. 5 is a force sense conversion table in which the relationship between the file size and the force sense value is described. It is described in the perceptual conversion table 153a that when the file size is 10 MB or more, a strong force sense of 1 kg is given in the direction of θ = 0 degrees. Further, it is described that when the file size is 5 MB or more and less than 10 MB, a weak force sense of 0.5 kg is given in the direction of θ = 0 degrees. Furthermore, it is stated that when the file size is less than 5 MB, no force is given.

Here, θ is perpendicular to the display surface of the operation display unit 110, with θ = 0 degrees in the direction toward the mobile terminal 100 (terminal direction) along the direction perpendicular to the display surface of the operation display unit 110. The angle is such that θ = 180 degrees in the direction away from the mobile terminal 100 along the vertical direction. The reference direction of θ (the direction toward the mobile terminal 100 along the direction perpendicular to the display surface of the operation display unit 110) is specified by short-range wireless communication between the mobile terminal 100 and the haptic device 200. Will be done.

The perceptual conversion table 153b shown in FIG. 6 is a force-sense conversion table in which the relationship between the file authority and the force-sense value is described. It is described in the perceptual conversion table 153b that when copying of a file is prohibited, a strong force sensation of 1 kg is sequentially given in each direction of θ = 0 degree, 90 degree, and 180 degree for 3 seconds. ing. It also states that if copying of the file is permitted, it will not give a sense of force.

The perceptual conversion table 153b may further describe the relationship between the read / write authority of the file and the force value.

The perceptual conversion table 153c shown in FIG. 7 is a force conversion table in which the relationship between the number of files associated with the object and the force value is described. It is described in the perceptual transformation table 153c that when the number of files associated with the object is 4 or more, a strong force sense of 1 kg is given in the direction of θ = 0 degrees. Further, it is described that when the number of files associated with the object is 1 to 3, a strong force sense of 0.5 kg is given in the direction of θ = 180 degrees.

When the perception described in the perception conversion table 153 is a force sense value, the force sense value is different if at least one of the direction, the intensity, and the timing is different depending on the attribute of the file. Good. Further, the relationship between the perception and the attribute of the file described in the perception conversion table 153 can be set by the user UR.

FIG. 8 is a diagram schematically showing a priority table 154 according to an embodiment of the present invention.

With reference to FIG. 8, the priority table 154 is a table in which each of the perceptual transformation tables 153a, 153b, and 153c is described with respect to the set priority and effectiveness. According to the priority table 154 of FIG. 8, the perceptual conversion table 153a for the file size and the perceptual conversion table 153b for the file authority are enabled, while the perceptual conversion table 153c for the number of files is disabled. I understand. It can also be seen that the priority order of the perceptual conversion table 153b for file authority, the perceptual conversion table 153a for file size, and the perceptual conversion table 153c for the number of files is set (the priority of the invalid perceptual conversion table 153c is Automatically set to the lowest).

The priority and effectiveness of the perception conversion table in the priority table 154 can be set by the user UR, as will be described later.

[Display system configuration]

Subsequently, the operation of the display system in the present embodiment will be described.

9 and 10 are sequence diagrams showing the operation of the display system according to the embodiment of the present invention.

With reference to FIG. 9, the user UR activates a predetermined operation (for example, the perception notification application 151 and the force sense presentation application 251) for activating the force sense presentation function in each of the mobile terminal 100 and the force sense device 200. Operation) is performed (S1). When the mobile terminal 100 and the force sensor 200 receive an operation from the user UR, they perform pairing, which is a work of authenticating each other in order to maintain security. Specifically, the mobile terminal 100 transmits a signal for pairing to the surroundings (S3). Upon receiving the signal from the mobile terminal 100, the force sense device 200 transmits a response signal for pairing to the mobile terminal 100 (S5). As a result, short-range wireless communication between the mobile terminal 100 and the force sensor 200 is established.

When the short-range wireless communication with the force sense device 200 is established, the mobile terminal 100 perceives the user UR by displaying the setting screen (FIG. 11) of the perception conversion table on the operation display unit 110. Request the setting of the conversion table (S7). The user UR sets the perceptual conversion table through the setting screen of the perceptual conversion table (S9), and notifies the mobile terminal 100 of the completion of the setting of the perceptual conversion table (S11). When the mobile terminal 100 receives the notification of the completion of the setting from the user UR, the mobile terminal 100 saves the set content (S12).

Here, the processing of steps S7 to S12 will be described in detail.

FIG. 11 is a diagram schematically showing a setting screen of a perception conversion table displayed on the operation display unit 110 of the mobile terminal 100 in step S7 of FIG.

With reference to FIGS. 9 and 11, the setting screen of the perception conversion table displayed in step S7 includes a message MG requesting the setting of the perception conversion table, and keys KY1, KY2, KY3, KY4, KY5, KY6, KY7. , And KY8. Message MG is displayed at the top of the setting screen. The keys KY1, KY2, KY3, KY4, KY5, KY6, KY7, and KY8 are displayed on the lower right side of the setting screen. On the lower left side of the setting screen, each item of the perceptual conversion table called "file size", "file authority", and "number of files" is displayed. The keys KY1 and KY2 are keys corresponding to the perceptual conversion table 153a of the file size. The keys KY3 and KY4 are keys corresponding to the perceptual conversion table 153b of the file authority. The keys KY5 and KY6 are keys corresponding to the perceptual conversion table 153c of the number of files. The keys KY1, KY3, and KY5 are keys for setting the priority order of the perceptual transformation table. The keys KY2, KY4, and KY6 are keys for setting the validity of the table.

In step S9, the user UR sets the priority order of the perceptual conversion tables 153a, 153b, and 153c by pressing each of the keys KY1, KY3, and KY5 in an arbitrary order. The user UR also sets the validity of the perceptual conversion tables 153a, 153b, and 153c by pressing the "ON" or "OFF" portion of each of the keys KY2, KY4, and KY6.

When each of the keys KY1, KY3, and KY5 is pressed in an arbitrary order, the mobile terminal 100 assigns the priority order of "1", "2", and "3" to each perception conversion table in the pressed order. Set. Here, the priority order of the perceptual conversion table is set in the order of the perceptual conversion table 153b of the file authority, the perceptual conversion table 153a of the file size, and the perceptual conversion table 153c of the number of files.

When the "ON" portion of the keys KY2, KY4, or KY6 is pressed, the mobile terminal 100 activates its perceptual conversion table. When the "OFF" part of the keys KY2, KY4, or KY6 is pressed by the user, the mobile terminal 100 invalidates the perception conversion table and does not use it when determining the perception. Here, the perceptual conversion table 153a for file authority and the perceptual conversion table 153b for file size are enabled, and the perceptual conversion table 153c for the number of files is disabled.

In step S11, the user UR notifies the mobile terminal 100 of the completion of the setting of the perceptual conversion table by pressing the "OK" key KY7.

In step S12, the mobile terminal 100 determines the set contents when the key KY7 is pressed, and registers the set contents in the priority table 154.

On the other hand, when the "cancel" key KY8 is pressed, the mobile terminal 100 cancels the set contents.

With reference to FIG. 9, following the process of step S12, the mobile terminal 100 changes the operating state of its own device from the normal mode to the force sense notification mode that gives force sense according to the received operation (S13). .. Next, the mobile terminal 100 notifies the user UR by displaying the fact that the operating state of the own device has been changed to the force sense notification mode on the operation display unit 110 for a certain period of time (S15), and the basic screen including the object (FIG. 3). ) Is displayed on the operation display unit 110 (S16).

Following the process of step S16, the user UR selects an object in the basic screen displayed on the operation display unit 110 by a touch operation or the like. Here, it is assumed that the object OB1 is selected.

When the mobile terminal 100 receives an operation on the operation display unit 110, the mobile terminal 100 refers to the attribute table 152 (FIG. 4) to determine whether or not an object exists at the coordinates at which the operation is accepted, and the coordinates at which the operation is accepted. If an object exists in, the selection of the object is accepted (S19). This process is performed using the coordinate management unit 107. Here, the selection of the object OB1 is accepted.

Next, the mobile terminal 100 refers to the attribute table 152 (FIG. 4) and extracts the attributes of the file associated with the selected object (S21). This process is performed using the attribute determination unit 108. Here, the attributes (size: 6MB, creator: user UR1, copy: acceptable) of the file A1 associated with the object OB1 are extracted.

Subsequently, the mobile terminal 100 refers to the priority table 154 and acquires the perceptual conversion table 153 having the highest priority (S23). The mobile terminal 100 uses the acquired perceptual conversion table 153 to refer to the force sensory value corresponding to the extracted attribute.

When the intensity at the referenced force sense value is greater than 0.0 kg, the mobile terminal 100 determines the force sense value based on the referenced force sense value (S25), and transfers the determined force sense value to the force sense device 200. Notify (S27).

When the intensity at the referenced force sense value is 0, the mobile terminal 100 discards the referenced force sense value. Then, the mobile terminal 100 further acquires the perceptual conversion table 153 having only one lower priority, and uses the acquired perceptual conversion table 153 to refer to the force sensory value corresponding to the extracted attribute. When the referenced force sense value is 0, the mobile terminal 100 determines the force sense value to 0 for all the valid perceptual conversion tables 153 (S25), and sets the determined force sense value to the force sense device 200. (S27).

Here, the perceptual transformation table having the highest priority is the perceptual transformation table 153b (FIG. 6). However, the authority of the extracted file A1 is copyable, and the referenced force value is 0. Therefore, using the size of the extracted file A1 of 6 MB, the force sense of "direction θ = 0, intensity 0.5 kg" in the perceptual conversion table 153a (FIG. 5), which is the second highest priority perceptual conversion table. This force sense value is determined as a force sense value to be notified to the force sense device 200 by referring to the value.

With reference to FIG. 10, the force sense device 200 receives the force sense value from the mobile terminal 100 (S29).

The user UR initiates a desired operation (for example, a drag operation for moving the storage location of the file A1) on the selected object OB1 (S31).

When the force sense device 200 receives a desired operation by the user UR using the position acquisition unit 209, the force sense device 200 gives the force sense (load) corresponding to the force sense value to the user UR (S32).

In step S32, when the force sense device 200 receives the force sense value from the mobile terminal 100, the force sense device 200 may immediately give the user a force sense value according to the force sense value regardless of whether or not a desired operation is started. Good.

FIG. 12 is a diagram schematically showing a state in which the force sense device 200 applies a load corresponding to the force sense value to the user UR in step S32 of FIG.

With reference to FIG. 12, the force sense device 200 gives the user UR a force sense (load) corresponding to the force sense value of “direction θ = 0, intensity 0.5 kg” as shown by the arrow AR1. Since θ is the direction toward the mobile terminal 100 (terminal direction) along the direction perpendicular to the display surface of the operation display unit 110, the force sense received by the user UR is relative to the display surface of the operation display unit 110. The direction is vertically toward the mobile terminal 100.

With reference to FIG. 10, when the user UR receives a force sense from the force sense device 200 (S33), the attribute of the file is inferred from the force sense, and the desired operation is continued based on the attribute of the inferred file. Whether or not it is determined (S35). For example, if the user UR infers that the size of the file A1 is large, it is assumed that the user UR cancels the operation of moving the file A1 because it takes a long time to move the file A1. Further, if the user UR infers that copying of the file A1 is prohibited, it is assumed that the user UR cancels the operation of copying the file A1.

Next, the mobile terminal 100 determines whether or not the elapsed time since the object was selected exceeds the threshold value (for example, 3.0 seconds) (S37). When the elapsed time exceeds the threshold value, the mobile terminal 100 changes the determined force sense value to 0 (S39) and notifies the force sense device 200 that the force sense value is set to 0 (S41).

Upon receiving the notification, the force sense device 200 stops applying the force sense (S43).

FIG. 13 is a flowchart showing the operation of the mobile terminal 100 according to the embodiment of the present invention.

With reference to FIG. 13, the CPU 101 of the mobile terminal 100 determines whether or not a touch operation to the operation display unit 110 has been detected (S101). In step S101, the CPU 101 repeats the process of step S101 until it is determined that the touch operation to the operation display unit 110 has been detected.

If it is determined in step S101 that a touch operation to the operation display unit 110 has been detected (YES in S101), the CPU 101 acquires the touched coordinates from the operation display unit 110 (S103). Next, the CPU 101 refers to the attribute table 152 and identifies the object if it exists at the acquired coordinates (S105). Subsequently, the CPU 101 determines whether or not the object has been specified (S107).

If it is determined in step S107 that the object has been identified (YES in S107), the CPU 101 determines whether or not a file is associated with the identified object (S109).

If it is determined in step S109 that the file is associated with the specified object (YES in S109), the CPU 101 refers to the perceptual conversion table 153 and determines the force value based on the attributes of the file (S111). , The determined force sense value is notified to the force sense device 200 (S113). Subsequently, the CPU 101 acquires the elapsed time since the touch operation is detected (S115), and determines whether or not the elapsed time exceeds the threshold value (S117). The CPU 101 repeats the process of step S117 until it is determined that the elapsed time exceeds the threshold value.

When it is determined in step S117 that the elapsed time exceeds the threshold value (YES in S117), the CPU 101 notifies the force sense device 200 that the force sense value is set to 0 (S119), and ends the process.

If it is determined in step S107 that the object is not specified (NO in S107), or if it is determined in step S109 that the file is not associated with the specified object (NO in S109), the CPU 101 ends the process.

According to the present embodiment, the user who selects the object is perceived according to the attributes of the file associated with the selected object, so that the attributes of the file can be intuitively known. As a result, it is possible to easily determine whether or not the operation related to the processing is appropriate before the operation for processing the selected file is completed, and the convenience can be improved.

[First modification]

In the above-described embodiment, the operation of the mobile terminal 100 that stops the application of the force sense to the user UR when the elapsed time from receiving the operation of selecting the object exceeds the threshold value has been described. Instead of this operation, the mobile terminal 100 may relax the intensity of the force sense given to the user UR to a value other than 0 when the elapsed time from receiving the operation of selecting an object exceeds the threshold value. .. There may be a plurality of threshold values.

FIG. 14 is a diagram schematically showing a perception conversion table 153 stored in the mobile terminal 100 in the first modification of the embodiment of the present invention.

With reference to FIG. 14, the perceptual conversion table 153 of the present modification shows the force sense value A when the elapsed time from receiving the operation of selecting an object is equal to or less than the threshold value (for example, 5.0 seconds). The force sensory value B when the elapsed time from receiving the operation of selecting an object exceeds the threshold value is set. The content of the force sensory value A is the same as the content of the perceptual conversion table 153a of FIG. The intensity of the force sense value of each item of the force sense value B is relaxed more than the intensity of the force sense value of each item of the force sense value A.

In the force sense value B, specifically, the intensity when the file size is 10 MB or more is relaxed from 1 kg to 0.5 kg, and the intensity when the file size is 5 MB or more and less than 10 MB is from 0.5 kg. It has been relaxed to 0.0 kg. The strength when the file size is less than 5 MB remains 0.0 kg.

In this modified example, when the elapsed time from receiving the operation of selecting an object exceeds another threshold value (for example, 10.0 seconds) longer than the above threshold value, the application of force sense is stopped. You may. Further, the intensity of the force sense value may be gradually relaxed with the elapsed time from receiving the operation of selecting the object.

According to this variant, the force sensation is relaxed as time passes after the object is selected, so the attributes of the file associated with the object while avoiding giving unnecessary force sensation to the user UR. Can be continuously informed.

[Second variant]

In the above-described embodiment, the case where the user UR is given a force sense according to the attribute of the file is shown. The perception given to the user UR may be visual, olfactory, auditory, etc., in addition to force. In this modification, a case where the user UR is given a visual sense (visual effect) according to the attributes of the file will be described. That is, when the mobile terminal 100 in this modification accepts the operation of selecting an object, it extracts the attributes of the file associated with the selected object and imparts the user UR a visual according to the extracted attributes. ..

The operation of this modification is the operation of the mobile terminal 100 alone, and the force sense device 200 is unnecessary in this modification.

FIG. 15 is a diagram schematically showing the operation of the mobile terminal 100 in the second modification of the embodiment of the present invention. Note that FIG. 15 shows only the object OB2 on the basic screen for convenience of explanation. The dotted line in FIG. 15 indicates the initial position of the object OB2.

With reference to FIG. 15, the basic screen is displayed on the operation display unit 110 of the mobile terminal 100. On the basic screen, the object OB2 exists at the initial position. The user UR selects the object OB2 (FIG. 15 (a)) and performs a drag operation on the operation display unit 110 to move the object OB2 in the downward direction in FIG. 15 (FIG. 15 (b)).

When the mobile terminal 100 accepts the drag operation, the mobile terminal 100 moves the object in the direction of the accepted drag operation on the operation display unit 110. The mobile terminal 100 changes the followability of the object OB2 to the drag operation according to the attributes of the file.

The mobile terminal 100 refers to the perceptual conversion table and determines the delay time according to the attributes of the file. Then, the object OB2 is moved on the operation display unit 110 so that the object OB2 follows the operation finger that performs the drag operation with a delay time determined. Specifically, the object OB2 stays in the initial position until the determined delay time elapses (FIG. 15 (b)). The object OB2 gradually moves in the direction of the operation finger on the operation display unit 110 after the determined delay time has elapsed (FIG. 15 (c)). The object OB2 finally moves to the position of the operating finger (FIG. 15 (d)).

FIG. 16 is a diagram schematically showing a visual conversion table 153 in a second modification of the embodiment of the present invention.

The perceptual conversion table 153 shown in FIG. 16 is a table in which the relationship between the file size and the delay time is described. In this perceptual conversion table 153, the larger the file size, the longer the delay time is set. Specifically, the perceptual conversion table 153 describes that the delay time is set to 2 seconds when the file size is 10 MB or more. Further, it is described that the delay time is set to 1 second when the file size is 5 MB or more and less than 10 MB. Further, it is described that the delay time is set to 0 when the file size is less than 5 MB.

According to this modification, the attributes of the file associated with the object can be intuitively transmitted to the user UR without using the force sense device 200.

Since the configurations and operations of the display system other than the above in the first and second modifications are the same as the configurations and operations of the above-described embodiment, their description will not be repeated.

[Other]

The above-described embodiments and modifications can be combined as appropriate. For example, by combining the first modification and the second modification, the visual, olfactory, or auditory vision given to the user UR may be gradually relaxed.

The processing in the above-described embodiment may be performed by software or by using a hardware circuit. It is also possible to provide a program that executes the processing according to the above-described embodiment, and record the program on a recording medium such as a CD-ROM, a flexible disk, a hard disk, a ROM, a RAM, or a memory card and provide the program to the user. You may decide to do it. The program is executed by a computer such as a CPU. Further, the program may be downloaded to the device via a communication line such as the Internet.

It should be considered that the above embodiments are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

100 Mobile Terminal 101,201 CPU (Central Processing Unit)
102,202 ROM (Read Only Memory)
103,203 RAM (Random Access Memory)
104,204 Storage device 105,205 Wireless communication unit 106,206 Short-range wireless communication unit 107 Coordinate management unit 108 Attribute judgment unit 109 Perception conversion unit 110 Operation display unit 151 Perception notification application 152 Attribute table 153, 153a, 153b, 153c Perception Conversion table 154 Priority table 200 Power sense device 206 Position acquisition unit 207 Power sense presentation unit 208 Operation display unit 209 Position acquisition unit 251 Power sense presentation application KY1, KY2, KY3, KY4, KY5, KY6, KY7, KY8 key MG OB, OB1, OB2, OB3, OB4 object UR user

Claims (9)

A display system including a display device and a force sensor device that is separate from the display device and is worn on the user's wrist.
The display device is
A display means to display the objects associated with the file, and
A selection receiving means that accepts an operation of selecting an object displayed by the display means from the user,
An extraction means that extracts the attributes of the file associated with the selected object when the operation is accepted by the selection reception means, and
Including a notification means for notifying the force sense device of the content of perception determined according to the attribute extracted by the extraction means.
The force sense device is
A receiving means for receiving a notification of the content of the perception from the display device, and
An operation detecting means for detecting an operation on a selected object after receiving the content of the perception by the receiving means, and an operation detecting means.
When the operation detecting means detects an operation on the selected object , the display includes a giving means for imparting a force sense, which is a perception corresponding to the content of the perception received by the receiving means, to the user. system. The display device is
A storage device that stores a table that describes the relationship between file attributes and perception,
Further including a determination means for determining the content of the perception using the table based on the attributes extracted by the extraction means.
The display system according to claim 1, wherein the notification means notifies the force sensor device of the content of the perception determined by the determination means. The storage device stores a plurality of the tables, and the types of file attributes described in each of the plurality of tables are different from each other.
The display system according to claim 2, wherein the display device further includes a priority setting receiving means for receiving the setting of the priority of the plurality of tables. The display system according to claim 3, wherein the type of file attribute described in each of the plurality of tables is the size of the file, the number of files, the authority of the file, or the creator of the file. The display system according to any one of claims 2 to 4, wherein the display device further includes a table setting receiving means for receiving a setting of a relationship between the attribute of a file described in the table and perception. The force sense device further includes a relaxation means for alleviating the force sense given by the giving means when the elapsed time from receiving the operation by the selection receiving means exceeds the threshold value. The display system according to any one of 5. The display system according to claim 6, wherein when the elapsed time from receiving the operation by the selection receiving means exceeds the threshold value, the relaxation means stops the giving of the force sense by the giving means. The operation detection means detects an operation of dragging the selected object and detects the operation .
The giving means, when the drag operation is detected by the operation detecting means , starts giving a force sense which is a perception according to the content of the perception received by the receiving means , claim 1 to 7. The display system described in either. The notification unit was determined in accordance with the attribute extracted by the extraction means, the intensity, direction, and at least one of contents of said different perceptual out timing notifying the haptic device, according to claim 8 The display system described in any of.

Images