JP5988141B2 - Portable terminal set, portable terminal jacket, portable terminal, information processing system, and program - Google Patents

Description

  The present invention relates to a portable terminal set, a portable terminal jacket, a portable terminal, an information processing system, and a program, and in particular, realizes a new user interface that makes it possible to recognize the softness of an object by using an illusionary haptic phenomenon. The present invention relates to a mobile terminal set, a mobile terminal jacket, a mobile terminal, an information processing system, and a program.

  As a user interface of a computer or the like, one using tactile sensation is known. As a typical user interface using a tactile sense, there is a touch input using a touch panel capable of inputting information by touching a display surface of a display.

  In addition to touch input on a screen, touch input on the back of a mobile terminal has been researched and developed (for example, see Non-Patent Documents 1 and 2). Also, it has been proposed to use the back surface of the portable terminal as an output interface (see, for example, Non-Patent Documents 3 and 4).

  Further, it has been proposed to detect a user's grip strength and operate an object on the screen according to the detected grip strength. For example, an input interface has been proposed in which a gripping force of a user holding the device is detected with a pressure sensor provided in a pointing device, and the thickness of a line drawn on the display is changed according to the detected gripping force (for example, Patent Document 1).

  In a freely deformable computer-like device, a sensor array that detects compression and strain on the device surface is provided, and when an operation to compress the device is detected, the document displayed on the device screen is converted into an icon. An input interface that performs an operation of (miniaturizing) has also been proposed (see, for example, Patent Document 2).

  Furthermore, a system that expresses feelings using grip strength and communicates the feelings has also been proposed (see, for example, Patent Document 3). Specifically, in the system, the transmission side measures the magnitude of the user's grip strength using a dedicated input device, and determines the type of feeling the user wants to input based on the measured value. Transmit feelings to the other party. The receiving side outputs a facial image with a facial expression corresponding to the type of sent feeling.

  By the way, there is known a phenomenon called visual inductive illusion that causes an illusion of operation feeling based on tactile sensation by controlling visual feedback with respect to an input operation by tactile sensation. The input interfaces described in Documents 1 to 3 are not input interfaces using the illusionary haptic phenomenon.

  As an input interface using an illusionary tactile phenomenon, for example, there are techniques described in Non-Patent Document 5 and Patent Document 4.

  Non-Patent Document 5 proposes an application in which a piston on a display screen is operated using a piston-shaped input device and an object ahead of the piston is deformed on the screen. In this application, the larger the deformation of the object on the screen by the piston operation, the more the user feels that the object is soft, and the application controls the visual feedback of the deformation of the object on the screen. Based on the illusion of softness sensation.

  Further, in Patent Document 4, a force sensor is arranged on the operation surface of the touch panel so that the finger pressing force can be detected, and the scroll image of the background is scrolled in accordance with the finger movement and the pressing force on the operation surface. There has been proposed a user interface that makes a finger feel lighter or heavier by changing the speed. As a result, it is possible to feel as if an actual phenomenon has occurred based on visual information.

Japanese Patent Laid-Open No. 1-095319 Japanese Patent Laid-Open No. 11-143606 JP 2000-0666807 A International Publication No. 2008/111245

ITmedia + D Mobile, CEATEC JAPAN 2010: Touch the back of the device to enter text-KDDI's "Back Input Interface", Internet <URL: http://plusd.itmedia.co.jp/mobile/articles/1010/07 /news081.html> Sony Corporation, PlayStation (R) Vita PlayStation (R) official site, Internet <URL: http: //www.jp.playstation.com/psvita/special/> Enomoto Laboratory Home Page, Research Contents “A Touch Screen in the Palm”, S. Fukushima, H. Kajimoto: Palm Touch Panel: Providing Touch Sensation Through the Device, In ACM International Conference on Interactive Tabletops and Surfaces, ITS '11, Page 79-82, Kobe, Japan,, November 13-16, 2011., Internet <URL: http: //kaji-lab.jp/en/index.php? Research> Masahiro Furukawa, News 2. KUSUGURI was appeared in SIGGRAPH ASIA 2011. KUSUGURI: Visual Tactile Integration for Tickling [Web] [Project Page] (Dec. 4, 2011), Masahiro Furukawa, Hiroyuki Kajimoto, Susumu Tachi, KUSUGURI: a shared tactile interface for bidirectional tickling, Augmented Human 2012, March 9-10, 2012, France Megeve, Internet <URL: http://masahirofurukawa.jp/> Anatole Lecuyer, Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback, Presence, Vol.18, No.1, pp.39-53, 2009

  However, Non-Patent Document 5 describes that, in the state where the hand operating the tactile input device and the screen on which visual feedback is performed are separated from each other, the induction of the visually inductive illusion of softness is promoted. ing. In other words, in the technique described in Non-Patent Document 5, when the tactile input and the visual feedback are performed in the same housing, this illusion is not promoted. In other words, the occurrence of the illusionary tactile sensation phenomenon has not been confirmed under the condition that visual feedback is received in the state where the hand performing the tactile input is visible.

  The technique described in Patent Document 4 causes an illusionary tactile sensation phenomenon using a pressing force on the screen of the touch panel, but does not support tactile input other than pressing on the screen of the touch panel.

  The present invention has been made in view of such circumstances, and makes it possible to realize a new user interface that makes it possible to recognize the softness of an object by using an illusionary tactile sensation phenomenon. is there.

  By using a portable terminal in which both tactile input based on grip strength and presentation of visual feedback are mounted in the same housing, the illusion of tactile sensation of grip strength can be seen in a state where the hand where the tactile input is performed can be seen. It was found to be expressed. The present invention utilizes this newly discovered illusionary haptic phenomenon.

The mobile terminal set according to the first aspect of the present invention is:
A mobile terminal set comprising a mobile terminal that can be held by a user and a jacket that is attached to the mobile terminal,
The jacket is
A pressure sensor that detects a pressure when the user grips the portable terminal and outputs a pressure value as a result thereof;
The portable terminal is
Based on the softness information indicating the degree of softness and the pressure value indicated by the pressure sensor, the morphological attribute of the illusionary tactile image that allows the user to recognize the softness is determined and display control of the illusionary tactile image is performed. An image display control unit to perform;
A softness information control unit that changes the softness information and sets a plurality of softness information in the image display control unit;
A display unit that displays the illusionary tactile image controlled by the image display control unit.

The mobile terminal according to the second aspect of the present invention is:
A portable terminal that can be held by a user,
A pressure sensor for detecting pressure when the user grips the portable terminal;
Based on the softness information indicating the degree of softness and the pressure value indicated by the pressure sensor, the morphological attribute of the illusionary tactile image that allows the user to recognize the softness is determined and display control of the illusionary tactile image is performed. An image display control unit to perform;
A softness information control unit that changes the softness information and sets a plurality of softness information in the image display control unit;
A display unit that displays the illusionary tactile image controlled by the image display control unit.

The program according to the third aspect of the present invention is:
On your mobile device,
Change the softness information representing the degree of softness, set multiple softness information,
According to the set softness information and the pressure value indicated by the pressure sensor when the user grips the portable terminal, the morphological attribute of the illusionary tactile image that makes the user recognize the softness is determined and the complex This is to execute processing to control the display of tactile images.

An information processing system according to a fourth aspect of the present invention includes:
An information processing system comprising a mobile terminal that can be held by a user and a server,
The portable terminal is
Form of an illusionary tactile image that allows the user to recognize softness according to a pressure value indicated by a pressure sensor that detects pressure when the user grips the mobile terminal and softness information indicating the degree of softness An image display control unit for controlling the display of the illusionary tactile image by determining a visual attribute;
A softness information control unit that changes the softness information and sets a plurality of softness information in the image display control unit;
A display unit that displays the illusionary tactile image that is display-controlled by the image display control unit,
The server
A storage unit for storing a plurality of softness information;
A transmission unit that transmits the softness information to the mobile terminal.

  In the first to fourth aspects of the present invention, in the mobile terminal, softness information indicating the degree of softness is changed, a plurality of the softness information is set, and the set softness information, Depending on the pressure value indicated by the pressure sensor when the user grips the portable terminal, the morphological attribute of the illusionary tactile image that allows the user to recognize the softness is determined, and display control of the illusionary tactile image is performed.

  According to the first to fourth aspects of the present invention, it is possible to realize a new user interface that makes it possible to recognize the softness of an object using an illusionary tactile sensation phenomenon.

It is a block diagram which shows the functional structural example implement | achieved with the portable terminal to which this invention is applied. It is a figure explaining the user interface which a portable terminal implement | achieves. It is a flowchart explaining the visual feedback process by a portable terminal. It is a figure which shows the structural example in case a portable terminal does not have a pressure sensor. It is a block diagram which shows the structural example in case a portable terminal does not have a pressure sensor. It is a block diagram which shows the hardware structural example of a portable terminal. It is a figure which shows the other example of compression expression. It is a figure which shows the structural example of the information processing system to which the visual feedback process is applied. It is a figure which shows the example of a goods database. It is a flowchart explaining the goods confirmation process of an information processing system. It is a figure which shows the example of a goods list screen. It is a figure which shows the example of a softness confirmation screen. It is a figure which shows the example of a softness condition selection screen.

<Embodiment to which the present invention is applied>
[Configuration example of mobile terminal]
FIG. 1 is a block diagram showing an example of a functional configuration realized by a mobile terminal to which the present invention is applied.

  The mobile terminal 1 of FIG. 1 includes a pressure sensor 11, a visual feedback determination unit 12, a softness constant control unit 13, and a display unit 14. The mobile terminal 1 detects the pressure applied by the user grasping the mobile terminal 1, and changes the image displayed on the display unit 14 according to the detected pressure value. That is, the mobile terminal 1 has a user interface that allows the user to recognize the softness by giving feedback to the user's vision according to the grip strength of the user.

[Description of user interface]
With reference to FIG. 2, the user interface which the portable terminal 1 implement | achieves is demonstrated in detail.

  FIG. 2A is a diagram illustrating a state where the user is holding the mobile terminal 1.

  As shown in FIG. 2A, the user is holding the mobile terminal 1 so as to hold the side surface of the mobile terminal 1. A pressure sensor 11 (not shown) is embedded in the side surface of the casing of the mobile terminal 1. An illusionary tactile sensation image Q for giving visual feedback to the user is displayed on the display unit 14 of the mobile terminal 1. In the present embodiment, the illusionary tactile sensation image Q is a rectangular (quadrangle) image.

  As shown in FIG. 2B, when the user increases the force to grip the side surface of the mobile terminal 1, the pressure change is detected by the pressure sensor 11 and applied to the mobile terminal 1 (the pressure sensor 11 on the side surface). The illusionary tactile sensation image Q displayed on the display unit 14 changes according to the pressure. Specifically, the illusionary tactile sensation image Q is compressed and displayed such that the longer the pressure applied to the mobile terminal 1 is, the shorter the short side (lateral side) of the rectangular illusionary tactile sensation image Q is.

  Returning to FIG. 1, each configuration of the mobile terminal 1 will be described.

  The pressure sensor 11 is provided on the side surface of the casing of the mobile terminal 1, detects (detects) the pressure applied by the user (hand), and supplies the pressure value obtained as a result to the visual feedback determination unit 12. .

The visual feedback determination unit 12 determines the morphological attributes of the rectangular illusionary tactile image Q according to the pressure value supplied from the pressure sensor 11 and the softness constant supplied from the softness constant control unit 13. Specifically, the visual feedback determination unit 12 determines the length of the short side of the rectangular illusionary tactile sensation image Q according to Expression (1) using the pressure value and the softness constant.
X = X ₀ −PressureValue × C (1)
However, when X ₀ <PressureValue × C, X = 0 (2)

Here, X ₀ and X correspond to the length from the reference to the end of the short side with reference to the center line of the short side of the rectangular illusionary tactile image Q, and the illusionary tactile sensation displayed on the display unit 14. The value is half of the short side of the image Q. X ₀ represents an initial value, and X represents a value after control in which deformation control is performed according to the pressure value. PressureValue is a pressure value supplied from the pressure sensor 11, and when the pressure sensor 11 is composed of a plurality of pressure sensors, a statistical value such as the total, average, median, etc. of all or a plurality of pressure sensors, Alternatively, it is calculated using a part of pressure values selected from a plurality of pressure sensors. C is a softness constant as softness information representing the degree of softness.

Equation (1) is equivalent to a modification of the general formula F = K × (X ₀ -X) of the spring model expressed by force F (= PressureValue) and spring coefficient K, and the softness constant C is the spring It corresponds to the reciprocal of the coefficient K (C = 1 / K).

According to Expression (1), as described with reference to FIG. 2, the shorter the side of the illusionary tactile sensation image Q displayed on the display unit 14 is, the greater the force with which the user grips the mobile terminal 1 (grip strength). Becomes shorter. X / X ₀ corresponds to the compression rate of the short side of the illusionary tactile image Q, and the display control of the illusionary tactile image Q is performed such that the compression rate of the illusionary tactile image Q increases as the pressure value PressureValue increases.

  The visual feedback determination unit 12 determines the short side length X of the rectangular illusionary tactile image Q as described above, and causes the display unit 14 to display the illusionary tactile image Q having the short side length X.

  The softness constant control unit 13 receives an operation by the user, a change in the softness constant C from an external program, and the like, and supplies the received softness constant C to the visual feedback determination unit 12.

  The display unit 14 displays the illusionary tactile image Q in accordance with the display control of the visual feedback determination unit 12.

[Processing flow of visual feedback processing]
With reference to the flowchart of FIG. 3, the visual feedback process by the portable terminal 1 is demonstrated.

  First, in step S <b> 1, the softness constant control unit 13 receives a predetermined softness constant C from a user operation or an external program, and supplies the received softness constant C to the visual feedback determination unit 12.

  In step S <b> 2, the pressure sensor 11 detects the pressure applied by the user (hand) and supplies the detected pressure value PressureValue to the visual feedback determination unit 12.

  In step S <b> 3, the visual feedback determination unit 12 uses the pressure value PressureValue supplied from the pressure sensor 11 and the softness constant C supplied from the softness constant control unit 13 to calculate a rectangular shape according to Equation (1). The length X of the short side of the illusionary tactile image Q is determined.

  In step S4, the visual feedback determination unit 12 causes the display unit 14 to display the illusionary tactile image Q having the determined short side length X. The display unit 14 displays an illusionary tactile sensation image Q having a short side length X in accordance with the display control of the visual feedback determination unit 12.

  In step S5, the softness constant controller 13 determines whether or not the softness constant C has been changed.

  If it is determined in step S5 that the softness constant C has been changed, the process returns to step S1, and the processes in steps S1 to S5 described above are repeated. Thereby, display control of the illusionary tactile sensation image Q having the short side length X corresponding to the detected pressure value PressureValue is executed using the softness constant C after the change.

  On the other hand, if it is determined in step S5 that the softness constant C has not been changed, the process proceeds to step S6, and the visual feedback determination unit 12 determines whether or not to end the process.

  If it is determined in step S6 that the process is not terminated, the process returns to step S2, and the processes in steps S2 to S6 described above are repeated. Therefore, when it is determined not to end the process, the processes of steps S2 to S6 described above are executed at predetermined time intervals.

  On the other hand, if it is determined in step S6 that the process is to be terminated, the visual feedback process in FIG. 3 is terminated.

  As described above, according to the visual feedback process, the mobile terminal 1 detects the pressure applied to the mobile terminal 1 by the user and displays the illusionary tactile sensation displayed on the display unit 14 according to the detected pressure value PressureValue. Change the image Q. At this time, due to the difference in the value of the softness constant C, the user feels that the illusionary tactile sensation about the pressing force (grip strength) is visually as if the softness of the illusionary tactile sensation image Q displayed on the display unit 14 has changed. Caused by feedback. For example, when the user repeatedly grips the mobile terminal 1 with a certain strength and then repeats the operation of releasing the force, if the softness constant C is decreased, the degree of compression of the illusionary tactile image Q when the user grips is reduced. Therefore, the user has an illusion that the illusionary tactile image Q has become relatively hard. Conversely, when the value of the softness constant C is increased, the degree of compression of the illusionary tactile sensation image Q when the user grips it increases, so that the user has the illusion that the illusionary tactile sensation image Q has become relatively soft. In this way, by changing the value of the softness constant C, a new user interface that makes it possible to recognize the softness of a predetermined target object can be realized.

  The visual feedback process (visual feedback control) of the mobile terminal 1 uses the illusion of gripping force and visual feedback to the last, so a configuration for performing direct tactile feedback on the user's hand is not necessary, It can be downsized. Therefore, it can be integrally mounted on a small device such as a portable terminal.

[Configuration example when the mobile terminal does not have a pressure sensor]
In the example described above, the mobile terminal 1 is described as including the pressure sensor 11, but there may be a case where the mobile terminal 1 does not include the pressure sensor 11. In such a case, a configuration as shown in FIG. 4 can be adopted.

  That is, as shown in FIG. 4, a jacket (cover case) 22 that can be attached to a portable terminal 21 that does not include the pressure sensor 11 is prepared. In the jacket 22, the pressure sensor 11 is embedded in a portion that is a side surface of the mobile terminal 21 in a state of being attached to the mobile terminal 21.

  FIG. 5 shows a block diagram of the configuration shown in FIG. 5, parts corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted.

  The portable terminal 21 includes a visual feedback determination unit 12, a softness constant control unit 13, and a display unit 14. The jacket 22 includes the pressure sensor 11.

  The pressure sensor 11 of the jacket 22 and the visual feedback determination unit 12 of the portable terminal 21 are connected by, for example, USB (Universal Serial Bus), and the pressure value PressureValue detected by the pressure sensor 11 is serially communicated to determine the visual feedback determination unit. 12 is supplied. Data transmission / reception between the pressure sensor 11 and the visual feedback determination unit 12 can be performed by other communication methods, for example, wireless communication such as Bluetooth (registered trademark) or NFC (Near Field Communication), pressure value PressureValue data, and the microphone terminal. It is also possible to implement a method of inputting as an audio signal via a signal, a method of inputting as bit information using light blinking via an imaging unit, or the like.

[Hardware Configuration Block Diagram of Portable Terminal 21]
FIG. 6 is a block diagram illustrating a hardware configuration example of the mobile terminal 21.

  In FIG. 6, a CPU (Central Processing Unit) 31 is an arithmetic processing unit that executes various processes, such as a basic program such as an OS (Operating System), the visual feedback determination unit 12 and the softness constant control unit 13 described above. Execute the program etc. corresponding to. The CPU 31 is connected to a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, and an NVRAM (Non Volatile RAM) 34 via a bus 35. An input / output interface 36 is also connected to the bus 35, and a storage unit 37 and a drive 38 are connected to the input / output interface 36.

  The ROM 32 stores basic programs such as an OS and data in advance. Application programs and data stored in the ROM 32 and the storage unit 37 are loaded into the RAM 33 and the NVRAM 34. The RAM 33 and the NVRAM 34 also appropriately store data necessary for the CPU 31 to execute various processes.

  The drive 38 reads data from a removable medium 39 such as a semiconductor memory, which is mounted as necessary, and supplies the data to the CPU 31 via the input / output interface 36 or the like, and supplies data supplied from the CPU 31 to the removable medium 39. Write. Data to be read and written may include data of an application program to be installed.

  An input unit 40, a voice input / output unit 41, a touch screen 42, a communication unit 43, a telephone line network communication unit 44, and a photographing unit 45 are connected to the input / output interface 36.

  The input unit 40 includes hardware buttons such as a menu button, a home button, a back button, and a power button. The input unit 40 receives input from a user operation and supplies it to the CPU 31 and the like. The voice input / output unit 41 includes a speaker, a microphone, and the like, and inputs and outputs voice for a call.

  The touch screen 42 includes a display unit 42A and a position detection unit 42B, and can perform various screen displays and operation input by bringing a finger or the like in close proximity or in contact (touch).

  The display unit 42A is composed of a device (display device) such as a liquid crystal panel, and displays a predetermined screen (image) according to data supplied via the input / output interface 36.

  The position detection unit 42B is configured by a device having a function of receiving (detecting) an external input (proximity / contact) such as an electrostatic touch panel.

  In the touch screen 42, the display unit 42A and the position detection unit 42B as described above are integrated, and the screen is displayed on the display unit 42A, and the screen displayed on the display unit 42A is displayed on the position detection unit 42B. External operation inputs (touch and proximity) can be accepted.

  The position signal output by the position detection unit 42B is supplied to the CPU 31 via the input / output interface 36. The CPU 31 recognizes what operation input has been made based on the position signal from the position detection unit 42B. Various processes are performed according to the operation input. The entire touch screen 42 or the display unit 42A corresponds to the display unit 14 shown in FIGS.

  The communication unit 43 performs wired communication such as USB communication and wireless communication such as WiFi. The telephone line network communication unit 44 performs voice communication or packet communication with other devices via the mobile telephone communication network. The imaging unit 45 includes an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Mental Oxide Semiconductor) sensor. The photographing unit 45 photographs a subject and supplies image data of the photographed subject to the CPU 31 or the like via the input / output interface 36.

  The portable terminal 21 is configured as described above. The mobile terminal 21 includes an NFC communication unit that performs non-contact communication using electromagnetic induction at a close distance of about several centimeters, a wireless communication unit that uses a Bluetooth (registered trademark) system, and infrared communication that performs infrared communication. You may make it further provide other structures, such as a part. The hardware configuration of the portable terminal 1 is obtained by further adding a pressure sensor 11 to the configuration of FIG.

[Modification]
In the example described above, the visual feedback determination unit 12 determines the length X of the short side of the illusionary tactile sensation image Q according to Equation (1). It is also possible to determine the length X of the short side.

式（３）のM,B,K,X1は、所定の実数でありKは、式（１）の柔らかさ定数Cを決定していたバネ係数である。（dX／dt）^２は、錯触覚画像Qの短辺の長さXの２階微分、（dX／dt）は、錯触覚画像Qの短辺の長さXの１階微分を表す。dtは、圧力センサ１１からの圧力値PressureValueに応じた視覚フィードバック制御を行う制御サイクルに相当する微小単位時間を表す。 For example, the visual feedback determination unit 12 solves the pressure value PressureValue supplied from the pressure sensor 11 by substituting the pressure value PressureValue into the second-order differential equation of the following equation (3), thereby performing visual feedback control according to the pressure value. The length X of the short side of the illusionary tactile image Q can also be obtained.

M, B, K, and X1 in Expression (3) are predetermined real numbers, and K is a spring coefficient that has determined the softness constant C in Expression (1). (DX / dt) ² represents the second derivative of the short side length X of the illusionary tactile image Q, and (dX / dt) represents the first derivative of the short side length X of the illusionary tactile image Q. dt represents a minute unit time corresponding to a control cycle in which visual feedback control according to the pressure value PressureValue from the pressure sensor 11 is performed.

  In this way, by allowing a plurality of methods to be selected for controlling the length X of the short side of the rectangular illusionary tactile sensation image Q, various soft expressions can be expressed.

Expressions (1) and (3) are expressions for obtaining the length X of the short side of the illusionary tactile image Q corresponding to the pressure value PressureValue when the user grasps. Therefore, with the formulas (1) and (3) alone, the illusionary tactile image Q displayed on the display unit 14 as soon as the user loosens the grasped hand has an initial size (the length of the short side). X ₀ ).

  In contrast, when the user releases the gripped hand, the lateral width of the illusionary tactile image Q vibrates in a manner like a beyond beyon, depending on the strength (pressure value PressureValue) when gripped. When the video expression is changed and gradually attenuates, the user can more visually recognize the softness of the object. It is the same as recognizing the softness of the jelly with the image of the jelly swaying after the jelly is struck and released.

式（４）において、長さXは式（１）または式（３）で求めた既知の値であるので、（dX／dt）^２および（dX／dt）については計算することができる。そして、M,B,K,X2は、式（３）と同様、所定の実数であるので、力Fは求めることができる。 When performing such a video representation of the damped vibration, first, the visual feedback determination unit 12 calculates the length X of the short side of the illusionary tactile sensation image Q obtained by the formula (1) or the formula (3) by the formula Substituting into (4), the force F at the time of length X is obtained.

In the formula (4), since the length X is a known value obtained by the formula (1) or the formula (3), (dX / dt) ² and (dX / dt) can be calculated. Since M, B, K, and X2 are predetermined real numbers as in Equation (3), the force F can be obtained.

以上のようにして得られたdt時間経過後の長さXを、再び式（４）乃至式（６）と順に代入することで、さらに次のdt時間経過後の長さXを求めることができる。したがって、式（４）乃至式（６）の演算を繰り返すことで求めた長さXとなるように、表示部１４の錯触覚画像Qを表示制御することで、減衰振動の映像表現が可能となる。 Next, the visual feedback determination unit 12 obtains the moving speed (dX / dt) after elapse of dt time when the force F is applied, by the equation (5), and further, the moving speed (dX / dt after elapse of dt time). ), The length (position) X after elapse of dt time is obtained by the equation (6).

The length X after the elapse of the next dt time can be obtained by substituting the length X after the elapse of the dt time as described above in order with the equations (4) to (6) again. it can. Therefore, by controlling the display of the illusionary tactile sensation image Q on the display unit 14 so as to have the length X obtained by repeating the calculations of Expressions (4) to (6), it is possible to express the image of the damped vibration. Become.

  In the above-described embodiment, an example has been described in which the shape when the illusionary tactile sensation image Q is compressed is uniform at any position in the vertical direction. However, as shown in FIG. 7A, for example, the compression rate may be different depending on the position in the vertical direction so that the middle of the long side of the illusionary tactile sensation image Q is particularly compressed. In addition, as shown in FIG. 7B, the compression shape may be changed so that not only the horizontal direction of the illusionary tactile sensation image Q is compressed, but also the compressed area escapes in the vertical direction. Further, both the horizontal direction and the vertical direction may be compressed (reduced). Furthermore, the illusionary tactile sensation image Q may be a polygonal image other than a rectangle or a circle, or may be a three-dimensional image such as a sphere or a rectangular parallelepiped.

  In the above-described embodiment, the pressure sensor 11 provided on the long side of the rectangular mobile terminal 1 detects the pressure with which the user holds the mobile terminal 1, and accordingly, the illusionary tactile image Q Is compressed and displayed in the pressure direction, but the pressure sensor 11 is arranged at a position where the portable terminal 1 is gripped by the user, and the illusionary tactile sensation image Q is in the direction of the pressure detected by the pressure sensor 11. The shape of the mobile terminal 1, the position of the pressure sensor 11, and the shape of the illusionary tactile sensation image Q are not limited to the above-described forms as long as they are deformed accordingly. For example, the mobile terminal 1 may be circular, the pressure sensor 11 may be disposed along the entire circumference, and the illusionary tactile image Q may also be circular.

It is said that the illusionary tactile sensation is most likely to occur when the parameter representing the softness of the target object changes. Therefore, for example, (specifically, the pressure value PressureValue to be detected to a predetermined value PV ₁ below) when the user is holding lightly portable terminal 1 and softness constant C _1, Once grasped strongly (specifically Specifically, when the detected pressure value PressureValue becomes larger than the value PV ₁ ), the user holds the mobile terminal 1, such as changing to the softness constant C ₂ and calculating Equation (1). The softness constant C may be changed on the way. Thereby, it becomes possible to generate an illusion more powerfully.

<Example of application to which the present invention is applied>
Next, an application example to which the visual feedback process described above is applied will be described.

[Configuration example of information processing system]
FIG. 8 shows a configuration example of an information processing system in which the above-described visual feedback processing is applied to online shopping using a plurality of types of pillows as merchandise.

  The information processing system in FIG. 8 is configured to include a mobile terminal 1 having the configuration shown in FIG. 1 and a server (application server) 51 that holds data of multiple types of pillows as products. The server 51 is connected via a network such as a mobile telephone communication network, the Internet, a LAN (Local Area Network), and a WAN (Wide Area Network).

  The mobile terminal 1 is, for example, a high-function mobile terminal called a smartphone, and an application program (hereinafter simply referred to as an application) may be the same as a server (server 51) connected via a network. ) Can be downloaded from the storage device into the terminal and executed on the terminal.

  The mobile terminal 1 connects to the server 51 and displays product information based on the product data acquired from the server 51 by executing a shopping app, which is an application for online shopping, downloaded in the terminal. . The shopping application has the visual feedback determination unit 12 and the softness constant control unit 13 shown in FIG. 1 as functions.

  The storage unit 52 of the server 51 stores a product database that stores data of a plurality of types of pillows as products. The server 51 stores predetermined data in response to a request from the shopping application of the mobile terminal 1. It transmits to the shopping application of the portable terminal 1. The server 51 also performs purchase procedure processing when the user performs a purchase operation of a predetermined pillow from the shopping application. In the present specification, description of processing not related to the present invention such as purchase procedure processing is omitted.

  Although illustration of the hardware configuration of the server 51 is omitted, the server 51 includes a CPU, a ROM, a RAM, and the like in addition to a storage unit 52 such as a hard disk. The server 51 also includes a communication unit that performs data communication by connecting to a LAN or the Internet, an input unit including a keyboard and a mouse, and an output unit including a speaker and a display. The server 51 can execute various processes including processes to be described later according to a program loaded in the RAM from a program stored in the storage unit 52 or the ROM.

[Product database example]
FIG. 9 shows an example of a product database stored in the storage unit 52 of the server 51.

  For each of multiple types of pillows as a product, the product ID that identifies the product, the product name that is the name of the product, the price that is the sales price of the product, the image file of the product image, and the softness set for the product pillow A constant C is stored in association with each product. The product database may include information items other than these.

In the example of FIG. 9, as the first product (pillow), product ID = “00001”, product name = “special buckwheat pillow”, price = “¥ 3500”, image file = “IMG00001.jpg”, soft Stores constant = “C ₁ ”,. In addition, as the second product (pillow), product ID = “00002”, product name = “hotel specification feather pillow”, price = “¥ 8500”, image file = “IMG00002.jpg”, softness constant = “C ₂ ", ... are stored. As the third product (pillow), product ID = “00003”, product name = “pipe pillow”, price = “\ 1800”, image file = “IMG00003.jpg”, softness constant = “C ₃ ”, -Is stored.

The product database also stores data of samples (hereinafter also referred to as reference samples) that serve as a basis for softness when the user confirms the softness of each product. In other words, the reference sample is for confirming how the degree of softness known by the user is displayed in the compressed expression of the illusionary tactile image Q of the shopping app. Accordingly, it is sufficient that at least one reference sample is registered, but in this embodiment, as shown in FIG. 9, the “reference / buckwheat pillow” which is a product having the minimum softness constant C _min is used. And “standard / feather pillow”, which is a product having the maximum softness constant C _max , is registered.

  The softness constant C of each product is determined in advance based on a test by a monitor user. Specifically, having multiple monitor users touch the actual product to recognize the softness, the multiple monitor users have a softness constant C that provides a soft feeling comparable to the softness of the actual product. The values are set, and their statistical representative values (eg, average value, median, etc.) are determined as the softness constant C of the product. If necessary, a reference sample may be prepared, and the actual softness of the reference sample may be compared with the softness constant set for the reference sample.

  Note that the softness constant of the reference sample is sufficient if the user can fully recognize the softness of the reference sample without experiencing the softness of the reference sample in the compressed expression of the illusionary tactile image Q of the shopping app. If it is sufficient to compare the softness, it can be omitted.

[Product Check Process Flowchart]
With reference to the flowchart (sequence diagram) of FIG. 10, the product confirmation process for confirming the softness of the product will be described.

  First, in step S <b> 21, the mobile terminal 1 activates the shopping application in response to the activation operation of the user who activates the shopping application.

  In step S22, the shopping application of the mobile terminal 1 accesses the server 51 and transmits a request for product data. The access destination URL of the server 51 is stored in the shopping application.

  In step S23, the server 51 acquires the product data from the product database in the storage unit 52, and in step S24, transmits the acquired product data to the shopping application of the mobile terminal 1. Here, the product data transmitted to the mobile terminal 1 is the product ID, product name, price, and image file of each product, and the softness constant is not transmitted at this time.

  In step S <b> 25, the shopping application of the mobile terminal 1 receives the product data transmitted from the server 51 and displays a product list screen on the display unit 14 based on the received product data.

  FIG. 11 shows an example of a product list screen displayed on the display unit 14 of the mobile terminal 1 by the process of step S25.

  On the product list screen, the product ID, product name, price, and image (image corresponding to the image file) are displayed for each product based on the product data acquired from the server 51. Each product has a “softness confirmation” button B that is operated when the user confirms the softness of the product, together with a product ID, a product name, a price, and an image. In the example of FIG. 11, the product ID = “00001” has a “softness check” button B1, the product ID = “00002” has a “softness check” button B2, and the product ID = “00003”. , “Check softness” button B3 is displayed.

  When the user operates the “softness check” button B of the desired product by touching with a finger or the like, the shopping app of the mobile terminal 1 sets the “softness check” button B of the desired product as step S26. In response to the operation, in step S27, a request for the product (hereinafter referred to as “designated product”) softness constant C designated by the user is transmitted to the server 51.

  In step S28, the server 51 receives a request for the softness constant C of the designated product from the shopping application, and obtains the softness constant C of the designated product and the softness constant C of the reference sample from the product database in the storage unit 52. get.

  In step S <b> 29, the server 51 transmits the acquired softness constant C of the designated product and the softness constant C of the reference sample to the mobile terminal 1.

  In step S <b> 30, the shopping application of the mobile terminal 1 receives the softness constant C transmitted from the server 51 and displays a softness confirmation screen using the received softness constant C on the display unit 14.

  FIG. 12 shows an example of the softness confirmation screen of the designated product displayed on the display unit 14 of the mobile terminal 1 by the process of step S30. The softness confirmation screen of FIG. 12 shows the softness confirmation screen when the “softness confirmation” button B2 of the product with the product ID = “00002” is pressed on the product list screen of FIG.

  The softness confirmation screen is operated when returning to the reference sample or the combo box CB for selecting the specified product, the illusionary tactile image Q having the softness constant C selected in the combo box CB, and the product list screen. A “Back” button BK is provided.

  In step S <b> 31, the shopping application of the mobile terminal 1 executes visual feedback processing based on the pressure value PressureValue detected by the pressure sensor 11 of the mobile terminal 1. In other words, in step S31, the visual feedback process described with reference to FIG. 3 is executed.

  The softness confirmation operation on the softness confirmation screen will be described with reference to FIG.

  First, the user operates the combo box CB and selects an item to be checked for softness by selecting either the reference sample or the designated product item. When either the reference sample or the designated product is selected in the combo box CB, the softness constant control unit 13 of the shopping app visually displays the softness constant C of the reference sample or the designated product selected in the combo box CB. This is supplied to the feedback determination unit 12.

  Then, as described with reference to FIG. 2, when the user grips the side surface of the mobile terminal 1, the pressure sensor 11 detects the pressure applied to the mobile terminal 1, and according to the detected pressure value PressureValue, The length X of the short side of the illusionary tactile sensation image Q on the softness confirmation screen in FIG. 12 changes.

  Returning to FIG. 10, in step S <b> 32, the shopping application of the mobile terminal 1 determines whether an operation for returning to the product list has been performed, that is, whether the “return” button BK on the softness confirmation screen of FIG. 12 has been pressed. .

  If it is determined in step S32 that the operation for returning to the product list has not been performed, the process returns to step S31, and the visual feedback process using the softness confirmation screen of FIG. 12 is repeated.

  On the other hand, if it is determined in step S32 that the operation for returning to the product list has been performed, the process returns to step S25, and the processes of steps S25 to S32 described above are repeated.

  As described above, the product confirmation process for confirming the softness of the product in the shopping application is executed.

  In the information processing system to which the visual feedback processing is applied, the reference database and the softness constant C of each product are stored in the product database of the server 51, and the softness constant C of the designated product specified by the user is stored in the reference sample. Along with the softness constant C, it is transmitted to the shopping application of the mobile terminal 1.

  The softness constant C of the specified product and the reference sample is switched according to the operation of the combo box CB on the softness confirmation screen. Under the set softness constant C, the softness depends on the grip strength of the user. The illusionary tactile image Q on the confirmation screen is deformed and displayed. As a result, the user can experience the softness of the product he or she desires by comparing the softness of the reference sample with the illusion of touching force, thus assisting the user in understanding the softness of the product. It becomes possible to do. Accordingly, it is possible to select a product while imagining the softness in Internet shopping where the actual product cannot be touched.

  In the above-described example, the user designates a desired product from the product list screen, and the user can confirm the softness of the designated designated product using the softness confirmation screen. However, the user can directly specify desired softness, and the shopping application can search the product database for a product corresponding to the specified softness and present it to the user.

  FIG. 13 shows an example of a softness condition selection screen displayed on the display unit 14 of the mobile terminal 1 when a desired softness is designated.

  On the softness condition selection screen, an illusionary tactile sensation image Q that is deformed and displayed according to the user's grip strength, a slide bar SB that instructs to change the softness constant C, and a softness constant C that corresponds to the current position of the slide bar SB are displayed. A decision button DB to be set is provided.

  In the slide bar SB, the position of the soft constant C of the reference sample on the slide bar is displayed as “reference 1” and “reference 2” so that the comparison with the softness of the reference sample is easy. Yes.

  The user operates the direction button of the mobile terminal 1, the + button or the − button, or the slide bar SB on the softness condition selection screen to set the softness constant C to a desired value and holds the mobile terminal 1. The operation of confirming the deformation state of the illusionary tactile image Q is repeated until a desired softness constant C is found. When the desired softness constant C is found, the determination button DB on the softness condition selection screen is operated (pressed). The shopping application transmits the softness constant C when the determination button DB is operated to the server 51, and the server 51 searches the product database for the product closest to the transmitted softness constant C. Then, the shopping application receives the softness constant C of the product corresponding to the softness specified by the user, transmitted from the server 51, and causes the display unit 14 to display the softness confirmation screen of FIG.

  Thus, if the softness condition selection screen is used, the softness desired by the user can be directly designated.

  In addition, this softness condition selection screen, for example, when exchanging a softness request between the product orderer and the manufacturer, the words “softer” or “softer than product A” Rather than telling, you can set the desired softness on the softness condition selection screen and transmit the set softness constant C, so that the softness can be expressed numerically and requested. Can transmit the softness more accurately.

  Furthermore, in the example described above, the softness constant C of the product database has been described as being determined by a test by the monitor user, but the softness felt after purchase by the user who purchased the product using the softness condition selection screen. It is also possible to receive feedback of the softness constant C from the user who purchased the product.

When accepting feedback of the softness constant C, the shopping application transmits the softness constant C determined using the softness condition selection screen to the server 51 in association with the product ID of the purchased product. Server 51, the softness constants set by the user _{C? Feedback,} softness constant before feedback that is stored in product database and C _before, the following equation to determine the softness constant C _after post-feedback .
C _after = (C _before × N + C _feedback ) / (N + 1)
Here, N represents the number of samples used to determine the softness constant C _{before before} feedback stored in the product database, and is stored in the product database in association with the product ID.

  In the example described above, the shopping application executed on the mobile terminal 1 that is a smartphone displays a product list screen and a softness confirmation screen based on the product data acquired from the server 51, and the user holds the mobile terminal 1. The example of compressing and displaying the illusionary tactile sensation image Q based on the pressure value PressureValue at the time has been described, but the product list screen and the softness confirmation screen described above are displayed on the Web browser activated on the mobile terminal 1. It may be. In this case, the pressure value PressureValue indicating the user's grip strength detected by the pressure sensor 11 is transmitted to the server 51 as HTML data, and the web application of the server 51 is based on the designated product, and the illusionary tactile sensation image Q after the deformation control is short. The side length X is determined, and HTML data for displaying the determined illusionary tactile image Q after compression control is transmitted to the web browser of the mobile terminal 1. That is, in this case, the visual feedback determination unit 12 and the softness constant control unit 13 are provided in the server 51 as part of the Web application. Thus, the functions implemented in each of the mobile terminal 1 and the server 51 can be appropriately shared based on the performance requirements of each device.

<Examples of other applications>
Other application examples to which visual feedback processing is applied will be described.

[Example applied to cooking recipes]
The visual feedback process described above can be applied to a cooking recipe.

  For example, when the degree of softness of udon and bread dough is provided as a recipe, a “softness check” button B for checking the softness of the dough is provided on the recipe screen along with ingredients and cooking procedures.

  On the softness confirmation screen displayed when the “softness confirmation” button B is operated, for example, food with typical softness such as konjac or jelly is presented as a reference sample. Using such a softness confirmation screen, the softness of the udon noodles and bread dough can be experienced more accurately with the compression of the illusionary tactile image Q.

  In addition, the softness of the fabric changes with the kneading time. Therefore, the softness constant control unit 13 may change the softness constant C according to the display time of the softness confirmation screen by regarding the display time of the softness confirmation screen as the kneading time. Thereby, the user can experience the softness after 5 minutes on the softness confirmation screen, for example. Such control processing can be realized by defining the softness constant C stored in the product database as a function depending on the display time of the softness confirmation screen.

  The softness of the dough also varies depending on the combination of the amount of water, the amount of powder, and the kneading time. Therefore, for example, in a cooking recipe, on the softness confirmation screen, the amount of water, the amount of powder, and the kneading time slider bar (similar to the slider bar SB in FIG. 13) are provided, and the amount of water, The user may be able to confirm the softness of the dough when the amount of powder and the kneading time are a combination of the values set with the slider bar. Such a control process is provided with a plurality of pretreatment coefficients (corresponding to the amount of water, the amount of powder, and the kneading time) as parameters for determining the softness constant C, and an equation using a plurality of pretreatment coefficients. This can be achieved by defining a softness constant C.

  Further, in the cooking recipe, a screen similar to the softness condition selection screen shown in FIG. 13 is provided so that the user can register (store) favorite softness when cooking this time. Good. At this time, for example, when the determination button DB for determining the softness in FIG. 13 is pressed, a comment or the like can be input, and the softness constant C and information such as the comment can be registered in association with each other. As a result, for example, since it was too soft when cooked before, it is possible to experience it in comparison with past softness, such as making it a little bit this time.

  The objects for comparing and experiencing the softness are not limited to the same type of object, but may be different types. For example, if the softness constant C is determined and presented so that the relative relationship does not contradict the softness (stickiness) of jelly, konnyaku, and koji, it will not actually eat koji, referring to different types of softness. Even softness can be discriminated.

[Example applied to a game]
The visual feedback process described above can be applied to a game (game program).

  For example, in a role-playing game where an item is acquired, a softness constant C is set for each item type, and a character operated by the user touches the item to obtain an object with a predetermined hardness or softness. It can be applied to such processing.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  In the above-described embodiment, the mobile terminal 1 has been described as a smartphone. However, the mobile terminal 1 is not limited to a smartphone, and may be a tablet terminal such as a PND (Portable Navigation Device) or an electronic book reader.

  In this specification, the steps described in the flowchart are not necessarily processed in chronological order, but are performed in parallel or when they are called in chronological order according to the described order. It may be executed at a necessary timing.

  In this specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Accordingly, a plurality of devices housed in separate housings and connected via a network and a single device housing a plurality of modules in one housing are all systems. .

  DESCRIPTION OF SYMBOLS 1 Mobile terminal, 11 Pressure sensor, 12 Visual feedback determination part, 13 Softness constant control part, 14 Display part, 21 Mobile terminal, 22 Jacket, 31 CPU, 33 RAM, 37 Storage part, 42 Touch screen, 43 Communication part, 51 servers, 52 storage units

Claims (10)

A mobile terminal set comprising a mobile terminal that can be held by a user and a jacket that is attached to the mobile terminal,
The jacket is
A pressure sensor that detects a pressure when the user grips the portable terminal and outputs a pressure value as a result thereof;
The portable terminal is
Based on the softness information indicating the degree of softness and the pressure value indicated by the pressure sensor, the morphological attribute of the illusionary tactile image that allows the user to recognize the softness is determined and display control of the illusionary tactile image is performed. An image display control unit to perform;
A softness information control unit that changes the softness information and sets a plurality of softness information in the image display control unit;
A portable terminal set comprising: a display unit that displays the illusionary tactile image whose display is controlled by the image display control unit.   The jacket for portable terminals used with the portable terminal set of Claim 1. A portable terminal that can be held by a user,
A pressure sensor for detecting pressure when the user grips the portable terminal;
Based on the softness information indicating the degree of softness and the pressure value indicated by the pressure sensor, the morphological attribute of the illusionary tactile image that allows the user to recognize the softness is determined and display control of the illusionary tactile image is performed. An image display control unit to perform;
A softness information control unit that changes the softness information and sets a plurality of softness information in the image display control unit;
A portable terminal comprising: a display unit that displays the illusionary tactile image whose display is controlled by the image display control unit. The morphological attribute of the illusionary tactile image is a compression ratio of the illusionary tactile image,
The portable terminal according to claim 3, wherein the image display control unit determines the compression ratio of the illusionary tactile image so that the compression ratio of the illusionary tactile image increases as the pressure value increases. The mobile terminal according to claim 4, wherein the image display control unit performs display control to attenuate and vibrate the illusionary tactile image after displaying the illusionary tactile image at the compression rate according to the pressure value. The mobile terminal according to claim 3, wherein the softness information control unit changes the softness information according to a display time of the illusionary tactile image on the display unit. The mobile terminal according to claim 3, wherein the softness information is determined based on a plurality of parameters. The mobile terminal according to claim 3, further comprising a registration unit that registers the desired softness information. On your mobile device,
Change the softness information representing the degree of softness, set multiple softness information,
According to the set softness information and the pressure value indicated by the pressure sensor when the user grips the portable terminal, the morphological attribute of the illusionary tactile image that makes the user recognize the softness is determined and the complex A program that executes processing to control display of tactile images. An information processing system comprising a mobile terminal that can be held by a user and a server,
The portable terminal is
Form of an illusionary tactile image that allows the user to recognize softness according to a pressure value indicated by a pressure sensor that detects pressure when the user grips the mobile terminal and softness information indicating the degree of softness An image display control unit for controlling the display of the illusionary tactile image by determining a visual attribute;
A softness information control unit that changes the softness information and sets a plurality of softness information in the image display control unit;
A display unit that displays the illusionary tactile image that is display-controlled by the image display control unit,
The server
A storage unit for storing a plurality of softness information;
An information processing system comprising: a transmission unit that transmits the softness information to the mobile terminal.

Images