US20180088698A1 - Electronic device and drive controlling method - Google Patents

Electronic device and drive controlling method Download PDF

Info

Publication number
US20180088698A1
US20180088698A1 US15/828,056 US201715828056A US2018088698A1 US 20180088698 A1 US20180088698 A1 US 20180088698A1 US 201715828056 A US201715828056 A US 201715828056A US 2018088698 A1 US2018088698 A1 US 2018088698A1
Authority
US
United States
Prior art keywords
amplitude
image
photographic subject
region
electronic device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/828,056
Other languages
English (en)
Inventor
Tatsuya Suzuki
Yuichi KAMATA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujitsu Ltd
Original Assignee
Fujitsu Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujitsu Ltd filed Critical Fujitsu Ltd
Assigned to FUJITSU LIMITED reassignment FUJITSU LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SUZUKI, TATSUYA, KAMATA, Yuichi
Publication of US20180088698A1 publication Critical patent/US20180088698A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/043—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means using propagating acoustic waves
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
    • G06F3/023—Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
    • G06F3/0233—Character input methods
    • G06F3/0236—Character input methods using selection techniques to select from displayed items
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/0412—Digitisers structurally integrated in a display
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/0416—Control or interface arrangements specially adapted for digitisers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04847—Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/014—Force feedback applied to GUI
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/044—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by capacitive means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/045—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact

Definitions

  • the disclosures herein relate to an electronic device and a drive controlling method.
  • Non-Patent Document Kim, Seung-Chan, Ali KAr, and Ivan Poupyrev. “Tactile rendering of 3D features on touch surfaces.” Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 2013.
  • an electronic device includes an imaging part configured to acquire an image and a range image in a field of view that includes a photographic subject; a range image extracting part configured to extract a range image of the photographic subject based on the image and the range image; a gloss determining part configured to determine whether the photographic subject is a glossy object based on a data lacking portion included in the range image of the photographic subject; a display part configured to display the image; a top panel disposed on a display surface side of the display part and having a manipulation surface; a position detector configured to detect a position of a manipulation input performed on the manipulation surface; a vibrating element configured to be driven by a driving signal for generating a natural vibration in an ultrasound frequency band on the manipulation surface so as to generate the natural vibration in the ultrasound frequency band on the manipulation surface; an amplitude data allocating part configured to allocate, as amplitude data of the driving signal, first amplitude to a display region of the photographic subject that has been determined to be the glossy object by the gloss determining part, and
  • FIG. 1 is a perspective view illustrating an electronic device of a first embodiment
  • FIG. 2 is a plan view illustrating the electronic device of the first embodiment
  • FIG. 3 is a cross-sectional view of the electronic device taken along line A-A of FIG. 2 ;
  • FIG. 4 is a bottom view illustrating the electronic device of the first embodiment.
  • FIGS. 5A and 5B are drawings illustrating crests of a standing wave formed in parallel with a short side of a top panel, of standing waves generated on the top panel by a natural vibration in an ultrasound frequency band;
  • FIGS. 6A and 6B are drawings illustrating cases in which a kinetic friction force applied to a user's fingertip performing a manipulation input changes by the natural vibration in the ultrasound frequency band generated on the top panel of the electronic device;
  • FIG. 7 is a drawing illustrating a configuration of the electronic device according to the first embodiment
  • FIG. 8 is a drawing illustrating an example of use of the electronic device
  • FIG. 9 is a drawing illustrating a range image acquired by an infrared camera
  • FIG. 10 is a drawing illustrating a range image acquired by an infrared camera
  • FIG. 11 is a drawing illustrating a range image including noise
  • FIG. 12 is a flowchart illustrating processing for allocating amplitude data executed by the electronic device of the first embodiment
  • FIG. 14 is a flowchart illustrating in detail a part of the flow illustrated in FIG. 12 ;
  • FIGS. 15A through 15D are drawings illustrating image processing that is performed according to the flow illustrated in FIG. 14 ;
  • FIG. 16 is a flowchart illustrating processing for acquiring a ratio of noise
  • FIG. 17 is a drawing illustrating the amplitude data allocated by an amplitude data allocating part to a specific region
  • FIG. 18 is a drawing illustrating amplitude data for a glossy object and amplitude data for a non-glossy object stored in a memory
  • FIG. 19 is a drawing illustrating data stored in the memory
  • FIG. 20 is a flowchart illustrating processing executed by a drive controlling part of the electronic device of the embodiment
  • FIG. 21 is a drawing illustrating an example of an operation of the electronic device of the first embodiment
  • FIG. 22 is a drawing illustrating a use scene of the electronic device
  • FIG. 23 is a flowchart illustrating processing for allocating amplitude data executed by an electronic device of a second embodiment
  • FIG. 24 is a drawing illustrating a probability distribution of a ratio of noise
  • FIG. 25 is a drawing illustrating a method for determining a threshold by using a mode method
  • FIG. 26 is a flowchart illustrating a method for acquiring an image of a specific region according to a third embodiment
  • FIGS. 27A through 27D are drawings illustrating image processing performed according to the flow illustrated in FIG. 26 ;
  • FIG. 28 is a side view illustrating an electronic device of a fourth embodiment.
  • FIG. 1 is a perspective view illustrating an electronic device 100 of a first embodiment.
  • a manipulation input part 101 of the electronic device 100 includes a display panel disposed under a touch panel.
  • Various buttons 102 A or sliders 102 B in a graphic user interface (GUI) are displayed on the display panel.
  • GUI graphic user interface
  • the user of the electronic device 100 touches the manipulation input part 101 with the fingertip in order to manipulate GUI manipulation parts 102 .
  • FIG. 2 is a plan view illustrating the electronic device 100 of the first embodiment.
  • FIG. 3 is a cross-sectional view of the electronic device 100 taken along line A-A of FIG. 2 .
  • FIG. 4 is a bottom view illustrating the electronic device 100 of the first embodiment. Further, as illustrated in FIGS. 2 through 4 , a XYZ coordinate system, which is a rectangular coordinate system, is defined.
  • the electronic device 100 includes a housing 110 , a top panel 120 , a double-sided adhesive tape 130 , a vibrating element 140 , a touch panel 150 , a display panel 160 , and a substrate 170 .
  • the electronic device 100 includes a camera 180 , an infrared camera 190 , and an infrared light source 191 .
  • the camera 180 , the infrared camera 190 , and the infrared light source 191 are provided on the bottom of the electronic device 100 (see FIG. 4 ).
  • the housing 110 is made of a plastic, for example. As illustrated in FIG. 3 , the substrate 170 , the display panel 160 , and the touch panel 150 are provided in a recessed portion 110 A, and the top panel 120 is bonded to the housing 110 with the double-sided adhesive tape 130 .
  • the camera 180 , the infrared camera 190 , and the infrared light source 191 are provided on the bottom of the electronic device 100 (see FIG. 4 ).
  • the top panel 120 is a thin, flat member having a rectangular shape when seen in a plan view and made of transparent glass or reinforced plastics such as polycarbonate.
  • a surface 120 A (on a positive side in the z-axis direction) of the top panel 120 is an exemplary manipulation surface on which a manipulation input is performed by the user of the electronic device 100 .
  • the vibrating element 140 is bonded to a surface on a negative side in the z-axis direction of the top panel 120 .
  • the four sides of the top panel 120 when seen in a plan view are bonded to the housing 110 with the double-sided adhesive tape 130 .
  • the double-sided adhesive tape 130 may be any double-sided tape that can bond the four sides of the top panel 120 to the housing 110 and is not necessarily formed in a rectangular ring shape as illustrated in FIG. 3 .
  • the touch panel 150 is disposed on the negative side in the z-axis direction of the top panel 120 .
  • the top panel 120 is provided to protect the surface of the touch panel 150 . Also, an additional panel, a protective film, and the like may be separately provided on the surface of the top panel 120 .
  • the top panel 120 vibrates when the vibrating element 140 is driven.
  • a standing wave is generated on the top panel 120 by vibrating the top panel 120 at the natural vibration frequency.
  • the vibrating element 140 is bonded to the surface on the negative side in the z-axis direction of the top panel 120 , along the short side extending in an x-axis direction, at the positive side in the y-axis direction.
  • the vibrating element 140 may be any element as long as it can generate vibrations in an ultrasound frequency band.
  • the vibrating element 140 may use any element including piezoelectric elements such as a piezoelectric device.
  • the vibrating element 140 is driven by a driving signal output from the drive controlling part described later.
  • the amplitude (intensity) and frequency of a vibration generated by the vibrating element 140 are set by the driving signal.
  • an on/off action of the vibrating element 140 is controlled by the driving signal.
  • the ultrasound frequency band is referred to as a frequency band of approximately 20 kHz or more.
  • a frequency at which the vibrating element 140 vibrates is equal to the natural frequency of the top panel 120 . Therefore, the vibrating element 140 is driven by the driving signal so as to vibrate at the natural vibration frequency of the top panel 120 .
  • the touch panel 150 is disposed on (the positive side in the z-axis direction of) the display panel 160 and under (the negative side in the z-axis direction of) the top panel 120 .
  • the touch panel 150 is illustrated as an example of a position detector that detects a position where the user of the electronic device 100 touches the top panel 120 (hereinafter referred to as a position of a manipulation input).
  • GUI graphic user interface
  • GUI manipulation parts Various graphic user interface buttons and the like (hereinafter referred to as GUI manipulation parts) are displayed on the display panel 160 located under the touch panel 150 . Therefore, the user of the electronic device 100 touches the top panel 120 with the fingertip in order to manipulate GUI manipulation parts.
  • GUI graphic user interface
  • the touch panel 150 may be a position detector that can detect a position of a manipulation input performed by the user on the top panel 120 .
  • the touch panel 150 may be a capacitance type or a resistive type position detector.
  • the embodiment in which the touch panel 150 is a capacitance type position detector will be described. Even if there is a clearance gap between the touch panel 150 and the top panel 120 , the touch panel 150 can detect a manipulation input performed on the top panel 120 .
  • the top panel 120 is disposed on the input surface side of the touch panel 150 .
  • the top panel 120 may be integrated into the touch panel 150 .
  • the surface of the touch panel 150 becomes the surface 120 A of the top panel 120 as illustrated in FIG. 2 and FIG. 3 , and thus becomes the manipulation surface.
  • the top panel 120 illustrated in FIG. 2 and FIG. 3 may be omitted.
  • the surface of the touch panel 150 becomes the manipulation surface in this case as well. In this case, the panel having the manipulation surface may be vibrated at the natural frequency of that panel.
  • the touch panel 150 may be disposed on the top panel 120 .
  • the surface of the touch panel 150 becomes the manipulation surface in this case as well.
  • the touch panel 150 is a capacitance type
  • the top panel 120 illustrated in FIG. 2 and FIG. 3 may be omitted.
  • the surface of the touch panel 150 becomes the manipulation surface in this case as well.
  • the panel having the manipulation surface may be vibrated at a natural frequency of that panel.
  • the display panel 160 may be any display part that can display images.
  • the display panel 160 may be a liquid crystal display panel, an organic electroluminescence (EL) panel, or the like, for example.
  • the display panel 160 is placed inside the recessed portion 110 A of the housing 110 and placed on (the positive side in the z-axis direction of) the substrate 170 using a holder and the like (not illustrated).
  • the display panel 160 is driven and controlled by the driver IC 161 , which will be described later, and displays GUI manipulation parts, images, characters, symbols, figures, and the like according to the operating condition of the electronic device 100 .
  • a position of a display region of the display panel 160 is associated with coordinates of the touch panel 150 .
  • each pixel of the display panel 160 may be associated with coordinates of the touch panel 150 .
  • the substrate 170 is disposed inside the recessed portion 110 A of the housing 110 .
  • the display panel 160 and the touch panel 150 are disposed on the substrate 170 .
  • the display panel 160 and the touch panel 150 are fixed to the substrate 170 and housing 110 using the holder and the like (not illustrated).
  • various circuits necessary to drive the electronic device 100 are mounted on the substrate 170 .
  • the camera 180 which is a digital camera configured to acquire a color image, acquires an image in a field of view that includes a photographic subject.
  • the image in the field of view acquired by the camera 180 includes an image of a photographic subject and an image of a background.
  • the camera 180 is an example of a first imaging part.
  • CMOS complementary metal-oxide semiconductor
  • the camera 180 may be a digital camera for monochrome photography.
  • the infrared camera 190 acquires a range image in the field of view that includes the photographic subject by irradiating infrared light from the light source 191 onto photographic subject and imaging the reflected light.
  • the range image in the field of view acquired by the infrared camera 190 includes a range image of a photographic subject and a range image of a background.
  • the infrared camera 190 is a projection-type range image camera.
  • the projection-type range image camera is a camera that projects infrared light and the like onto a photographic subject and reads the infrared light reflected from the photographic subject.
  • a time-of-flight (ToF) range image camera is an example of such a projection-type range image camera.
  • the ToF range image camera is a camera that measures a distance between the ToF range image camera and the photographic subject based on a roundtrip time that the projected infrared light travels.
  • the ToF range image camera includes the infrared camera 190 and the infrared light source 191 .
  • the infrared camera 190 and the infrared light source 191 are an example of a second imaging part.
  • the camera 180 and the infrared camera 190 are disposed proximate to each other on the bottom surface of the housing 110 . Because image processing is performed by using both an image acquired by the camera 180 and a range image acquired by the infrared camera 190 , a size, direction, and the like of an object in the image acquired by the camera 180 can match with those of the range image acquired by the infrared camera by disposing the camera 180 and the infrared camera 190 proximate to each other. Smaller differences in the size and direction between the object of the image acquired by the camera 180 and the object of the range image acquired by the infrared camera 190 make the image processing easier.
  • the electronic device 100 having the above-described configuration extracts a range image of the photographic subject based on the image in the field of view acquired by the camera 180 and the range image acquired by the infrared camera 190 . Subsequently, the electronic device 100 determines whether the photographic subject is a glossy object based on noise included in the range image of the photographic subject.
  • a glossy object is a metallic ornament.
  • An example of a non-glossy object is a stuffed toy.
  • the electronic device 100 drives the vibrating element 140 to vibrate the top panel 120 at a frequency in the ultrasound frequency band when the user touches the image of the photographic subject displayed on the display panel 160 and moves the finger along the surface 120 A of the top panel 120 .
  • the frequency in the ultrasound frequency band is a resonance frequency of a resonance system that includes the top panel 120 and the vibrating element 140 . At this frequency, a standing wave is generated on the top panel 120 .
  • the electronic device 100 drives the vibrating element 140 by using a driving signal having larger amplitude, compared to when the photographic subject is a non-glossy object.
  • the electronic device 100 drives the vibrating element 140 by using a driving signal having smaller amplitude, compared to when the photographic subject is a glossy object.
  • a driving signal having larger amplitude is used to drive the vibrating element 140 when the photographic subject is a non-glossy object, compared to when the photographic subject is a glossy object.
  • a driving signal having smaller amplitude is used to drive the vibrating element 140 when the photographic subject is a non-glossy object, compared to when the photographic subject is a non-glossy object.
  • the vibrating element 140 When the vibrating element 140 is driven by using a driving signal having relatively smaller amplitude, a thinner layer of air is present by a squeeze effect between the surface 120 A of the top panel 120 and the finger. As a result, a higher kinetic friction coefficient and a tactile sensation of touching the surface of a non-glossy object can be provided.
  • the amplitude of a driving signal may be changed in accordance with the elapsed time.
  • the amplitude of a driving signal may be changed in accordance with the elapsed time so that the user's fingertip can be provided with a tactile sensation of touching a stuffed toy.
  • the electronic device 100 does not drive the vibrating element 140 when the user touches other regions than the photographic subject image displayed on the display panel 160 .
  • the electronic device 100 provides, through the top panel 120 , the user with the tactile sensation of the photographic subject by changing the amplitude of the driving signal depending on whether the photographic subject is a glossy object.
  • FIGS. 5A and 5B are drawings illustrating crests of a standing wave formed in parallel with a short side of a top panel, of standing waves generated on the top panel by a natural vibration in an ultrasound frequency band.
  • FIG. 5A is a side view
  • FIG. 5B is a perspective view.
  • the same XYZ coordinates as those in FIG. 2 and FIG. 3 are defined.
  • the amplitude of the standing wave is exaggeratingly illustrated in FIGS. 5A and 5B .
  • the vibrating element 140 is omitted in FIGS. 5A and 5B .
  • the natural vibration frequency (resonance frequency) f of the top panel 120 is expressed by the following formals (1) and (2), where E is the Young's modulus of the top panel 120 , ⁇ is the density of the top panel 120 , ⁇ is the Poisson's ratio of the top panel 120 , l is the length of a long side of the top panel 120 , t is the thickness of the top panel 120 , and k is a periodic number of the standing wave generated along the direction of the long side of the top panel 120 . Because the standing wave has the same waveforms in every half cycle, the periodic number k takes values at intervals of 0.5 (i.e., 0.5, 1, 1.5, 2, etc.).
  • coefficient ⁇ included in formula (2) corresponds to the coefficients other than k 2 included in formula (1).
  • the waveform of the standing wave in FIGS. 5A and 5B is provided as an example in which the periodic number k is 10.
  • the periodic number k is 10.
  • GorillaTM glass having the length of a long side 1 of 140 mm, length of a short side of 80 mm, and thickness t of 0.7 mm is used as the top panel 120 and if the periodic number k is 10, the natural vibration frequency f will be 33.5 kHz.
  • a driving signal whose frequency is 33.5 kHz may be used.
  • top panel 120 is a flat member, when the vibrating element 140 (see FIG. 2 and FIG. 3 ) is driven to generate a natural vibration in the ultrasound frequency band, the top panel 120 deflects, and as a result, a standing wave is generated on the surface 120 A as illustrated in FIGS. 5A and 5B .
  • the embodiment in which the single vibrating element 140 is bonded to the surface on the negative side in the z-axis direction of the top panel 120 , along the short side extending in the x-axis direction, at the positive side in the y-axis direction will be described.
  • two vibrating elements 140 may be used. If two vibrating elements 140 are used, the other vibrating element 140 may be bonded to the surface on the negative side in the z-axis direction of the top panel 120 , along the short side extending in the x-axis direction, at the negative side in the y-axis direction.
  • two vibrating elements 140 are axisymmetrically disposed with respect to a centerline parallel to the two short sides of the top panel 120 .
  • the two vibrating elements 140 may be driven in the same phase if the periodic number k is an integer. If the periodic number k is a decimal (a number containing an integer part and a fractional part), the two vibrating elements 140 may be driven in opposite phases.
  • FIGS. 6A and 6B are drawings illustrating cases in which a kinetic friction force applied to a user's fingertip performing a manipulation input changes by the natural vibration in the ultrasound frequency band generated on the top panel of the electronic device.
  • the user touches the top panel 120 with the fingertip, the user performs a manipulation input by moving the finger toward the near side from the far side of the top panel 120 along the direction of an arrow.
  • the vibration can be switched on and off by turning on and off the vibrating element 140 .
  • the vibration can be switched on and off by turning on and off the vibrating element 140 (see FIG. 2 and FIG. 3 ).
  • regions that the user's finger touches while the vibration is turned off are represented in gray and regions that the user's finger touches while the vibration is turned on are represented in white.
  • FIGS. 5A and 5B the natural vibration in the ultrasound frequency band is generated on the entire top panel 120 .
  • FIGS. 6A and 6B illustrate operation patterns in which the vibration is switched on and off when the user's finger moves toward the near side from the far side of the top panel 120 .
  • FIGS. 6A and 6B when seen in the depth direction, the regions of the top panel 120 that the user's finger touches while the vibration is turned off are represented in gray and the regions of the top panel 120 that the user's finger touches while the vibration is turned on are represented in white.
  • the vibration is turned off when the user's finger is located on the far side of the top panel 120 , and the vibration is turned on while the user's finger moves toward the near side.
  • the kinetic friction force applied to the fingertip increases on the far side of the top panel 120 represented in gray.
  • the kinetic friction force applied to the fingertip decreases on the near side of the top panel 120 represented in white.
  • the user who performs the manipulation input as illustrated in FIG. 6A senses that the kinetic friction force applied to the fingertip is decreased when the vibration is turned on. As a result, the user feels a sense of slipperiness with the finger. In this case, because the surface 120 A of the top panel 120 becomes more slippery, the user senses as if a recessed portion exists on the surface 120 A of the top panel 120 when the kinetic friction force decreases.
  • the kinetic friction force applied to the fingertip decreases on the far side of the top panel 120 represented in white.
  • the kinetic friction force applied to the fingertip increases on the near side of the top panel 120 represented in gray.
  • the user who performs the manipulation input as illustrated in FIG. 6B senses that the kinetic friction force applied to the fingertip is increased when the vibration is turned off. As a result, the user feels a sense of non-slipperiness or roughness with the finger. In this case, because the surface 120 A of the top panel 120 becomes of higher roughness, the user senses as if a projecting portion exists on the surface of the top panel 120 when the kinetic friction force increase.
  • the user can sense projections and recesses with the fingertip in the cases illustrated in FIGS. 6A and 6B .
  • a person's tactile sensation of projections and recesses is disclosed in “The Printed-matter Typecasting Method for Haptic Feel Design and Sticky-band Illusion,” (The collection of papers of the 11th SICE system integration division annual conference (SI2010, Sendai), December 2010, pages 174 to 177).
  • a person's tactile sensation of projections and recesses is also disclosed in “The Fishbone Tactile Illusion” (Collection of papers of the 10th Congress of the Virtual Reality Society of Japan, September, 2005).
  • FIG. 7 is a drawing illustrating the configuration of the electronic device 100 of the first embodiment.
  • the electronic device 100 includes the vibrating element 140 , an amplifier 141 , the touch panel 150 , a driver integrated circuit (IC) 151 , the display panel 160 , a driver IC 161 , the camera 180 , the infrared camera 190 , the infrared light source 191 , a controlling part 200 , a sinusoidal wave generator 310 , and an amplitude modulator 320 .
  • IC driver integrated circuit
  • the controlling part 200 includes an application processor 220 , a communication processor 230 , a drive controlling part 240 , and a memory 250 .
  • the controlling part 200 is implemented by an IC chip.
  • the single controlling part 200 is implemented by the application processor 220 , the communication processor 230 , the drive controlling part 240 , and the memory 250 will be described.
  • the drive controlling part 240 may be provided outside the controlling part 200 as a separate IC chip or processor.
  • necessary data for drive control of the drive controlling part 240 may be stored in a separate memory from the memory 250 .
  • the housing 110 the top panel 120 , the double-sided adhesive tape 130 , and the substrate 170 (see FIG. 2 ) are omitted.
  • the amplifier 141 a driver IC 151 , the driver IC 161 , the application processor 220 , the drive controlling part 240 , the memory 250 , the sinusoidal wave generator 310 , and the amplitude modulator 320 will be described.
  • the amplifier 141 is disposed between the amplitude modulator 320 and the vibrating element 140 .
  • the amplifier 141 amplifies a driving signal output from the amplitude modulator 320 and drives the vibrating element 140 .
  • the driver IC 151 is coupled to the touch panel 150 , detects position data representing the position where a manipulation input is performed on the touch panel 150 , and outputs the position data to the controlling part 200 . As a result, the position data is input to the application processor 220 and the drive controlling part 240 .
  • the driver IC 161 is coupled to the display panel 160 , inputs rendering data output from the application processor 220 to the display panel 160 , and displays, on the display panel 160 , images based on the rendering data. In this way, GUI manipulation parts, images, or the like based on the rendering data are displayed on the display panel 160 .
  • the application processor 220 performs processes for executing various applications of the electronic device 100 .
  • a camera controlling part 221 an image processing part 222 , a range image extracting part 223 , a gloss determining part 224 , and an amplitude data allocating part 225 are particularly described.
  • the camera controlling part 221 controls the camera 180 , the infrared camera 190 , and the infrared light source 191 .
  • the camera controlling part 221 performs imaging processing by using the camera 180 .
  • the camera controlling part 221 causes infrared light to be output from the infrared light source 191 and performs imaging processing by using the infrared camera 190 .
  • Image data representing images acquired by the camera 180 and range image data representing range images acquired by the infrared camera 190 are input to the camera controlling part 221 .
  • the camera controlling part 221 outputs the image data and the range image data to the range image extracting part 223 .
  • the image processing part 222 executes image processing other than that executed by the range image extracting part 223 and the gloss determining part 224 .
  • the image processing executed by the image processing part 222 will be described later.
  • the range image extracting part 223 extracts a range image of a photographic subject based on the image data and the range image data input from the camera controlling part 221 .
  • the range image of the photographic subject is data in which each pixel of the image representing the photographic subject is associated with data representing a distance between a lens of the infrared camera 190 and the photographic subject. The processing for extracting a range image of a photographic subject will be described later with reference to FIG. 8 and FIG. 12 .
  • the gloss determining part 224 analyzes noise included in the range image of the photographic subject extracted by the range image extracting part 223 . Based on the analysis result, the gloss determining part 224 determines whether the photographic subject is a glossy object. The processing for determining whether the photographic subject is a glossy object based on analysis result of noise will be described with reference to FIG. 12 .
  • the amplitude data allocating part 225 allocates amplitude data of the driving signal of the vibrating element 140 to the image of the photographic subject determined to be the glossy object by the gloss determining part 224 or to the image of the photographic subject determined to be the non-glossy object by the gloss determining part 224 .
  • the processing executed by the amplitude data allocating part 225 will be described later with reference to FIG. 12 .
  • the communication processor 230 executes processing necessary for the electronic device 100 to perform third generation (3G), fourth generation (4G), Long-Term Evolution (LTE), and Wi-Fi communications.
  • the drive controlling part 240 outputs amplitude data to the amplitude modulator 320 when two predetermined conditions are met.
  • the amplitude data is data that represents an amplitude value for adjusting the intensity of driving signals used to drive the vibrating element 140 .
  • the amplitude value is set according to the degree of time change of the position data.
  • the moving speed of the user's fingertip along the surface 120 A of the top panel 120 is used as the degree of time change of the position data.
  • the moving speed of the user's fingertip is calculated by the drive controlling part 240 based on the degree of time change of the position data input from the driver IC 151 .
  • the drive controlling part 240 vibrates the top panel 120 in order to change a kinetic friction force applied to the user's fingertip when the fingertip moves along the surface 120 A of the top panel 120 . Such a kinetic friction force is generated while the fingertip is moving. Therefore, the drive controlling part 240 causes the vibrating element 140 to vibrate when the moving speed becomes equal to or greater than a predetermined threshold speed.
  • the first predetermined condition is that the moving speed is greater than or equal to the predetermined threshold speed.
  • the amplitude value represented by the amplitude data output from the drive controlling part 240 becomes zero when the moving speed is less than the predetermined threshold speed.
  • the amplitude value is set to a predetermined amplitude value according to the moving speed when the moving speed becomes equal to or greater than the predetermined threshold speed. In a case where the moving speed becomes equal to or greater than the predetermined threshold speed, the higher the moving speed is, the smaller the amplitude value is set, and the lower the moving speed is, the larger the amplitude value is set.
  • the drive controlling part 240 outputs the amplitude data to the amplitude modulator 320 when the position of the user's fingertip performing a manipulation input is located in a predetermined region where a vibration is to be generated.
  • the second predetermined condition is that the position of the user's fingertip performing a manipulation input is located in a predetermined region where a vibration is to be generated.
  • Whether or not the position of the fingertip performing a manipulation input is located in a predetermined region where a vibration is to be generated is determined based on whether or not the position of the fingertip performing the manipulation input is located inside the predetermined region. Also, the predetermined region where the vibration is to be generated is a region where a photographic subject, which is specified by the user, is displayed.
  • a position of a GUI manipulation part displayed on the display panel 160 , a position of a region that displays an image, a position of a region representing an entire page, and the like on the display panel 160 are specified by region data representing such regions.
  • the region data exists in all applications for each GUI manipulation part displayed on the display panel 160 , for each region that displays an image, and for each region that displays an entire page.
  • a type of an application executed by the electronic device 100 is relevant in determining, as the second predetermined condition, whether the position of the user's fingertip performing a manipulation input is located in a predetermined region where a vibration is to be generated. This is because displayed contents of the display panel 160 differ depending on the type of the application.
  • a type of a manipulation input which is performed by moving the fingertip along the surface 120 A of the top panel 120 , differs depending on the type of the application.
  • One type of manipulation input performed by moving the fingertip along the surface 120 A of the top panel 120 is what is known as a flick operation, which is used to operate GUI manipulation parts, for example.
  • the flick operation is performed by flicking (snapping) the fingertip on the surface 120 A of the top panel 120 for a relatively short distance.
  • a swipe operation is performed, for example.
  • the swipe operation is performed by brushing the fingertip along the surface of the top panel 120 for a relatively long distance.
  • the swipe operation is performed when the user turns over pages or photos, for example.
  • a drag operation is performed to drag the slider.
  • Manipulation inputs performed by moving the fingertip along the surface 120 A of the top panel 120 are selectively used depending on the type of the application. Therefore, a type of an application executed by the electronic device 100 is relevant in determining whether the position of the user's fingertip performing a manipulation input is located in a predetermined region where a vibration is to be generated.
  • the drive controlling part 240 determines whether the position represented by the position data input from the driver IC 151 is located in a predetermined region where a vibration is to be generated.
  • the two predetermined conditions required for the drive controlling part 240 to output amplitude data to the amplitude modulator 320 are that the moving speed of the fingertip is greater than or equal to the predetermined threshold speed and that coordinates of the position of the manipulation input are located in a predetermined region where a vibration is to be generated.
  • the electronic device 100 drives the vibrating element 140 to vibrate the top panel 120 at a frequency in the ultrasound frequency band.
  • the predetermined region where the vibration is to be generated is a region where the photographic subject specified by the user is displayed on the display panel 160 .
  • the drive controlling part 240 reads amplitude data representing an amplitude value and outputs the amplitude data to the amplitude modulator 320 .
  • the memory 250 stores data and programs necessary for the application processor 220 to execute applications and stores data and programs necessary for the communication processor 230 to execute communication processing.
  • the sinusoidal wave generator 310 generates sinusoidal waves necessary to generate a driving signal for vibrating the top panel 120 at a natural vibration frequency. For example, in order to vibrate the top panel 120 at a natural frequency f of 33.5 kHz, a frequency of the sinusoidal waves becomes 33.5 kHz.
  • the sinusoidal wave generator 310 inputs sinusoidal wave signals in the ultrasound frequency band into the amplitude modulator 320 .
  • the amplitude modulator 320 generates a driving signal by modulating the amplitude of a sinusoidal wave signal input from the sinusoidal wave generator 310 based on amplitude data input from the drive controlling part 240 .
  • the amplitude modulator 320 generates a driving signal by modulating only the amplitude of the sinusoidal wave signal in the ultrasound frequency band input from the sinusoidal wave generator 310 without modulating a frequency or a phase of the sinusoidal wave signal.
  • the driving signal output from the amplitude modulator 320 is a sinusoidal wave signal in the ultrasound frequency band obtained by modulating only the amplitude of the sinusoidal wave signal in the ultrasound frequency band input from the sinusoidal wave generator 310 .
  • the amplitude of the driving signal becomes zero. This is the same as the case in which the amplitude modulator 320 does not output the driving signal.
  • FIG. 8 is a drawing illustrating an example of use of the electronic device 100 .
  • the user takes photographs of a stuffed toy 1 and a metallic ornament 2 by using the camera 180 and the infrared camera 190 of the electronic device 100 . More specifically, the user takes a photograph of the stuffed toy 1 by using the camera 180 and takes a photograph of the metallic ornament 2 by using the camera 180 . Also, the user takes a photograph of the stuffed toy 1 by using the infrared camera 190 and takes a photograph of the metallic ornament 2 by using the infrared camera 190 .
  • the first step is performed by the camera controlling part 221 .
  • the stuffed toy 1 is a stuffed animal character.
  • the stuffed toy 1 is made of non-glossy fabrics and gives the user a fluffy tactile sensation when the user touches the stuffed toy 1 with the finger.
  • the stuffed toy 1 is an example of a non-glossy object.
  • the metallic ornament 2 is an ornament having a shape of a skull.
  • the metallic ornament has a smooth curved surface and gives the user a slippery tactile sensation when the user touches the metallic ornament 2 with the finger.
  • the metallic ornament 2 is an example of a glossy object.
  • the glossy object as used herein means that the surface of the object is flat or curved, is smooth to some degree, reflects light to some degree, and provides a slippery tactile sensation to some degree when the user touches the object.
  • whether an object is glossy or non-glossy is determined by its tactile sensation.
  • a tactile sensation differs from person to person. Therefore, for example, a boundary (threshold) for determining whether an object is glossy can be set according to the user's preference.
  • an image 1 A of the stuffed toy 1 and an image 2 A of the metallic ornament 2 are acquired.
  • the image 1 A and the image 2 A are acquired by separately photographing the stuffed toy 1 and the metallic ornament 2 by the camera 180 .
  • the image 1 A and the image 2 A are displayed on the display panel 160 of the electronic device 100 .
  • the second step is performed by the camera controlling part 221 and the image processing part 222 .
  • the electronic device 100 performs image processing for the image 1 A and the image 2 A. Subsequently, the electronic device 100 creates an image 1 B and an image 2 B.
  • the image 1 B and the image 2 B represent regions (hereinafter referred to as specific region(s)) that display the photographic subjects (the stuffed toy 1 and the metallic ornament 2 ) included in the image 1 A and the image 2 A, respectively.
  • a specific region that displays the photographic subject is indicated in white and a background region other than the photographic subject is indicated in black.
  • the region indicated in black is a region where no data exists.
  • the region indicated in white represents pixels of the image of the photographic subject and corresponds to the display region of the stuffed toy 1 .
  • a region that displays the photographic subject is indicated in white and a background region other than the photographic subject is indicated in black.
  • the region indicated in black is a region where no data exists.
  • the region indicated in white represents pixels of the image of the photographic subject and corresponds to the display region of the metallic ornament 2 .
  • the third step is performed by the image processing part 222 .
  • a range image 1 C of the stuffed toy 1 and a range image 2 C of the metallic ornament 2 are acquired.
  • the range image 1 C and the range image 2 C are acquired by separately photographing the stuffed toy 1 and the metallic ornament 2 by the infrared camera 190 .
  • the fourth step is performed by the image processing part 222 simultaneously with the second step and the third step.
  • a range image 1 D in the specific region and a range image 2 D in the specific region are acquired respectively by extracting, from the range images 1 C and 2 C, images that correspond to pixels in the specific regions included in the images 1 B and 2 B.
  • the fifth step is performed by the range image extracting part 223 .
  • ratios of noise included in the range images 1 D and 2 D of the specific regions are calculated. It is determined whether the calculated ratios of noise are equal to or greater than a predetermined value.
  • the photographic subject corresponding to the range image 1 D or the range image 2 D of the specific region is a glossy object.
  • the photographic subject corresponding to the range image 1 D or the range image 2 D of the specific region is a non-glossy object.
  • the photographic subject (stuffed toy 1 ) corresponding to the range image 1 D of the specific region is determined to be a non-glossy object, and the photographic subject (metallic ornament 2 ) corresponding to the range image 2 D of the specific region is determined to be a glossy object.
  • amplitude data representing relatively small amplitude that corresponds to the non-glossy object is allocated.
  • amplitude data representing relatively large amplitude that corresponds to the glossy object is allocated.
  • the sixth step is now completed.
  • the sixth step is performed by the gloss determining part 224 and the amplitude data allocating part 225 .
  • the image 2 A of the metallic ornament 2 is displayed on the display panel 160 of the electronic device 100 .
  • the vibrating element 140 is driven and the tactile sensation appropriate to the metallic ornament 2 is provided.
  • FIG. 9 and FIG. 10 are drawings illustrating range images acquired by the infrared camera 190 .
  • infrared light is irradiated from the infrared light source 191 onto an object 3 (a photographic subject). Then, the infrared light is diffusely reflected by the surface of the object 3 . The infrared light reflected by the object 3 is imaged by the infrared camera 190 . As a result, a range image is acquired.
  • a range image 5 illustrated at the bottom of FIG. 10 includes a range image 3 A of the object 3 and a range image 4 A of the background.
  • the range image is provided with range information for each pixel.
  • the distance from the infrared camera 190 is illustrated in greyscale for convenience of explanation.
  • a region nearer to the infrared camera 190 is indicated in light gray and a region farther from the infrared camera 190 is indicated in dark gray.
  • the object 3 is nearer than the background. Therefore, the range image 3 A of the object 3 is indicated in light gray and the range image 4 A of the background is indicated in dark gray.
  • a part (a part enclosed in a box) of the range image 5 illustrated in the lower side of FIG. 10 is enlarged and illustrated in the upper side of FIG. 10 .
  • the range image 5 is provided with range information for each pixel.
  • the range image 3 A of the object 3 has range information of 100 (mm) and the range image 4 A of the background has range information of 300 (mm).
  • noise included in the range image will be described with reference to FIG. 11 .
  • FIG. 11 is a drawing illustrating the range image 5 including noise 3 A 1 .
  • the object 3 is a glossy object, it has high specular reflection characteristics that cause high reflection in a certain direction. Therefore, for some pixels, there may be a case in which reflected light does not return to the infrared camera 190 . Such pixels, for which reflected light of infrared light irradiated from the infrared light source 191 did not return, lack optical data about the reflected light and thus become the noise 3 A 1 . Because the noise 3 A 1 does not have any optical data, it is illustrated in black. Further, the noise 3 A 1 is regarded as a data lacking portion that lacks data about reflected light.
  • the electronic device 100 of the first embodiment determines whether the object 3 is a glossy object by using the noise 3 A 1 , and allocates amplitude data based on the determined result.
  • FIG. 12 and FIG. 13 illustrate flowcharts of processing for allocating amplitude data executed by the electronic device 100 of the first embodiment.
  • the processing illustrated in FIG. 12 and FIG. 13 is executed by the application processor 220 .
  • the application processor 220 determines a threshold (step S 100 ).
  • the threshold is used as a reference value for determining whether a ratio of noise included in the range image of the specific region is small or large in step S 170 , which is performed later.
  • the processing in Step S 100 is executed by the image processing part 222 of the application processor 220 .
  • the application processor 220 displays, on the display panel 160 , an input screen for setting a threshold, and prompts the user to set a threshold.
  • the user sets a threshold by manipulating the input screen of the display panel 160 .
  • the processing executed by the application processor 220 when the user manipulates the input screen will be described below with reference to FIG. 13 .
  • the application processor 220 photographs a photographic subject by using the camera 180 and the infrared camera 190 (step S 110 ).
  • the application processor 220 displays, on the display panel 160 , a message requesting the user to photograph the photographic subject.
  • the processing in step S 110 is achieved.
  • step S 110 is executed by the camera controlling part 221 and corresponds to the first step illustrated in FIG. 8 .
  • step S 110 Upon the completion of step S 110 , the application processor 220 executes steps S 120 and S 130 simultaneously with step S 140 .
  • the application processor 220 acquires a color image from the camera 180 (step S 120 ).
  • the processing in step S 120 is executed by the camera controlling part 221 and the image processing part 222 and corresponds to the second step illustrated in FIG. 8 .
  • the application processor 220 acquires an image of the specific region by image-processing the color image acquired in step S 120 (step S 130 ).
  • the processing in step S 130 is executed by the image processing part 222 and corresponds to the third step illustrated in FIG. 8 . Further, the details of processing for acquiring an image of the specific region will be described with reference to FIG. 14 and FIG. 15 .
  • Step S 140 corresponds to the fourth step illustrated in FIG. 8 .
  • the application processor 220 acquires a range image of the specific region based on the image of the specific region acquired in step S 130 and the range image acquired in step S 140 (step S 150 ).
  • the range image of the specific region represents a range image of the photographic subject.
  • the processing in step S 150 is executed by the camera controlling part 221 and the image processing part 222 and corresponds to the fifth step illustrated in FIG. 8 .
  • step S 160 the application processor 220 obtains a ratio of noise included in the range image of the specific region acquired in step S 150 to the range image of the specific region (step S 160 ).
  • the processing in step S 160 is executed by the gloss determining part 224 and corresponds to the sixth step illustrated in FIG. 8 .
  • the details of a method for obtaining a ratio of noise will be described with reference to FIG. 16 .
  • the application processor 220 determines whether the ratio of noise obtained in step S 100 is equal to or greater than the threshold acquired in step S 100 (step S 170 ).
  • Step S 170 corresponds to the sixth step illustrated in FIG. 8 .
  • step S 180 A When the application processor 220 determines that the ratio of noise is not equal to or greater than the threshold (NO in S 170 ), it is determined that the specific region is a non-glossy region (step S 180 A).
  • the processing in step S 180 A is executed by the gloss determining part 224 and corresponds to the sixth step illustrated in FIG. 8 .
  • step S 180 B When the application processor 220 determines that the ratio of noise is equal to or greater than the threshold (YES in S 170 ), it is determined that the specific region is a glossy region (step S 180 B).
  • the processing in step S 180 B is executed by the gloss determining part 224 and corresponds to the sixth step illustrated in FIG. 8 .
  • the application processor 220 allocates amplitude data based on the result determined in step S 180 A or in step S 180 B to the specific region (step S 190 ).
  • the application processor 220 stores data representing the specific region to which the amplitude data is allocated in the memory 250 .
  • the processing in step S 190 is executed by the amplitude data allocating part 225 and corresponds to the sixth step illustrated in FIG. 8 .
  • the processing illustrated in FIG. 13 is started upon the start of step S 100 .
  • the application processor 220 sets the number m (m represents an integer of 1 or more) of glossy objects (hereinafter referred to as glossy objects) to 1 (step S 101 A).
  • This setting is a preparation for acquiring a color image of the first glossy object.
  • the application processor 220 acquires a range image of the m-th glossy object (step S 102 A). In the same way as described in steps S 110 , S 120 , S 130 , S 140 , and S 150 , based on the color image acquired from the camera 180 and the range image acquired from the infrared camera 190 , a range image of the glossy object is acquired by acquiring a range image of the specific region that corresponds to the glossy object.
  • the range image of only the glossy object which is included in the field of view when photographed by the camera 180 and the infrared camera 190 , respectively, is acquired as the range image of the specific region that corresponds to the glossy object.
  • the color image and the range image employed in step S 102 A may be acquired by photographing a glossy object at hand by using the camera 180 and the infrared camera 190 .
  • the user may read the color image and the range image preliminarily saved in the memory 250 of the electronic device 100 .
  • the application processor 220 obtains a ratio of noise of the m-th glossy object (step S 103 A).
  • the ratio of noise can be obtained in the same way as step S 160 by processing the range image of the specific region, which has been acquired in step S 102 A.
  • the application processor 220 determines whether the ratio of noise is equal to or greater than 50% (step S 104 A).
  • the threshold for determining the ratio of noise is set to 50% as an example herein.
  • the user may set any threshold value according to the user's preference.
  • the application processor 220 determines that the ratio of noise is equal to or greater than 50% (YES in S 104 A)
  • the range image of the specific region, whose ratio of noise has been determined to be equal to or greater than 50% is discarded (step S 105 A). This is because the range image of the specific region whose ratio of noise is equal to or greater than 50% is not suitable for a region of the glossy object (glossy region) where the range image of the specific region is included.
  • the application processor 220 causes the flow to return to step S 102 A.
  • the application processor 220 determines that the ratio of noise is not equal to or greater than 50% (NO in S 104 A)
  • the range image of the specific region and its ratio of noise are employed as glossy region data (step S 107 A).
  • the application processor 220 saves the glossy region data employed in step S 107 A in the memory 250 (step S 108 A). Upon the completion of step S 108 A, the application processor 220 causes the flow to proceed to step S 101 B.
  • the application processor 220 sets the number n (n represents an integer of 1 or more) of non-glossy objects (hereinafter referred to as non-glossy objects) to 1 (step S 101 B). This is a preparation for acquiring a color image of the first non-glossy object.
  • the application processor 220 acquires a range image of the n-th non-glossy object (step S 102 B).
  • a range image of the non-glossy object is acquired by acquiring a range image of the specific region that corresponds to the non-glossy object.
  • the range image of only the non-glossy object which is included in the field of view when photographed by the camera 180 and the infrared camera 190 , respectively, is acquired as the range image of the specific region that corresponds to the non-glossy object.
  • the color image and the range image employed in step S 102 B may be acquired by photographing a non-glossy object at hand by using the camera 180 and the infrared camera 190 .
  • the user may read the color image and the range image preliminarily saved in the memory 250 of the electronic device 100 .
  • the application processor 220 obtains a ratio of noise of the n-th non-glossy object (step S 103 B).
  • the ratio of noise can be obtained in the same way as step S 160 by processing the range image of the specific region, which has been acquired in step S 102 B.
  • the application processor 220 determines whether the ratio of noise is equal to or greater than 50% (step S 104 B).
  • the threshold for determining the ratio of noise is set to 50% as an example herein.
  • the user may set any threshold value according to the user's preference.
  • the application processor 220 determines that the ratio of noise is equal to or greater than 50% (YES in S 104 B)
  • the range image of the specific region, whose ratio of noise has been determined to be equal to or greater than 50% is discarded (step S 105 B). This is because the range image of the specific region whose ratio of noise is equal to or greater than 50% is not suitable for a region of the non-glossy object (non-glossy region) where the range image of the specific region is included.
  • the application processor 220 determines that the ratio of noise is not equal to or greater than 50% (NO in S 104 B)
  • the range image of the specific region and its ratio of noise are employed as non-glossy region data (step S 107 B).
  • the application processor 220 saves the non-glossy region data employed in step S 107 B in the memory 250 (step S 108 B).
  • the application processor 220 displays, on the display panel 160 , the ratios of noise of the specific regions included in the glossy region data and in the non-glossy region data saved in the memory 250 (step S 109 B).
  • the application processor 220 sets a threshold to the value specified by the user's manipulation input (step S 109 C).
  • the ratio of noise of the specific region is 5%
  • the ratio of noise of the specific region is 0%
  • the user sets a threshold for the ratio of noise to 2.5%, for example.
  • the threshold described in S 100 is determined.
  • FIG. 14 is a flowchart illustrating the processing in step S 130 in detail. The flow illustrated in FIG. 14 will be described with reference to FIGS. 15A through 15D .
  • FIGS. 15A through 15D are drawings illustrating image processing performed in step S 130 .
  • the application processor 220 sets either one of the larger area or the smaller area of the color image, which will be classified into two regions in step S 132 , as the specific region (step S 131 ). Whether the larger area or the smaller area is set as the specific region is decided by the user.
  • the specific region refers to a region that represents a display region of a photographic subject.
  • the reason why the above-described setting is configured is because a magnitude relationship between a photographic subject and a background becomes different, depending on whether the photographic subject is photographed in a larger size or photographed in a smaller size.
  • a region having a smaller area than that of the other region is set as the specific region, as an example herein.
  • the application processor 220 acquires the color image that has been classified into the two regions, one of which is the photographic subject and the other is the background, by using a graph-cut method (step S 132 ). For example, by performing a graph-cut method for the image 2 A (color image) illustrated in FIG. 15A , an image 2 A 1 illustrated in FIG. 15B is obtained. The image 2 A 1 is classified into a region 2 A 11 and a region 2 A 12 .
  • the application processor 220 calculates an area of one region 2 A 11 and an area of the other region 2 A 12 (steps S 133 A and S 133 B).
  • an area of the region 2 A 11 and an area of the region 2 A 12 may be calculated by counting the number of pixels included in the region 2 A 11 and the region 2 A 12 , respectively.
  • pixels may be counted, starting with the pixel closest to the origin 0, moving in the positive direction of the x-axis (positive column direction), and moving down row by row in the positive direction of the y-axis (positive row direction). In this way, all pixels may be counted.
  • the number of pixels in the region 2 A 11 is 92,160 pixels and the number of pixels in the region 2 A 12 is 215,040 pixels.
  • the application processor 220 compares the area calculated in step S 133 A with the area calculated in step S 133 B (step S 134 ).
  • the application processor 220 determines the specific region based on the compared result (step S 135 ).
  • a region having a smaller area than that of the other region has been set to be the specific region that represents the display region of the photographic subject in step S 131 . Therefore, of the region 2 A 11 and the region 2 A 12 , the region 2 A 11 having a smaller area is determined as the specific region.
  • the application processor 220 acquires an image of the specific region (step S 136 ).
  • the image 2 B (see FIG. 15D ), which corresponds to FIG. 15B in which the region 2 A 11 is the specific region, is acquired.
  • the specific region which is a region that displays the photographic subject, is indicated in white.
  • the background region other than the region that displays the photographic subject is indicated in black.
  • only the specific region contains data.
  • the data contained in the specific region represents pixels of the image of the photographic subject.
  • FIG. 16 is a flowchart illustrating processing for acquiring a ratio of noise.
  • the flow in FIG. 16 illustrates the details of the processing for determining the ratio of noise in step S 160 .
  • the flow illustrated in FIG. 16 is executed by the amplitude data allocating part 225 .
  • P refers to the number of pixels included in the specific region.
  • I(k) refers to a value representing a distance given to the k-th (1 ⁇ k ⁇ P) pixel, of the pixels included in the specific region.
  • N (0 ⁇ N ⁇ P) refers to the number of pixels in which noise appears.
  • R (0% ⁇ R ⁇ 100%) refers to a ratio of noise.
  • the k-th pixel may be counted by assigning an order to each pixel, starting with the pixel closest to the origin 0, moving in the positive direction of the x-axis (positive column direction), and moving down row by row in the positive direction of the y-axis (positive row direction).
  • the application processor 220 acquires the number of pixels P of the range image included in the specific region (step S 161 ).
  • the number of pixels in the region that has been determined to be the specific region may be acquired. For example, 92,160 pixels, which is the number of pixels in the region 2 A 11 illustrated in FIG. 15C , is acquired.
  • the application processor 220 refers to the value I(k) that represents the distance given to the k-th pixel (step S 163 ).
  • the value I(k) may be read from the k-th pixel in the specific region.
  • the application processor 220 determines whether the value I(k) that represents the distance given to the k-th pixel exists (step S 164 ). When the value I(k) that represents the distance is zero, it is determined that the value I(k) does not exist. When the value I(k) that represents the distance is not zero (if the positive value exists), it is determined that the value I(k) exists.
  • step S 166 When the application processor 220 determines that the value I(k) that represents the distance exists (YES in S 164 ), the application processor 220 causes the flow to proceed to step S 166 .
  • the application processor 220 determines whether k>P is established (step S 167 ).
  • the application processor 220 obtains a ratio of noise (step S 168 ).
  • the ratio of noise is acquired by the following formula (3):
  • the ratio of noise which is expressed as a percentage, is a ratio of the number of pixels in which noise appears to the total number of pixels P.
  • step S 160 the ratio of noise in step S 160 is obtained.
  • FIG. 17 is a drawing illustrating the amplitude data allocated by the amplitude data allocating part to the specific region.
  • FIG. 17 illustrates the amplitude data (voltage values) allocated to pixels located in the first column, the second column, and the third column in the x-axis direction, all of which are located in the first row in the y-axis direction.
  • the first column in the x-axis direction represents the column closest to the origin 0 in the x-axis direction.
  • the first row in the y-axis direction represents the row closest to the origin 0 in the y-axis direction.
  • the data in FIG. 17 illustrates amplitude values that are given to the pixels closest to the origin of the specific region. Further values exist in the x-axis direction and in the y-axis direction.
  • amplitude data for glossy objects and amplitude data for non-glossy objects are stored in the memory 250 .
  • the amplitude data allocating part 225 may read such amplitude data when allocating the amplitude data to each pixel in the specific region.
  • FIG. 18 is a drawing illustrating amplitude data for glossy objects and amplitude data for non-glossy objects stored in the memory 250 .
  • the amplitude data for glossy objects is set to 1.0 (V) and the amplitude data for non-glossy objects is set to 0.5 (V), for example.
  • amplitude data may be set to different values for each pixel in the specific region. For example, in the case of the stuffed toy 1 (see FIG. 8 ) whose surface has projecting and recessed portions, amplitude data may be changed periodically by a certain number of pixels. By allocating such amplitude data to the specific region, a tactile sensation of the surface of the stuffed toy 1 can be properly produced.
  • FIG. 19 is a drawing illustrating data stored in the memory 250 .
  • the data illustrated in FIG. 19 is data that associates data representing types of applications, region data representing coordinate values of specific regions, and pattern data representing vibration patterns with one another.
  • the application IDs may be assigned to each specific region with which vibration data is associated. Namely, the application ID of the specific region of the stuffed toy 1 (see FIG. 8 ) may be different from the application ID of the specific region of the metallic ornament 2 (see FIG. 8 ).
  • formulas f 1 to f 4 that express coordinate values of specific regions are illustrated.
  • the formulas f 1 to f 4 are formulas that express coordinates of specific regions such as the specific regions (see the third step in FIG. 8 ) included in the image 1 B and the image 2 B.
  • the pattern data that represents vibration patterns P 1 to P 4 are illustrated.
  • the pattern data P 1 to P 4 is data in which the amplitude data illustrated in FIG. 18 is allocated to each pixel in the specific region.
  • FIG. 20 is a flowchart illustrating processing executed by the drive controlling part of the electronic device of the embodiment.
  • An operating system (OS) of the electronic device 100 executes control for driving the electronic device 100 for each predetermined control cycle. Therefore, the drive controlling part 240 performs the flow illustrated in FIG. 20 repeatedly for each predetermined control cycle.
  • OS operating system
  • the drive controlling part 240 starts the processing upon the electronic device 100 being turned on.
  • the drive controlling part 240 acquires the region data with which a vibration pattern is associated in accordance with the type of the current application type (step S 1 ).
  • the drive controlling part 240 determines whether the moving speed is greater than or equal to the predetermined threshold speed (step S 2 ).
  • the moving speed may be calculated by using vector processing.
  • the threshold speed may be set to the minimum speed of the moving speed of the fingertip when manipulation inputs such as what are known as the flick operation, the swipe operation, or the drag operation are performed by moving the fingertip. Such a minimum speed may be set based on, for example, experiment results, the resolution of the touch panel 150 , and the like.
  • step S 3 the drive controlling part 240 determines whether the current coordinates represented by the position data are located in the specific region represented by the region data obtained in step S 1 (step S 3 ).
  • the vibration pattern corresponding to the current coordinates represented by the position data is obtained from the data illustrated in FIG. 19 (step S 4 ).
  • the drive controlling part 240 outputs the amplitude data (step S 5 ).
  • the amplitude modulator 320 generates the driving signal by modulating the amplitude of the sinusoidal wave output from the sinusoidal wave generator 310 , and the vibrating element 140 is driven.
  • step S 2 when the drive controlling part 240 determines that the moving speed is not equal to or greater than the predetermined threshold speed (NO in S 2 ) or when the drive controlling part 240 determines, in step S 3 , that the current coordinates are not located in the specific region represented by the region data obtained in step S 1 , the drive controlling part 240 sets the amplitude value to zero (step S 6 ).
  • the drive controlling part 240 outputs amplitude data whose amplitude value is zero, and the amplitude modulator 320 generates a driving signal by modulating the amplitude of the sinusoidal wave output from the sinusoidal wave generator 310 to zero. Therefore, the vibrating element 140 is not driven.
  • FIG. 21 is a drawing illustrating an example of an operation of the electronic device of the first embodiment.
  • a horizontal axis represents time and a vertical axis represents an amplitude value of the amplitude data.
  • the moving speed of the user's fingertip along the surface 120 A of the top panel 120 is assumed to be almost constant.
  • a glossy object is displayed on the display panel 160 . The user traces the image of the glossy object.
  • the user's fingertip located outside the specific region, begins to move leftward along the surface of the top panel 120 at a time point t 1 . Subsequently, at a time point t 2 , when the fingertip enters the specific region that displays the glossy object, the drive controlling part 240 causes the vibrating element 140 to vibrate.
  • the amplitude of the vibration pattern at this time is A 11 .
  • the vibration pattern has a driving pattern in which the vibration continues while the fingertip is moving in the specific region.
  • the drive controlling part 240 sets the amplitude value to zero. Therefore, immediately after the time point t 3 , the amplitude becomes zero.
  • the drive controlling part 240 outputs the amplitude data having the constant amplitude value (A 11 ), for example. Therefore, the kinetic friction force applied to the user's fingertip is lowered while the user's fingertip is touching and tracing the image of the object displayed in the specific region. As a result, the sensation of slipperiness and smoothness can be provided to the user's fingertip. Accordingly, the user can feel the tactile sensation of the glossy object. In the case of a non-glossy object, as the amplitude is smaller, the tactile sensation becomes gentle. For example, when the non-glossy object is the stuffed toy 1 (see FIG. 8 ), a fluffy and soft tactile sensation is provided.
  • FIG. 22 is a drawing illustrating a use scene of the electronic device 100 .
  • the user After the amplitude data is allocated to the specific region, the user displays the image 2 A of the metallic ornament 2 having a shape of a skull on the display panel 160 of the electronic device 100 .
  • the vibrating element 140 is not driven (see FIGS. 2, 3, and 7 ). Therefore, no squeeze effect is generated.
  • the vibrating element 140 is driven by the driving signal whose intensity has been modulated by using the amplitude data allocated to the specific region, as described above.
  • the user's fingertip moves slowly in other regions than the specific region that displays the metallic ornament 2 , as indicated by a short arrow, and the user's fingertip moves at a fast speed in the specific region that displays the metallic ornament 2 , as indicated by a long arrow.
  • the vibrating element 140 is driven by the driving signal whose intensity has been modulated by using smaller amplitude data than that of the metallic ornament 2 . Therefore, a tactile sensation of touching the fluffy stuffed toy 1 can be provided to the user.
  • the electronic device 100 and the drive controlling method that can provide a tactile sensation based on the presence or absence of gloss.
  • the user may freely set the amplitude data allocated to the specific region. In this way, the user can provide different tactile sensations according to the user preference.
  • the electronic device 100 is not required to include the camera 180 .
  • the electronic device 100 may obtain an infrared image from the infrared camera 190 and may obtain an image of the specific region by image-processing the infrared image instead of the above-described color image.
  • the infrared image refers to an image acquired by irradiating infrared light onto a photographic subject and converting the intensity of the reflected light into pixel values.
  • the infrared image is displayed in black and white.
  • the infrared image may be displayed on the display panel 160 of the electronic device 100 .
  • an infrared image acquired from the infrared camera 190 may be displayed on the display panel 160 .
  • a color image acquired from the camera 180 may be displayed on the display panel 160 .
  • An image acquired from the camera 180 is not required to be a color image and may be a black-and-white image.
  • a setting for determining a threshold in step S 100 differs from that of the first embodiment.
  • Other configurations are similar to those of the electronic device 100 of the first embodiment. Therefore, the same reference numerals are given to the similar configuration elements and thus their descriptions are omitted.
  • FIG. 23 is a flowchart illustrating processing for allocating amplitude data executed by an electronic device 100 of the second embodiment. The processing illustrated in FIG. 23 is executed by the application processor 220 .
  • steps S 101 A through S 108 A and steps S 101 B through S 108 B are similar to steps S 101 A through S 108 A and steps S 101 B through S 108 B illustrated in FIG. 13 .
  • step S 101 A processing for setting the number x 1 of glossy region data groups to 0 (zero) is added. Also, in step S 101 B, processing for setting the number y 1 of non-glossy region data groups to 0 (zero) is added.
  • the numbers x 1 and y 1 represent an integer of 2 or more, respectively.
  • step S 208 A is added between steps S 107 A and S 108 A.
  • step S 209 A is added between steps S 108 A and S 101 B.
  • step S 208 B is added between steps S 107 B and S 108 B. Also, following step S 108 B, steps S 209 B, S 210 A, and S 210 B are included.
  • the application processor 220 automatically determines a threshold by using a discriminant analysis method.
  • the discriminant analysis method is an approach for dividing histograms into two classes. Therefore, a description will be given with reference to FIG. 24 in addition to FIG. 23 .
  • FIG. 24 is a drawing illustrating a probability distribution of a ratio of noise.
  • the application processor 220 sets the number m (m represents an integer of 1 or more) of glossy objects (hereinafter glossy objects) to 1. Also, the application processor 220 sets the number x 1 of glossy region data groups to 0 (zero) (step S 101 A). This setting is a preparation for acquiring a color image of the first glossy object.
  • step S 102 A executes the same processing as that in step S 102 A through step S 107 A of the first embodiment.
  • step S 208 A Upon the glossy region data being employed in step S 107 A, the number x 1 of glossy region data groups is incremented by the application processor 220 (step S 208 A).
  • the application processor 220 saves the glossy region data employed in step S 107 A in the memory 250 (step S 108 A).
  • the application processor 220 determines whether the number x 1 of glossy region data groups reaches a predetermined number x 2 (step S 209 A).
  • the predetermined number x 2 which has been preliminarily set, is the necessary number of glossy region data groups.
  • the predetermined number x 2 of glossy region data groups may be determined and set by the user or may be preliminarily set by the electronic device 100 .
  • step S 101 B When the application processor 220 determines that the number x 1 of glossy region data groups reaches the predetermined number x 2 (YES in S 209 A), the flow proceeds to step S 101 B.
  • the flow returns to step S 106 A.
  • the processing is repeatedly performed until the number x 1 of glossy region data groups reaches the predetermined number x 2 .
  • the application processor 220 sets the number n (n represents an integer of 1 or more) of non-glossy objects (hereinafter referred to as non-glossy objects) to 1.
  • the application processor 220 also sets the number y 1 of non-glossy region data groups to 0 (zero) (step S 101 B). This setting is a preparation for acquiring a color image of the first non-glossy object.
  • step S 102 B executes the same processing as that in step S 102 B through step S 107 B of the first embodiment.
  • the number y 1 of non-glossy region data groups is incremented by the application processor 220 (step S 208 B).
  • the application processor 220 saves the glossy region data employed in step S 107 B in the memory 250 (step S 108 B).
  • the application processor 220 determines whether the number y 1 of non-glossy region data groups reaches a predetermined number y 2 (step S 209 B).
  • the predetermined number y 2 which has been preliminarily set, is the necessary number of pieces of non-glossy region data groups.
  • the predetermined number y 2 of non-glossy region data groups may be determined and set by the user or may be preliminarily set by the electronic device 100 .
  • step S 210 A When the application processor 220 determines that the number y 1 of non-glossy region data groups reaches the predetermined number y 2 (YES in S 209 B), the flow proceeds to step S 210 A.
  • the application processor 220 determines that the number y 1 of non-glossy region data groups does not reach the predetermined number y 2 (NO in S 209 B)
  • the flow returns to step S 160 B.
  • the processing is repeatedly performed until the number y 1 of glossy region data groups reaches the predetermined number y 2 .
  • step S 209 B the application processor 220 creates a probability distribution of the ratio of noise and obtains a degree of separation ⁇ (step S 210 A).
  • the application processor 220 sets a temporary threshold Th by using a discriminant analysis method as illustrated in FIG. 24 . Subsequently, the application processor 220 calculates the number ⁇ 1 of non-glossy region data samples, the mean ml of ratios of noise, the variance ⁇ 1 of ratios of noise, the number ⁇ 2 of glossy region data samples, the mean m 2 of ratios of noise, and the variance ⁇ 2 of ratios of noise.
  • a plurality of data groups employed as glossy region data is referred to as a glossy region data class.
  • a plurality of data groups employed as non-glossy region data is referred to as a non-glossy region data class.
  • the application processor 220 calculates intra-class variance and inter-class variance by using formulas (4) and (5). Subsequently, based on the intra-class variance and the inter-class variance, the application processor 220 calculates the degree of separation ⁇ by using a formula (6).
  • the application processor 220 repeatedly calculates the degree of separation ⁇ by setting different values as a temporary threshold Th.
  • the application processor 220 determines the temporary threshold Th that maximizes the degree of separation ⁇ as a threshold used in step S 100 (step S 210 B).
  • the threshold used in step S 100 can be determined.
  • the mode method is an approach for dividing histograms into two classes, similarly to the discriminant analysis method.
  • step S 210 B When the mode method is used, the following processing may be performed in place of step S 210 B.
  • FIG. 25 is a drawing illustrating a method for determining a threshold by using the mode method
  • a minimum value between the maximum value 1 and the maximum value 2 is searched.
  • a point that corresponds to the minimum value is determined to be the threshold used in step S 100 .
  • a method of acquiring an image of the specific region differs from that of step S 130 of the first embodiment.
  • Other configurations are similar to those of the electronic device 100 of the first embodiment. Therefore, the same reference numerals are given to the similar configuration elements and thus their descriptions are omitted.
  • FIG. 26 is a flowchart illustrating a method for acquiring an image of a specific region according to a third embodiment.
  • FIGS. 27A through 27D are drawings illustrating image processing performed according to the flow illustrated in FIG. 26 .
  • the application processor 220 acquires a background image by using the camera 180 (step S 331 ).
  • a background image 8 A is acquired by including only the background in the field of view and photographing the background without an object 7 (see FIG. 27B ) being placed.
  • the application processor 220 acquires an image of the object 7 by using the camera 180 (step S 332 ).
  • an object image 8 B is acquired by including both the object 7 and the background in the field of view and photographing the object 7 and the background by using the camera 180 .
  • the application processor 220 acquires a differential image 8 C of the object 7 by subtracting a pixel value of the background image 8 A from a pixel value of the object image 8 B (step S 333 ).
  • the differential image 8 C of the object 7 is acquired by subtracting the pixel value of the background image 8 A from the pixel value of the object image 8 B.
  • the application processor 220 acquires an image 8 D of a specific region by binarizing the differential image 8 C (step S 334 ). As illustrated in FIG. 27D , in the image 8 D of the specific region, a display region 8 D 1 (white region) of the object 7 has the value “1.” A region 8 D 2 (black region) other than the display region 8 D 1 of the object 7 has the value “0.” The display region 8 D 1 is the specific region.
  • a threshold that is as close to “0” as possible may be used so that the image 8 C is divided into the display region 8 D 1 , which has a pixel value, and the region 8 D 2 , which does not have a pixel value.
  • FIG. 28 is a side view illustrating an electronic device 400 of a fourth embodiment.
  • the side view illustrated in FIG. 28 corresponds to the side view illustrated in FIG. 3 .
  • the electronic device 400 of the fourth embodiment provides a tactile sensation by using a transparent electrode plate 410 disposed between the top panel 120 and the touch panel 150 , instead of providing a tactile sensation by using the vibrating element 140 as with the electronic device 100 of the first embodiment.
  • a surface opposite to the surface 120 A of the top panel 120 is an insulating surface. If the top panel 120 is a glass plate, an insulation coating may be formed on the surface opposite to the surface 120 A.
  • a voltage is applied to the electrode plate 410 when the position of the manipulation input is located outside the specific region and the position of the manipulation is in motion.
  • Generating an electrostatic force by applying a voltage to the electrode plate 410 causes a friction force applied to the user's fingertip to increase, compared to when no electrostatic force is generated.
  • an electronic device and a drive controlling method are provided in which a tactile sensation based on the presence or absence of gloss can be provided.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Acoustics & Sound (AREA)
  • User Interface Of Digital Computer (AREA)
US15/828,056 2015-06-25 2017-11-30 Electronic device and drive controlling method Abandoned US20180088698A1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2015/068370 WO2016208036A1 (ja) 2015-06-25 2015-06-25 電子機器、及び、駆動制御方法

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2015/068370 Continuation WO2016208036A1 (ja) 2015-06-25 2015-06-25 電子機器、及び、駆動制御方法

Publications (1)

Publication Number Publication Date
US20180088698A1 true US20180088698A1 (en) 2018-03-29

Family

ID=57585268

Family Applications (1)

Application Number Title Priority Date Filing Date
US15/828,056 Abandoned US20180088698A1 (en) 2015-06-25 2017-11-30 Electronic device and drive controlling method

Country Status (4)

Country Link
US (1) US20180088698A1 (zh)
JP (1) JP6500986B2 (zh)
CN (1) CN107710114A (zh)
WO (1) WO2016208036A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180143690A1 (en) * 2016-11-21 2018-05-24 Electronics And Telecommunications Research Institute Method and apparatus for generating tactile sensation
CN110007841A (zh) * 2019-03-29 2019-07-12 联想(北京)有限公司 一种控制方法和电子设备
US10762752B1 (en) 2017-09-06 2020-09-01 Apple Inc. Tactile notifications for electronic devices

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7054794B2 (ja) * 2019-12-09 2022-04-15 パナソニックＩｐマネジメント株式会社 入力装置

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130222303A1 (en) * 2006-03-24 2013-08-29 Northwestern University Haptic device with indirect haptic feedback

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06300543A (ja) * 1993-04-19 1994-10-28 Toshiba Eng Co Ltd 光沢物抽出装置
JP2003308152A (ja) * 2002-04-17 2003-10-31 Nippon Hoso Kyokai <Nhk> 触感提示装置
JP5541653B2 (ja) * 2009-04-23 2014-07-09 キヤノン株式会社 撮像装置及びその制御方法
CN103973971B (zh) * 2013-01-30 2017-10-13 奥林巴斯株式会社 信息设备以及信息设备的控制方法
US9158379B2 (en) * 2013-09-06 2015-10-13 Immersion Corporation Haptic warping system that transforms a haptic signal into a collection of vibrotactile haptic effect patterns
KR101550601B1 (ko) * 2013-09-25 2015-09-07 현대자동차 주식회사 촉감 피드백을 제공하는 곡면 터치 디스플레이 장치 및 그 방법
CN104662495B (zh) * 2013-09-26 2017-06-23 富士通株式会社 驱动控制装置、电子设备以及驱动控制方法
WO2015121971A1 (ja) * 2014-02-14 2015-08-20 富士通株式会社 触感提供装置、及び、システム
CN104199547B (zh) * 2014-08-29 2017-05-17 福州瑞芯微电子股份有限公司 一种虚拟触屏操作装置、系统及方法

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130222303A1 (en) * 2006-03-24 2013-08-29 Northwestern University Haptic device with indirect haptic feedback

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180143690A1 (en) * 2016-11-21 2018-05-24 Electronics And Telecommunications Research Institute Method and apparatus for generating tactile sensation
US10551925B2 (en) * 2016-11-21 2020-02-04 Electronics And Telecommunications Research Institute Method and apparatus for generating tactile sensation
US10762752B1 (en) 2017-09-06 2020-09-01 Apple Inc. Tactile notifications for electronic devices
US10977910B1 (en) * 2017-09-06 2021-04-13 Apple Inc. Tactile outputs for input structures of electronic devices
CN110007841A (zh) * 2019-03-29 2019-07-12 联想(北京)有限公司 一种控制方法和电子设备

Also Published As

Publication number Publication date
JPWO2016208036A1 (ja) 2018-03-29
WO2016208036A1 (ja) 2016-12-29
JP6500986B2 (ja) 2019-04-17
CN107710114A (zh) 2018-02-16

Similar Documents

Publication Publication Date Title
US20180088698A1 (en) Electronic device and drive controlling method
US9400571B2 (en) Drive controlling apparatus, electronic device and drive controlling method
US20220276713A1 (en) Touch Display Device with Tactile Feedback
CN105229582B (zh) 基于近距离传感器和图像传感器的手势检测
EP3095074B1 (en) 3d silhouette sensing system
US8787656B2 (en) Method and apparatus for feature-based stereo matching
CN111541907B (zh) 物品显示方法、装置、设备及存储介质
US20180288387A1 (en) Real-time capturing, processing, and rendering of data for enhanced viewing experiences
KR20140105985A (ko) 사용자 인터페이스 제공 방법 및 장치
CN112749613B (zh) 视频数据处理方法、装置、计算机设备及存储介质
US20130009891A1 (en) Image processing apparatus and control method thereof
CN107077285A (zh) 操作装置、具有操作装置的信息处理装置以及用于信息处理装置的操作受理方法
US11086435B2 (en) Drive control device, electronic device, and drive control method
Liu et al. Holoscopic 3D micro-gesture database for wearable device interaction
US10545576B2 (en) Electronic device and drive control method thereof
CN104732570B (zh) 一种图像生成方法及装置
JP6627603B2 (ja) 電子機器、及び、電子機器の駆動方法
WO2016178289A1 (ja) 電子機器及び振動制御プログラム
US20230377363A1 (en) Machine learning based multipage scanning
AU2015202408B2 (en) Drive controlling apparatus, electronic device and drive controlling method
JP6399216B2 (ja) 駆動制御装置、電子機器、駆動制御プログラム、及び駆動制御方法
KR20170042211A (ko) 디스플레이 방법 및 장치
Hettipathirana et al. iPanel: A Computer-Vision Based Solution for Interactive Keyboard and Mouse

Legal Events

Date Code Title Description
AS Assignment

Owner name: FUJITSU LIMITED, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SUZUKI, TATSUYA;KAMATA, YUICHI;SIGNING DATES FROM 20171108 TO 20171110;REEL/FRAME:044886/0436

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION