JP5380729B2 - Electronic device, display control method, and program - Google Patents

Description

  The present invention relates to an electronic device including a touch panel, a display control method in the electronic device, and a program for causing the electronic device to execute the display control method.

  Conventionally, an electronic device including a touch panel is known. Also, electronic devices equipped with a touch pad are known.

  As an electronic device including a touch panel, for example, Patent Document 1 discloses a device that detects the width of an object that touches the touch panel and switches processing for an operation input from the touch panel according to the detected width. ing.

  As an electronic device provided with a touch pad, for example, in Patent Document 2, when it is determined that a predetermined area of the touch pad is touched, thereafter, the input on the touch pad is set as a coordinate input for recognition. An apparatus is disclosed. Specifically, in Patent Document 2, when a predetermined area in the upper right corner that cannot be touched with a fingertip on a touch pad is touch-operated with a pen, the cursor is moved relative to the coordinate data of the touch position. An apparatus for switching from a normal input mode to a handwriting recognition mode is disclosed.

JP 2004-213312 A JP 2000-137571 A

  However, in the apparatus of Patent Document 1, since the process is switched depending on the detected width of the object, the process is always switched regardless of what type of object touches the touch panel. Therefore, there is a possibility that processing switching unintended for the user is performed. On the other hand, in the apparatus of Patent Document 2, in order to switch the operation mode of the apparatus to the handwriting recognition mode, the user needs to once contact the pen tip at a position different from the position where the handwriting is started.

  The present invention has been made in view of the above-described problems, and the purpose thereof is an electronic device capable of easily and reliably switching operation of an electronic device by a user, a display control method in the electronic device, The object is to provide a program for executing the display control method.

According to an aspect of the present invention, an electronic device is an electronic device including a touch panel, and stores a first shape data indicating a first shape and a second shape data indicating a second shape. A detection unit that detects that at least one object has touched the display surface of the touch panel, a specification unit that specifies the shape of the contact region for each contact region based on the detection, and a specification based on the first shape data In addition to determining whether there is a shape that matches the first shape among the determined shapes, there is a shape that matches the second shape among the specified shapes based on the second shape data The first determining means for determining whether or not to perform and different operations depending on whether or not there is a shape that matches the first shape based on the determination that a shape that matches the second shape exists. Let the electronics And a performing unit.

  Preferably, the electronic device determines whether there is a shape that matches the first shape and a shape that matches the second shape, according to the position of the contact area having the shape that matches the second shape. The apparatus further comprises second determination means for determining whether or not an input for designating an object displayed on the display surface has been received. The execution means causes the electronic device to execute different operations when it is determined that an input designating an object is received and when it is determined that the input is not received.

  Preferably, the contact region having a shape that matches the second shape is formed when the first object contacts the display surface. The electronic device determines whether or not an input for designating an object displayed on the display surface is received in accordance with the contact position of the first object based on the fact that there is no shape that matches the first shape. A judging means is further provided. If it is determined that an input for designating an object has been received, the execution means causes the electronic device to perform an operation based on the designation of the object.

  Preferably, the object is an icon indicating a file. The operation based on the designation of the object is an operation for setting the file in a selected state.

  Preferably, the contact region having a shape that matches the second shape is formed when the first object contacts the display surface. When it is determined that the input specifying the object is not received, the execution unit receives the handwriting input based on the movement of the first object on the display surface and displays the image based on the received handwriting input. Let the electronics run.

  Preferably, the contact region having a shape that matches the second shape is formed when the first object contacts the display surface. If it is determined that the input specifying the object is received, the execution means receives the handwritten input based on the movement of the first object on the display surface, displays the image based on the received handwritten input, and displays the displayed image. The electronic device is caused to execute association of the information based on the specified object with the designated object.

  Preferably, the electronic device further includes recognition means for recognizing the accepted handwritten input. When the electronic device receives an instruction to open the file when the object is an icon indicating the file, the execution unit performs a process of opening the file and an input process of the recognized character for the input area of the opened file. Let the electronics run.

  Preferably, the electronic device further includes recognition means for recognizing the accepted handwritten input. When the electronic device accepts an instruction to change the name of the object when the object is an icon indicating a file or an icon indicating a folder, the execution unit changes the name of the object using the recognized character Is executed by an electronic device.

  Preferably, when the contact position of the first object is a position where the object is displayed, the second determination unit determines that an input for designating the object has been received.

  Preferably, the contact region having a shape that matches the first shape is formed when the second object contacts the display surface. The first object is a stylus pen or a finger, and the second object is a palm of the hand.

Preferably, the touch panel includes a plurality of optical sensors built in along the display surface.
According to another aspect of the present invention, the display control method is a display control method in an electronic device provided with a touch panel, the step of detecting that at least one object has touched the display surface of the touch panel, and the touch detection. If so, based on the step of specifying the shape of the contact area for each contact area and the first shape data indicating the first shape, is there a shape that matches the first shape among the specified shapes? A step of determining whether or not there is a shape that matches the second shape among the specified shapes based on the second shape data indicating the second shape, and a second shape And a step of causing the electronic device to perform different operations depending on whether or not there is a shape that matches the first shape.

  According to still another aspect of the present invention, the program is a program for controlling an electronic device including a touch panel, and the program detects that at least one object has come into contact with the display surface of the touch panel. Step, if contact is detected, identifying the shape of the contact area for each contact area, and matching the first shape in each identified shape based on the first shape data indicating the first shape Determining whether there is a shape to match, and whether there is a shape that matches the second shape among the specified shapes, based on the second shape data indicating the second shape. And a step of executing different operations depending on whether or not there is a shape that matches the first shape based on the determination that a shape that matches the second shape exists. To the execution.

  There is an effect that the user can easily and surely perform the operation switching operation of the electronic device.

It is the figure which showed the external appearance of the electronic device. It is a block diagram showing the hardware constitutions of an electronic device. It is the figure which showed the structure of the liquid crystal panel of an electronic device, and the peripheral circuit of the said liquid crystal panel. It is sectional drawing of a liquid crystal panel and a backlight. It is the figure which showed the timing chart at the time of operating an optical sensor circuit. It is sectional drawing of a liquid crystal panel and a backlight, Comprising: It is the figure which showed the structure which a photodiode receives the light from a backlight in the case of a scan. It is the figure which showed schematic structure of the sensing command. It is the figure which showed the value of the data in each area | region of a sensing command, and the meaning content which the said value shows. It is the figure which showed schematic structure of the response data. It is the figure which showed the image obtained by scanning a finger | toe. It is a circuit diagram of another liquid crystal panel with a built-in photosensor. It is sectional drawing which showed the structure which a photodiode receives external light in the case of a scan. It is a block diagram showing the hardware constitutions of another electronic device. It is the figure which showed the functional block of an electronic device. It is the figure which showed the scan image at the time of scanning the nib of a stylus pen. It is the figure which showed the scan image at the time of scanning the nib of a stylus pen, and the palm of a hand. It is the flowchart which showed the processing flow in an electronic device. It is the figure which showed an example of the display content on the display surface of a liquid crystal panel. It is the figure which showed the state in which the user made the nib tip of the stylus pen contact the display surface of the liquid crystal panel. It is the figure which showed the state just before a user makes the nib of a stylus pen, and the palm of a hand contact the display surface of a liquid crystal panel. It is the figure which showed the display content of the display surface when an electronic device sets operation mode to the 2nd operation mode. It is the figure which showed the state after a user performed handwriting input using the stylus pen in the state of FIG. It is the figure which showed the display content of the display surface after performing the process of character recognition in the state of FIG. It is the figure which showed the state just before a user makes the nib of a stylus pen, and the palm of a hand contact the display surface of a liquid crystal panel. It is the figure which showed the display content of the display surface when an electronic device sets operation mode to the 3rd operation mode. FIG. 26 is a diagram showing a state immediately before one item is selected by the stylus pen in the case shown in FIG. 25. It is the figure which showed the state after one item was selected with the tyrus pen. FIG. 28 is a diagram illustrating a state after the user performs handwriting input using the stylus pen in the case illustrated in FIG. 27. It is the figure which showed the display content of the display surface after performing the process of character recognition in the case shown by FIG. FIG. 30 is a diagram illustrating a state after the electronic device receives an instruction to input a recognized character at the cursor display position of the image from the user in the case illustrated in FIG. 29. It is the figure which showed the state just before a user makes the nib of a stylus pen, and the palm of a hand contact the display surface of a liquid crystal panel. It is the figure which showed the display content of the display surface when an electronic device sets operation mode to the 3rd operation mode. FIG. 33 is a diagram showing a state immediately before another item is selected by the stylus pen in the case shown in FIG. 32. It is the figure which showed the state after another item was selected with the stylus pen. In the case shown in FIG. 34, it is the figure which showed the state after a user performed handwritten input using a stylus pen. It is the figure which showed the display content of the display surface after performing the process of character recognition in the case shown by FIG. FIG. 37 is a diagram showing a state after the electronic device has received an input instruction for changing the folder name to a recognized character from the user in the case shown in FIG. 36. It is the figure which showed the state just before designating an icon with the fingertip of a left hand, after making the nib of a stylus pen, and the palm of a right hand contact a display surface.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  An embodiment of the electronic apparatus according to the present invention will be described below with reference to FIGS.

<Appearance of electronic equipment>
FIG. 1 is a diagram illustrating an appearance of electronic device 100 according to the present embodiment. Referring to FIG. 1, electronic device 100 includes a first housing 100A and a second housing 100B.

The first casing 100A and the second casing 100B are foldably connected by a hinge 100C. The first housing 100A includes a liquid crystal panel 140 with a built-in optical sensor. The second housing 100B includes an optical sensor built-in liquid crystal panel 240. As described above, the electronic device 100 includes two optical sensor built-in liquid crystal panels.

  The electronic device 100 is configured as a portable device having a display function such as a PDA (Personal Digital Assistant), a notebook personal computer, a portable telephone, and an electronic dictionary.

<About hardware configuration>
Next, an aspect of a specific configuration of the electronic device 100 will be described with reference to FIG. FIG. 2 is a block diagram illustrating a hardware configuration of electronic device 100. The electronic device 100 includes a main body device 101, a display device 102, and a display device 103 as main components.

  The first housing 100A includes a display device 102. The second housing 100B includes a main body device 101 and a display device 103. Note that the electronic device 100 may include the main body device 101 in the first housing 100A instead of the second housing 100B.

  The main unit 101 includes a CPU (Central Processing Unit) 110, a RAM (Random Access Memory) 171, a ROM (Read-Only Memory) 172, a memory card reader / writer 173, a communication unit 174, a microphone 175, and a speaker. 176 and operation keys 177 are included. Each component is connected to each other by a data bus DB1. A memory card 1731 is attached to the memory card reader / writer 173.

  CPU 110 executes a program. The operation key 177 receives an instruction input from the user of the electronic device 100. The RAM 171 stores data generated by the execution of the program by the CPU 110 or data input via the operation keys 177 in a volatile manner. The ROM 172 stores data in a nonvolatile manner. The ROM 172 is a ROM capable of writing and erasing data such as an EPROM (Erasable Programmable Read-Only Memory) and a flash memory. The communication unit 174 performs wireless communication with other electronic devices (not shown). Although not illustrated in FIG. 2, the electronic device 100 may include an interface (IF) for connecting to another electronic device by wire.

  The display device 102 includes a driver 130, an optical sensor built-in liquid crystal panel 140 (hereinafter referred to as a liquid crystal panel 140), an internal IF 178, a backlight 179, and an image processing engine 180.

  The driver 130 is a drive circuit for driving the liquid crystal panel 140 and the backlight 179. Various drive circuits included in the driver 130 will be described later.

  The liquid crystal panel 140 is a device having a liquid crystal display function and a photosensor function. That is, the liquid crystal panel 140 can perform image display using liquid crystal and sensing using an optical sensor. Details of the liquid crystal panel 140 will be described later.

  An internal IF (Interface) 178 mediates exchange of data between the main device 101 and the display device 102.

  The backlight 179 is a light source disposed on the back surface of the liquid crystal panel 140. The backlight 179 irradiates the back surface with uniform light.

The image processing engine 180 controls the operation of the liquid crystal panel 140 via the driver 130. Here, the control is performed based on various data sent from the main apparatus 101 via the internal IF 178. Note that the various data includes commands to be described later. Further, the image processing engine 180 processes data output from the liquid crystal panel 140 and sends the processed data to the main apparatus 101 via the internal IF 178. Further, the image processing engine 180 includes a driver control unit 181, a timer 182, and a signal processing unit 183.

  The driver control unit 181 controls the operation of the driver 130 by sending a control signal to the driver 130. In addition, the driver control unit 181 analyzes a command transmitted from the main device 101. Then, the driver control unit 181 sends a control signal based on the analysis result to the driver 130. Details of the operation of the driver 130 will be described later.

The timer 182 generates time information and sends the time information to the signal processing unit 183.
The signal processing unit 183 receives data output from the optical sensor. Here, since the data output from the optical sensor is analog data, the signal processing unit 183 first converts the analog data into digital data. Further, the signal processing unit 183 performs data processing on the digital data in accordance with the content of the command sent from the main device 101. Then, the signal processing unit 183 sends data (hereinafter referred to as response data) including data after the above data processing and time information acquired from the timer 182 to the main unit 101. The signal processing unit 183 includes a RAM (not shown) that can store a plurality of scan data, which will be described later, continuously.

  The command includes a sensing command for instructing sensing by the optical sensor. Details of the sensing command and the response data will be described later (FIGS. 7 to 9).

  Note that the timer 182 is not necessarily provided in the image processing engine 180. For example, the timer 182 may be provided outside the image processing engine 180 in the display device 102. Alternatively, the timer 182 may be provided in the main body device 101. Further, the microphone 175 and the speaker 176 are not always provided in the electronic device 100, and may be configured so as not to include either or both of the microphone 175 and the speaker 176 depending on the embodiment of the electronic device 100.

  Here, the display device 102 includes a system liquid crystal. The system liquid crystal is a device obtained by integrally forming peripheral devices of the liquid crystal panel 140 on the glass substrate of the liquid crystal panel 140. In the present embodiment, the driver 130 (excluding the circuit that drives the backlight 179), the internal IF 178, and the image processing engine 180 are integrally formed on the glass substrate of the liquid crystal panel 140. Note that the display device 102 is not necessarily configured using the system liquid crystal, and the driver 130 (excluding a circuit that drives the backlight 179), the internal IF 178, and the image processing engine 180 are included in the glass substrate. Other substrates may be configured.

  The display device 103 includes a driver 230, an optical sensor built-in liquid crystal panel 240 (hereinafter referred to as “liquid crystal panel 240”), an internal IF 278, a backlight 279, and an image processing engine 280. The image processing engine 280 includes a driver control unit 281, a timer 282, and a signal processing unit 283.

The display device 103 has the same configuration as the display device 102. That is, the driver 230, the liquid crystal panel 240, the internal IF 278, the backlight 279, and the image processing engine 280 have the same configuration as the driver 130, the liquid crystal panel 140, the internal IF 178, the backlight 179, and the image processing engine 180 in the display device 102. Have each. The driver control unit 281, the timer 282, and the signal processing unit 283 have the same configurations as the driver control unit 181, the timer 182, and the signal processing unit 183 in the display device 102, respectively. Therefore, description of each functional block included in display device 103 will not be repeated.

  By the way, the processing in the electronic device 100 is realized by each hardware and software executed by the CPU 110. Such software may be stored in the ROM 172 in advance. The software may be stored in a memory card 1731 or other storage medium and distributed as a program product. Alternatively, the software may be provided as a program product that can be downloaded by an information provider connected to the so-called Internet. Such software is read from the storage medium by the memory card reader / writer 173 or other reading device, or downloaded via the communication unit 174 or communication IF (not shown), and then temporarily stored in the ROM 172. The The software is read from the ROM 172 by the CPU 110 and stored in the RAM 171 in the form of an executable program. CPU 110 executes the program.

  Each component constituting the main device 101 of the electronic device 100 shown in FIG. 2 is a general one. Therefore, it can be said that the essential part of the present invention is the software stored in the RAM 171, the ROM 172, the memory card 1731 and other storage media, or the software downloadable via the network. Since the hardware operation of main device 101 of electronic device 100 is well known, detailed description will not be repeated.

  The storage medium is not limited to a memory card, but is a CD-ROM, FD (Flexible Disk), hard disk, magnetic tape, cassette tape, optical disk (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile). Disc)), IC (Integrated Circuit) cards (excluding memory cards), optical cards, mask ROM, EPROM, EEPROM (Electronically Erasable Programmable Read-Only Memory), flash ROM, and other semiconductor memories, etc. It may be a medium to be used.

  The program here includes not only a program directly executable by the CPU but also a program in a source program format, a compressed program, an encrypted program, and the like.

<Configuration and drive of liquid crystal panel with built-in optical sensor>
Next, the configuration of the liquid crystal panel 140 and the configuration of peripheral circuits of the liquid crystal panel 140 will be described. FIG. 3 is a diagram showing a configuration of the liquid crystal panel 140 and peripheral circuits of the liquid crystal panel 140.

  Referring to FIG. 3, the liquid crystal panel 140 includes a pixel circuit 141, an optical sensor circuit 144, a scanning signal line Gi, a data signal line SRj, a data signal line SGj, a data signal line SBj, and a sensor signal line. SSj, sensor signal line SDj, read signal line RWi, and reset signal line RSi are included. Note that i is a natural number satisfying 1 ≦ i ≦ m, and j is a natural number satisfying 1 ≦ j ≦ n.

  2 includes a scanning signal line driving circuit 131, a data signal line driving circuit 132, an optical sensor driving circuit 133, a switch 134, as peripheral circuits of the liquid crystal panel 140. And an amplifier 135.

The scanning signal line drive circuit 131 receives the control signal TC1 from the driver control unit 181 shown in FIG. The scanning signal line drive circuit 131 applies a predetermined voltage in order from the scanning signal line G1 to each scanning signal line (G1 to Gm) based on the control signal TC1. More specifically, the scanning signal line driving circuit 131 sequentially selects one scanning signal line from the scanning signal lines (G1 to Gm) per unit time, and a TFT (to be described later) with respect to the selected scanning signal line. A voltage capable of turning on the gate of the thin film transistor 142 is applied (hereinafter referred to as a high level voltage). Note that a low level voltage is applied to a scanning signal line that is not selected without applying a high level voltage.

  The data signal line driving circuit 132 receives image data (DR, DG, DB) from the driver control unit 181 shown in FIG. The data signal line driving circuit 132 applies a voltage corresponding to one row of image data to the 3n data signal lines (SR1 to SRn, SG1 to SGn, SB1 to SBn) for each unit time. Apply sequentially.

  Note that although a driving method called a so-called line-sequential method has been described here, the driving method is not limited to this.

  The pixel circuit 141 is a circuit for setting the luminance (transmittance) of one pixel. Further, m × n pixel circuits 141 are arranged in a matrix. More specifically, m pixel circuits 141 are arranged in the vertical direction in FIG. 3 and n pixel circuits in the horizontal direction.

  The pixel circuit 141 includes an R subpixel circuit 141r, a G subpixel circuit 141g, and a B subpixel circuit 141b. Each of these three circuits (141r, 141g, 141b) includes a TFT 142, a pair of electrode pairs 143 including a pixel electrode and a counter electrode, and a capacitor (not shown).

  In addition, CMOS (Complementary Metal Oxide Semiconductor) capable of forming n-type transistors and p-type transistors can be realized, and the movement speed of carriers (electrons or holes) is several hundreds compared to amorphous silicon thin film transistors (a-Si TFTs). For example, a polycrystalline silicon thin film transistor (p-Si TFT) is used as the TFT 142 in the display device 102 because it is twice as fast. Note that the TFT 142 will be described as an n-channel field effect transistor. However, the TFT 142 may be a p-type channel field effect transistor.

  The source of the TFT 142 in the R subpixel circuit 141r is connected to the data signal line SRj. The gate of the TFT 142 is connected to the scanning signal line Gi. Further, the drain of the TFT 142 is connected to the pixel electrode of the electrode pair 143. A liquid crystal is disposed between the pixel electrode and the counter electrode. The G sub-pixel circuit 141g and the B sub-pixel circuit 141b have the same configuration as the R sub-pixel circuit 141r except that the data signal line to which the source of each TFT 142 is connected is different. Therefore, description of these two circuits (141g, 141b) will not be repeated.

  Here, setting of luminance in the pixel circuit 141 will be described. First, the high level voltage is applied to the scanning signal line Gi. By the application of the high level voltage, the gate of the TFT 142 is turned on. In this manner, with the gate of the TFT 142 turned on, a specified voltage (voltage corresponding to image data for one pixel) is applied to each data signal line (SRj, SGj, SBj). Thereby, a voltage based on the designated voltage is applied to the pixel electrode. As a result, a potential difference is generated between the pixel electrode and the counter electrode. Based on this potential difference, the liquid crystal responds and the luminance of the pixel is set to a predetermined luminance. Note that the potential difference is held by the capacitor (auxiliary capacitor) (not shown) until the scanning signal line Gi is selected in the next frame period.

  The optical sensor drive circuit 133 receives the control signal TC2 from the driver control unit 181 shown in FIG.

  Then, the optical sensor drive circuit 133 sequentially selects one signal line from the reset signal lines (RS1 to RSm) for each unit time based on the control signal TC2, and determines in advance for the selected signal line. At a given timing, the voltage VDDR that is higher than usual is applied. Note that a voltage VSSR lower than the voltage applied to the selected reset signal line is kept applied to the unselected reset signal line. For example, the voltage VDDR may be set to 0V and the voltage VSSR may be set to −5V.

  Further, the optical sensor driving circuit 133 sequentially selects one signal line from the readout signal lines (RW1 to RWm) for each unit time based on the control signal TC2, and determines in advance for the selected signal line. At a given timing, a voltage VDD higher than usual is applied. Note that the voltage VSSR is applied to the read signal line that is not selected. For example, the value of VDD may be set to 8V.

  The timing for applying the voltage VDDR and the timing for applying the voltage VDD will be described later.

  The optical sensor circuit 144 includes a photodiode 145, a capacitor 146, and a TFT 147. In the following description, it is assumed that the TFT 147 is an n-type channel field effect transistor. However, the TFT 147 may be a p-type channel field effect transistor.

  The anode of the photodiode 145 is connected to the reset signal line RSi. On the other hand, the cathode of the photodiode 145 is connected to one electrode of the capacitor 146. The other electrode of the capacitor 146 is connected to the read signal line RWi. Hereinafter, a connection point between the photodiode 145 and the capacitor 146 is referred to as a node N.

  The gate of the TFT 147 is connected to the node N. The drain of the TFT 147 is connected to the sensor signal line SDj. Further, the source of the TFT 147 is connected to the sensor signal line SSj. Details of sensing using the optical sensor circuit 144 will be described later.

  The switch 134 is a switch provided to switch whether or not to apply a predetermined voltage to the sensor signal lines (SD1 to SDn). The switching operation of the switch 134 is performed by the optical sensor driving circuit 133. Note that the voltage applied to the sensor signal lines (SD1 to SDn) when the switch 134 is turned on will be described later.

  The amplifier 135 amplifies the voltage output from each sensor signal line (SS1 to SSn). The amplified voltage is sent to the signal processing unit 183 shown in FIG.

  The image processing engine 180 controls the timing for displaying an image on the liquid crystal panel 140 using the pixel circuit 141 and the timing for sensing using the optical sensor circuit 144.

FIG. 4 is a cross-sectional view of the liquid crystal panel 140 and the backlight 179. Referring to FIG. 4, liquid crystal panel 140 includes an active matrix substrate 151A, a counter substrate 151B, and a liquid crystal layer 152. The counter substrate 151B is disposed to face the active matrix substrate 151A. The liquid crystal layer 152 is sandwiched between the active matrix substrate 151A and the counter substrate 151B. The backlight 179 is disposed on the opposite side of the liquid crystal layer 152 with respect to the active matrix substrate 151A.

  The active matrix substrate 151 </ b> A includes a polarizing filter 161, a glass substrate 162, a pixel electrode 143 a that constitutes the electrode pair 143, a photodiode 145, a data signal line 157, and an alignment film 164. Further, although not shown in FIG. 4, the active matrix substrate 151A includes the capacitor 146, the TFT 147, the TFT 142, and the scanning signal line Gi shown in FIG.

  In the active matrix substrate 151A, the polarizing filter 161, the glass substrate 162, the pixel electrode 143a, and the alignment film 164 are arranged in this order from the backlight 179 side. The photodiode 145 and the data signal line 157 are formed on the liquid crystal layer 152 side of the glass substrate 162.

  The counter substrate 151B includes a polarizing filter 161, a glass substrate 162, a light shielding film 163, color filters (153r, 153g, 153b), a counter electrode 143b constituting the electrode pair 143, and an alignment film 164.

  In the counter substrate 151B, the alignment film 164, the counter electrode 143b, the color filters (153r, 153g, 153b), the glass substrate 162, and the polarizing filter 161 are arranged in this order from the liquid crystal layer 152 side. The light shielding film 163 is formed in the same layer as the color filters (153r, 153g, 153b).

  The color filter 153r is a filter that transmits light having a red wavelength. The color filter 153g is a filter that transmits light having a green wavelength. The color filter 153b is a filter that transmits light having a blue wavelength. Here, the photodiode 145 is arranged at a position facing the color filter 153b.

  The liquid crystal panel 140 displays an image by blocking external light or light emitted from a light source such as a backlight 179 or transmitting the light. Specifically, the liquid crystal panel 140 changes the direction of liquid crystal molecules in the liquid crystal layer 152 by applying a voltage between the pixel electrode 143a and the counter electrode 143b, thereby blocking or transmitting the light. However, since the light cannot be completely blocked by the liquid crystal alone, a polarizing filter 161 that transmits only light having a specific polarization direction is provided.

  Note that the position of the photodiode 145 is not limited to the above position, and may be provided at a position facing the color filter 153r or a position facing the color filter 153g.

  Here, the operation of the optical sensor circuit 144 will be described. FIG. 5 is a diagram showing a timing chart when the optical sensor circuit 144 is operated. In FIG. 5, a voltage VINT indicates a potential at the node N in the photosensor circuit 144. A voltage VPIX is an output voltage from the sensor signal line SSj shown in FIG. 3 and is a voltage before being amplified by the amplifier 135.

  The following description will be divided into a reset period for resetting the optical sensor circuit 144, a sensing period for sensing light using the optical sensor circuit 144, and a readout period for reading the sensed result.

  First, the reset period will be described. In the reset period, the voltage applied to the reset signal line RSi is instantaneously switched from the low level (voltage VSSR) to the high level (voltage VDDR). On the other hand, the voltage applied to the read signal line RWi is kept at the low level (voltage VSSR). As described above, by applying the high-level voltage to the reset signal line RSi, a current starts to flow in the forward direction (from the anode side to the cathode side) of the photodiode 145. As a result, the voltage VINT which is the potential of the node N has a value represented by the following expression (1). In Equation (1), the forward voltage drop amount in the photodiode 145 is Vf.

VINT = VSSR + | VDDR−VSSR | −Vf (1)
Therefore, the potential of the node N is a value smaller by Vf than the voltage VDDR as shown in FIG.

  Here, since the voltage VINT is not more than the threshold value for turning on the gate of the TFT 147, there is no output from the sensor signal line SSj. For this reason, the voltage VPIX does not change. Further, a difference corresponding to the voltage VINT occurs between the electrodes of the capacitor 146. For this reason, the capacitor 146 accumulates charges corresponding to the difference.

  Next, the sensing period will be described. In the sensing period following the reset period, the voltage applied to the reset signal line RSi instantaneously switches from the high level (voltage VDDR) to the low level (voltage VSSR). On the other hand, the voltage applied to the read signal line RWi is kept at the low level (voltage VSSR).

  Thus, by changing the voltage applied to the reset signal line RSi to the low level, the potential of the node N becomes higher than the voltage of the reset signal line RSi and the voltage of the read signal line RWi. For this reason, in the photodiode 145, the voltage on the cathode side becomes higher than the voltage on the anode side. That is, the photodiode 145 is in a reverse bias state. In such a reverse bias state, when the photodiode 145 receives light from the light source, current starts to flow from the cathode side to the anode side of the photodiode 145. As a result, as shown in FIG. 5, the potential of the node N (that is, the voltage VINT) becomes lower with the passage of time.

  Since the voltage VINT continues to decrease in this way, the gate of the TFT 147 does not turn on. Therefore, there is no output from the sensor signal line SSj. For this reason, the voltage VPIX does not change.

  Next, the reading period will be described. In the readout period following the sensing period, the voltage applied to the reset signal line RSi is kept at the low level (voltage VSSR). On the other hand, the voltage applied to the read signal line RWi is instantaneously switched from the low level (voltage VSSR) to the high level (voltage VDD). Here, the voltage VDD is higher than the voltage VDDR.

  Thus, by applying a high level voltage instantaneously to the read signal line RWi, the potential of the node N is raised through the capacitor 146 as shown in FIG. Note that the increase width of the potential of the node N is a value corresponding to the voltage applied to the read signal line RWi. Here, since the potential of the node N (that is, the voltage VINT) is raised to a threshold value that turns on the gate of the TFT 147, the gate of the TFT 147 is turned on.

At this time, if a constant voltage is applied in advance to the sensor signal line SDj (see FIG. 3) connected to the drain side of the TFT 147, the sensor signal line SSj connected to the source side of the TFT 147.
From FIG. 5, a voltage corresponding to the potential of the node N is output as shown in the VPIX graph of FIG.

  Here, when the amount of light received by the photodiode 145 (hereinafter referred to as the amount of received light) is small, the slope of the straight line shown in the VINT graph of FIG. 5 becomes gentle. As a result, the voltage VPIX is higher than when the amount of received light is large. As described above, the optical sensor circuit 144 changes the value of the voltage output to the sensor signal line SSj in accordance with the amount of light received by the photodiode 145.

  By the way, in the above, the operation | movement was demonstrated paying attention to one optical sensor circuit 144 among the m * n optical sensor circuits which exist. Below, operation | movement of each photosensor circuit in the liquid crystal panel 140 is demonstrated.

  First, the optical sensor drive circuit 133 applies a predetermined voltage to all n sensor signal lines (SD1 to SDn). Next, the photosensor drive circuit 133 applies a voltage VDDR that is higher than normal to the reset signal line RS1. The other reset signal lines (RS2 to RSm) and read signal lines (RW1 to RWm) are kept in a state where a low level voltage is applied. As a result, the n photosensor circuits in the first row in FIG. 3 enter the reset period described above. Thereafter, the n photosensor circuits in the first row enter a sensing period. Further, thereafter, the n photosensor circuits in the first row enter a reading period.

  Note that the timing at which a predetermined voltage is applied to all n sensor signal lines (SD1 to SDn) is not limited to the above timing, and may be any timing that is applied at least before the readout period. .

  When the readout period of the n photosensor circuits in the first row is completed, the photosensor drive circuit 133 applies a voltage VDDR that is higher than usual to the reset signal line RS2. That is, the reset period of the n photosensor circuits in the second row starts. When the reset period ends, the n photosensor circuits in the second row enter a sensing period, and thereafter enter a reading period.

  Thereafter, the processing described above is sequentially performed on the n photosensor circuits in the third row, the n photosensor circuits in the fourth row,..., The n photosensor circuits in the m row. As a result, the sensing result of the first row, the sensing result of the second row,..., The sensing result of the m-th row are output in this order from the sensor signal lines (SS1 to SSn).

  In the display device 102, sensing is performed for each row as described above, and a sensing result is output from the liquid crystal panel 140 for each row. For this reason, hereinafter, the data after the signal processing unit 183 performs the above-described data processing on the data regarding the voltage for m rows from the first row to the m-th row output from the liquid crystal panel 140, This is called “scan data”. That is, scan data refers to image data obtained by scanning a scan target (for example, a user's finger). An image displayed based on the scan data is referred to as a “scanned image”. Further, in the following, sensing is referred to as “scan”.

  Moreover, in the above, although the structure which scans using all the m * n photosensor circuits was mentioned as an example, it is not limited to this. Scanning may be performed on a partial region of the surface of the liquid crystal panel 140 using a photosensor circuit selected in advance.

In the following, it is assumed that the electronic device 100 can take either of the two configurations. Further, switching between the components is performed by a command sent from the main body apparatus 101 based on an input via the operation key 177 or the like. Note that when scanning is performed on a partial area on the surface of the liquid crystal panel 140, the image processing engine 180 sets a scan target area. The setting of the area may be configured to be specified by the user via the operation key 177.

  As described above, when scanning is performed on a partial region of the surface of the liquid crystal panel 140, there are the following modes of use for displaying an image. The first is a mode in which an image is displayed in a surface area other than the partial area (hereinafter referred to as a scan area). The second is a mode in which no image is displayed in the surface area other than the scan area. Which mode is used is based on a command sent from the main apparatus 101 to the image processing engine 180.

  FIG. 6 is a cross-sectional view of the liquid crystal panel 140 and the backlight 179, showing a configuration in which the photodiode 145 receives light from the backlight 179 during scanning.

  Referring to FIG. 6, when the user's finger 900 is in contact with the surface of the liquid crystal panel 140, a part of the light emitted from the backlight 179 is part of the user's finger 900 (abbreviated in the contacted area). Reflected on the plane). The photodiode 145 receives the reflected light.

  Even in a region where the finger 900 is not in contact, part of the light emitted from the backlight 179 is reflected by the user's finger 900. Even in this case, the photodiode 145 receives the reflected light. However, since the finger 900 is not in contact with the surface of the liquid crystal panel 140 in the region, the amount of light received by the photodiode 145 is smaller than the region in which the finger 900 is in contact. Note that the photodiode 145 cannot receive most of the light emitted from the backlight 179 that does not reach the user's finger 900.

  Here, by turning on the backlight 179 at least during the sensing period, the optical sensor circuit 144 can output a voltage corresponding to the amount of light reflected by the user's finger 900 from the sensor signal line SSj. it can. In this manner, by controlling the backlight 179 to be turned on and off, the liquid crystal panel 140 has a contact position of the finger 900, a range in which the finger 900 is in contact (determined by the pressing force of the finger 900), and the liquid crystal panel 140. The voltage output from the sensor signal lines (SS1 to SSn) varies depending on the direction of the finger 900 with respect to the surface of the sensor.

  As described above, the display device 102 can scan an image (hereinafter, also referred to as a reflected image) obtained by reflecting light with the finger 900.

Note that examples of the scan object other than the finger 900 include a stylus.
By the way, in the present embodiment, a liquid crystal panel is described as an example of the display device of electronic device 100, but another panel such as an organic EL (Electro-Luminescence) panel is used instead of the liquid crystal panel. Also good.

<About data>
Next, the sensing command will be described. In the display device 102, the image processing engine 180 analyzes the content of the sensing command and sends back data (that is, response data) according to the analysis result to the main body device 101.

  FIG. 7 is a diagram showing a schematic configuration of a sensing command. Referring to FIG. 7, the sensing command includes a header data area DA01, a timing data area DA02, a data type data area DA03, a reading method data area DA04, and image gradation data. An area DA05, a data area DA06 indicating resolution, and a spare data area DA07 are included.

  FIG. 8 is a diagram showing data values in each area of the sensing command and the meaning contents indicated by the values.

  Referring to FIG. 8, the sensing command in which “00” is set in the data area indicating the timing requests image processing engine 180 to transmit scan data at that time. That is, the sensing command requests transmission of scan data obtained by scanning using the optical sensor circuit 144 after the image processing engine 180 receives the sensing command. A sensing command in which “01” is set in the data area indicating the timing requests transmission of scan data when the scan result has changed. Furthermore, a sensing command in which “10” is set in the data area indicating the timing requests transmission of scan data at regular intervals.

  A sensing command in which “001” is set in the data area indicating the data type requests transmission of the coordinate value of the center coordinate in the partial image. In addition, a sensing command in which “010” is set in the data area indicating the data type requests transmission of only a partial image whose scan result has changed. Note that the change in the scan result indicates that the previous scan result is different from the current scan result. Further, a sensing command in which “100” is set in the data area indicating the data type requests transmission of the entire image.

  Here, the “entire image” is an image generated by the image processing engine 180 based on the voltage output from each photosensor circuit when scanning using m × n photosensor circuits. . The “partial image” is a part of the entire image. The reason why the partial image is requested to transmit only the partial image whose scan result has changed will be described later.

  The coordinate value and the partial image or the entire image may be requested at the same time. Further, in the case of a configuration in which scanning is performed on a partial area on the surface of the liquid crystal panel 140, the entire image is an image corresponding to the area to be scanned.

  A sensing command in which “00” is set in the data area indicating the reading method requests to scan with the backlight 179 on. A sensing command in which “01” is set in the data area indicating the reading method requests scanning with the backlight 179 off. A configuration for scanning with the backlight 179 off is described later (FIG. 12). Furthermore, a sensing command in which “10” is set in the data area indicating the reading method requests scanning using both reflection and transmission. Note that using both reflection and transmission refers to scanning a scanning object by switching between a method in which the backlight 179 is turned on for scanning and a method in which the backlight is turned off for scanning.

  A sensing command in which “00” is set in the data area indicating the image gradation requests monochrome image data of black and white. A sensing command in which “01” is set in the data area indicating the image gradation requests multi-gradation image data. Furthermore, a sensing command in which “10” is set in the data area indicating the image gradation requests RGB color image data.

A sensing command in which “0” is set in the data area indicating the resolution requests image data with a high resolution. A sensing command in which “1” is set in the data area indicating the resolution requests image data with a low resolution.

  In addition to the data shown in FIG. 7 and FIG. 8, the sensing command includes designation of an area to be scanned (area of pixels that drive the optical sensor circuit 144), timing to perform scanning, and timing to turn on the backlight 179. Etc. are described.

  FIG. 9 is a diagram showing a schematic configuration of response data. The response data is data corresponding to the content of the sensing command, and is data that the image processing engine 180 of the display device 102 transmits to the main body device 101.

  Referring to FIG. 9, the response data includes a header data area DA11, a coordinate data area DA12, a time data area DA13, and an image data area DA14. Here, the value of the center coordinates of the partial image is written in the data area DA12 indicating the coordinates. In addition, time information acquired from the timer 182 of the image processing engine 180 is written in the data area indicating the time. Further, image data (that is, scan data) after being processed by the image processing engine 180 is written in the data area indicating the image.

  FIG. 10 is a diagram illustrating an image obtained by scanning the finger 900 (that is, a scanned image). Referring to FIG. 10, an image of a region W1 surrounded by a thick solid line is an entire image, and an image of a region P1 surrounded by a broken line is a partial image. The center point C1 of the cross indicated by a thick line is the center coordinate.

  In the present embodiment, a pixel (that is, a predetermined gradation or a pixel having a photosensor circuit which is a rectangular region and whose output voltage from the sensor signal line SSj is equal to or higher than a predetermined value). An area including all of the pixels having a luminance equal to or higher than a predetermined luminance is set as a partial image area.

  The center coordinates are coordinates determined in consideration of the gradation of each pixel in the partial image area. Specifically, for each pixel in the partial image, the center coordinate is determined by performing a weighting process based on the gradation of the pixel and the distance between the pixel and the rectangular center point (that is, the centroid). Is done. That is, the center coordinates do not necessarily match the centroid of the partial image.

  However, the position of the center coordinates is not necessarily limited to the above, and the center coordinates may be the coordinates of the centroid or the coordinates near the centroid.

  When “001” is set in the data area indicating the data type of the sensing command, the image processing engine 180 writes the value of the central coordinate in the data area DA12 indicating the coordinate. In this case, the image processing engine 180 does not write image data in the data area DA14 indicating an image. After writing the value of the center coordinate, the image processing engine 180 sends response data including the value of the center coordinate to the main body device 101. As described above, when “001” is set in the data area indicating the data type, the sensing command requests the output of the value of the center coordinate without requesting the output of the image data.

Further, when “010” is set in the data area indicating the data type of the sensing command, the image processing engine 180 writes the image data of the partial image whose scan result has changed in the data area DA14 indicating the image. . In this case, the image processing engine 180 does not write the value of the center coordinate in the data area DA12 indicating the coordinate. The image processing engine 180 writes the image data of the partial image whose scan result has changed, and then sends response data including the image data of the partial image to the main body device 101. As described above, when “010” is set in the data area indicating the data type, the sensing command does not request the output of the value of the center coordinate, and the image data of the partial image whose scan result has changed. Request output.

  As described above, the reason for requesting transmission of only the partial image whose scan result has changed is that the scan data of the partial image area of the scan data is more important than the scan data of the other area. This is because the data is high, and the scan data in the region corresponding to the region of the partial image in the scan data is likely to change due to the contact state with the scan object such as the finger 900.

  When “011” is set in the data area indicating the data type of the sensing command, the image processing engine 180 writes the value of the central coordinate in the data area DA12 indicating the coordinate and also displays the data area indicating the image. The image data of the partial image whose scan result has changed is written in DA14. Thereafter, the image processing engine 180 sends response data including the value of the center coordinate and the image data of the partial image to the main body device 101. As described above, when “011” is set in the data area indicating the data type, the sensing command requests the output of the value of the center coordinate and the output of the image data of the partial image whose scan result has changed. To do.

  If “100” is set in the data area indicating the data type of the sensing command, the image processing engine 180 displays the image of the entire image in the data area DA14 indicating the image of the response data shown in FIG. Write data. In this case, the image processing engine 180 does not write the value of the center coordinate in the data area DA12 indicating the coordinate. After writing the image data of the entire image, the image processing engine 180 sends response data including the image data of the entire image to the main body device 101. As described above, when “100” is set in the data area indicating the data type, the sensing command requests the output of the image data of the entire image without requesting the output of the value of the center coordinate.

  When “101” is set in the data area indicating the data type of the sensing command, the image processing engine 180 writes the value of the central coordinate in the data area DA12 indicating the coordinate and also displays the data area indicating the image. Write the image data of the entire image to DA14. Thereafter, the image processing engine 180 sends response data including the value of the center coordinate and the image data of the entire image to the main body device 101. As described above, when “101” is set in the data area indicating the data type, the sensing command requests the output of the value of the center coordinate and the output of the image data of the entire image.

<Regarding First Modification of Configuration>
By the way, the structure of the liquid crystal panel 140 is not limited to the structure shown in FIG. Hereinafter, a liquid crystal panel having a mode different from that in FIG. 3 will be described.

  FIG. 11 is a circuit diagram of a photosensor built-in liquid crystal panel 140A according to the different embodiment. Referring to FIG. 11, photosensor built-in liquid crystal panel 140A (hereinafter referred to as liquid crystal panel 140A) includes three photosensor circuits (144r, 144g, 144b) in one pixel. Thus, the liquid crystal panel 140A is different from the liquid crystal panel 140 including one photosensor circuit in one pixel in that the liquid crystal panel 140A includes three photosensor circuits (144r, 144g, and 144b) in one pixel. The configuration of the optical sensor circuit 144 is the same as that of each of the three optical sensor circuits (144r, 144g, 144b).

In addition, the three photodiodes (145r, 145g, 145b) in one pixel have a color filter 153r, a color filter 153g, and a color filter 153, respectively.
It is arranged at a position facing b. Therefore, the photodiode 145r receives red light, the photodiode 145g receives green light, and the photodiode 145b receives blue light.

  In addition, since the liquid crystal panel 140 includes only one photosensor circuit 144 in one pixel, two data signal lines for the TFT 147 disposed in one pixel are a sensor signal line SSj and a sensor signal line SDj. Met. However, since the liquid crystal panel 140A includes three photosensor circuits (144r, 144g, 144b) in one pixel, there are six data signal lines for TFTs (147r, 147g, 147b) arranged in one pixel. It becomes.

  Specifically, the sensor signal line SSRj and the sensor signal line SDRj are arranged corresponding to the TFT 147r connected to the cathode of the photodiode 145r arranged at a position facing the color filter 153r. A sensor signal line SSGj and a sensor signal line SDGj are arranged corresponding to the TFT 147g connected to the cathode of the photodiode 145g arranged at a position facing the color filter 153g. Further, a sensor signal line SSBj and a sensor signal line SDBj are disposed corresponding to the TFT 147b connected to the cathode of the photodiode 145b disposed at a position facing the color filter 153b.

  In such a liquid crystal panel 140A, the white light emitted from the backlight 179 passes through the three color filters (153r, 153g, 153b), and red, green, and blue are displayed on the surface of the liquid crystal panel 140A. It mixes and becomes white light. Here, when white light is reflected by the scan object, a part of the white light is absorbed by the pigment on the surface of the scan object, and a part is reflected. And the reflected light permeate | transmits three color filters (153r, 153g, 153b) again.

  At this time, the color filter 153r transmits light having a red wavelength, and the photodiode 145r receives light having the red wavelength. The color filter 153g transmits light having a green wavelength, and the photodiode 145g receives light having the green wavelength. The color filter 153b transmits light having a blue wavelength, and the photodiode 145b receives light having the blue wavelength. That is, the light reflected by the scan object is separated into three primary colors (R, G, B) by the three color filters (153r, 153g, 153b), and each photodiode (145r, 145g, 145b) The light of the color corresponding to is received.

  When a part of white light is absorbed by the pigment on the surface of the scan target, the amount of light received by each photodiode (145r, 145g, 145b) is different for each photodiode (145r, 145g, 145b). For this reason, the output voltages of the sensor signal line SSRj, the sensor signal line SSGj, and the sensor signal line SSBj are different from each other.

  Therefore, the image processing engine 180 determines the R gradation, the G gradation, and the B gradation according to each output voltage, so that the image processing engine 180 sends the RGB color image to the main body device 101. Can send.

  As described above, when the electronic apparatus 100 includes the liquid crystal panel 140A, the scan target can be scanned in color.

  Next, a scanning method different from the above-described scanning method (that is, the method of scanning the reflected image in FIG. 6) will be described with reference to FIG.

FIG. 12 is a cross-sectional view showing a configuration in which a photodiode receives external light during scanning. As shown in FIG. 12, part of the outside light is blocked by the finger 900. Therefore, the photodiode disposed under the surface region of the liquid crystal panel 140 that is in contact with the finger 900 can hardly receive external light. In addition, although the photodiodes disposed under the surface area where the shadow of the finger 900 is formed can receive a certain amount of external light, the amount of external light received is small compared to the surface area where no shadow is formed.

  Here, by turning off the backlight 179 at least during the sensing period, the optical sensor circuit 144 can output a voltage corresponding to the position of the finger 900 with respect to the surface of the liquid crystal panel 140 from the sensor signal line SSj. it can. In this manner, by controlling the backlight 179 to be turned on and off, the liquid crystal panel 140 has a contact position of the finger 900, a range in which the finger 900 is in contact (determined by the pressing force of the finger 900), and the liquid crystal panel 140. The voltage output from the sensor signal lines (SS1 to SSn) varies depending on the direction of the finger 900 with respect to the surface of the sensor.

  As described above, the display device 102 can scan an image (hereinafter also referred to as a shadow image) obtained by blocking external light with the finger 900.

  Further, the display device 102 may be configured to perform scanning by turning on the backlight 179 and then performing scanning again by turning off the backlight 179. Alternatively, the display device 102 may be configured to perform scanning by turning off the backlight 179 and then performing scanning again by turning on the backlight 179.

  In this case, since two scanning methods are used together, two scan data can be obtained. Therefore, it is possible to obtain a highly accurate result as compared with the case of scanning using only one scanning method.

<About display devices>
The operation of the display device 103 is controlled in accordance with a command (for example, a sensing command) from the main body device 101 as in the operation of the display device 102. The display device 103 has the same configuration as the display device 102. Therefore, when the display device 103 receives the same command from the main body device 101 as the display device 102, the display device 103 performs the same operation as the display device 102. For this reason, description of the configuration and operation of the display device 103 will not be repeated.

  Note that the main device 101 can send commands having different commands to the display device 102 and the display device 103. In this case, the display device 102 and the display device 103 perform different operations. Further, the main device 101 may send a command to either the display device 102 or the display device 103. In this case, only one display device performs an operation according to the command. Further, the main device 101 may send a command having the same command to the display device 102 and the display device 103. In this case, the display device 102 and the display device 103 perform the same operation.

  Note that the size of the liquid crystal panel 140 of the display device 102 and the size of the liquid crystal panel 240 of the display device 103 may be the same or different. Further, the resolution of the liquid crystal panel 140 and the resolution of the liquid crystal panel 240 may be the same or different.

<About Second Modification of Configuration>
In the present embodiment, a configuration in which electronic device 100 includes a liquid crystal panel with a built-in optical sensor such as liquid crystal panel 140 and liquid crystal panel 240 will be described. However, only one liquid crystal panel has a built-in photosensor. It may be.

  FIG. 13 is a block diagram illustrating a hardware configuration of electronic device 1300. Similar to the electronic device 100, the electronic device 1300 includes a first housing 100A and a second housing 100B. The first housing 100A includes a display device 102. The second housing 100B includes a liquid crystal panel that does not incorporate a photosensor (that is, a liquid crystal panel having only a display function). The electronic device 1300 is different from the electronic device 100 in which the second housing 100B includes a liquid crystal panel 240 with a built-in optical sensor, in that the second housing 100B includes a liquid crystal panel with no built-in photosensor. Such an electronic device 1300 performs the sensing described above using the display device 102 of the first housing 100A.

  Further, the second casing 100B may include, for example, a resistive film type or capacitive type touch panel instead of the liquid crystal panel 240 incorporating the photosensor.

  In this embodiment, the display device 102 includes the timer 182 and the display device 103 includes the timer 282. However, the display device 102 and the display device 103 may share one timer. .

  In the present embodiment, electronic device 100 is described as a foldable device, but electronic device 100 is not necessarily limited to a foldable device. For example, the electronic device 100 may be a sliding device configured such that the first housing 100A slides with respect to the second housing 100B.

<< Specific examples of implementation >>
<Functional block>
FIG. 14 is a diagram illustrating functional blocks of the electronic device 100. Referring to FIG. 14, electronic device 100 includes control unit 10, liquid crystal panels 140 and 240, and storage device 90. The control unit 10 includes a display control unit 11, a scan unit 12, a detection unit 13, a specification unit 14, a first determination unit 15, a second determination unit 16, a third determination unit 17, and an execution unit 18. A selection processing unit 21, a handwriting input reception unit 31, a character recognition unit 32, a character input processing unit 41, and an association unit 42.

  The storage device 90 includes a RAM 171 and a hard disk (not shown). The storage device 90 stores at least first shape data indicating a first shape and second shape data indicating a second shape. The first shape and the second shape are different from each other. The first shape and the second shape are graphic shapes for determining coincidence with the shape of the image included in the scan image.

  The display control unit 11 displays images on the display surfaces of the liquid crystal panels 140 and 240 based on image data stored in a VRAM (not shown) in the storage device 90. The display control unit 11 includes position information indicating the position of the image displayed on the display surface of the liquid crystal panel 140 on the display surface, and the image displayed on the display surface of the liquid crystal panel 240 on the display surface. The position information indicating the position is sent to the second determination unit 16 and the third determination unit 17.

  The scanning unit 12 causes the liquid crystal panels 140 and 240 to perform scanning. The scanning unit 12 sends scan data to the detection unit 13 and the handwriting input reception unit 31.

  The detection unit 13 detects that at least one object has touched the display surfaces of the liquid crystal panels 140 and 240 based on the scan data. When the detection unit 13 detects the contact, the detection unit 13 sends scan data to the specifying unit 14, the second determination unit 16, and the third determination unit 17.

  The specifying unit 14 specifies the shape of the contact area for each contact area based on the scan data. For example, when a plurality of fingers come into contact with the display surface, the specifying unit 14 specifies that each of the plurality of contact areas has a substantially oval shape (see a white spot in the area P1 in FIG. 10). The identification unit 14 sends information indicating each identified shape to the first determination unit 15. The shape of the contact area other than the finger will be described later (FIGS. 15 and 16).

  Based on the first shape data stored in the storage device 90, the first determination unit 15 determines whether there is a shape that matches the first shape among the shapes specified by the specifying unit 14. For example, when the specifying unit 14 specifies n (n is a natural number) shapes, it is determined whether or not there is a shape that matches the first shape among the n shapes. In other words, when the specifying unit 14 specifies n shapes, the first determination unit 15 determines whether at least one of the n shapes matches the first shape.

  In the present embodiment, the term “match” includes not only the case where the shapes match each other completely, but also the case where the degree of match between the shapes is a predetermined value or more. Examples where the first determining unit 15 determines that the shapes match each other include a case where the specified shape and the first shape are similar, and a case where the characteristic portions of both shapes are similar. The first determination unit 15 may determine whether or not the shapes match each other using, for example, image authentication technology.

  The first determination unit 15 further determines whether there is a shape that matches the second shape among the shapes specified by the specification unit 14 based on the second shape data stored in the storage device 90. To do. Also in this determination, the first determination unit 15 performs the same processing as the determination of whether or not there is a shape that matches the first shape. The first shape and the second shape are two-dimensional shapes.

  The first determination unit 15 sends the determination results to the execution unit 18. If the first determination unit 15 determines that there is a shape that matches the first shape and a shape that matches the second shape, the first determination unit 15 sends the determination result to the second determination unit 16. If the first determination unit 15 determines that there is no shape that matches the first shape, the first determination unit 15 sends the determination result to the third determination unit 17.

  When the second determination unit 16 receives the determination result from the first determination unit 15, the second determination unit 16 makes the following determination based on the position information received from the display control unit 11. That is, the second determination unit 16 determines whether or not an input for designating an object displayed on the display surface of the liquid crystal panels 140 and 240 has been received according to the position of the contact area having a shape that matches the second shape. To do. In other words, when it is determined that there is a shape that matches the first shape and a shape that matches the second shape, the second determination unit 16 responds to the position of the contact region having a shape that matches the second shape. Then, it is determined whether or not an input for designating an object displayed on the display surface of the liquid crystal panels 140 and 240 has been received. The second determination unit 16 sends the determination result to the execution unit 18.

  When the third determination unit 17 receives the determination result from the first determination unit 15, the third determination unit 17 makes the following determination based on the position information received from the display control unit 11. That is, the third determination unit 17 determines whether or not an input for designating an object displayed on the display surface of the liquid crystal panels 140 and 240 has been received according to the position of the contact area having a shape that matches the second shape. To do. The third determination unit 17 sends the determination result to the execution unit 18.

Hereinafter, the contact region having a shape that matches the first shape is a region formed by the object A contacting the display surface. Here, the object A is the palm of the hand. More specifically, the object A is a region including the little ball of the palm of the hand. The “little finger ball” refers to a bulge portion on the lower side (wrist side) of the base of the little finger. The “region including the little finger ball” is a surface portion of the user's hand that normally contacts the paper (or the table on which the paper is placed) when the user writes, for example, on the paper using a pen or the like. The first shape is the shape of the contact surface (see FIG. 16) obtained when the palm of the hand is pressed against the liquid crystal panels 140 and 240, for example. More specifically, the first shape is the shape of the contact surface obtained when the surface of the palm of the hand near the little ball is pressed. The electronic device 100 may accept the registration of the first shape data indicating the first shape by the liquid crystal panels 140 and 240 scanning the palm of the user's hand.

  The contact region having a shape that matches the second shape is a region formed by the object B coming into contact with the display surface. Here, the object B is a pen tip or a fingertip of a stylus pen. The second shape is the shape of the contact surface obtained when the stylus pen tip or fingertip is pressed against the liquid crystal panels 140 and 240, for example. For example, the second shape is a shape showing a circle or a substantially ellipse. Hereinafter, the case where the object B is the pen tip of a stylus pen will be described as an example.

  Next, the process of the execution part 18 is demonstrated. In the following, a case where a user performs input using the liquid crystal panel 140 out of the liquid crystal panel 140 and the liquid crystal panel 240 will be described as an example.

  When it is determined that there is a shape that matches the second shape, the execution unit 18 causes the electronic device 100 to perform different operations depending on whether or not there is a shape that matches the first shape. The different operations are an operation of processing an input to the electronic device 100 by the pen tip after determining whether there is a shape that matches the first shape as a first command, and using the input as a second command. It is an operation to process. That is, even when the electronic device 100 accepts an input for designating the same position on the liquid crystal panel 140 with the pen tip, the execution unit 18 has a case where a shape that matches the first shape exists and a case where the shape does not exist. The input is processed as a different input (command).

  Further, when it is determined that there is a shape that matches the first shape and a shape that matches the second shape, the execution unit 18 further receives the input when it is determined that the input specifying the object has been received. The electronic device 100 is caused to execute an operation different from that when it is determined that the electronic device is not. One of the different operations is an operation based on the designation of the object, and an operation for accepting handwriting input accompanying the movement of the pen tip after the operation is performed. The other of the different operations is a handwriting input acceptance operation accompanying the movement of the pen tip.

  Hereinafter, when it is determined that there is a shape that matches the second shape, the operation mode of the electronic device 100 when the shape that matches the first shape does not exist is referred to as a “first operation mode”. In the following, when it is determined that there is a shape that matches the first shape and a shape that matches the second shape, the operation of electronic device 100 when it is determined that an input for designating an object is not received The mode is referred to as a “second operation mode”. Furthermore, in the following, when it is determined that there is a shape that matches the first shape and a shape that matches the second shape, the operation mode of the electronic device 100 when it is determined that an input for designating an object has been received. , Referred to as “third operation mode”.

(First operation mode)
In the first operation mode, the execution unit 18 causes the electronic device 100 to execute an operation based on the designation of the object. For example, when the object is a file, the operation based on the designation of the object is an operation for switching the file from the non-selected state to the selected state.

Specifically, the execution unit 18 sends information specifying the specified object to the selection processing unit 21. The selection processing unit 21 sets the object in a selected state. The selection processing unit 21 sends information specifying the selected object to the display control unit 11. Display control unit 1
1 displays the object in a state indicating that the object is selected. For example, the display control unit 11 performs control to switch the color of the object from the color before selection to a color different from that before selection.

(Second operation mode)
In the second operation mode, the execution unit 18 accepts handwriting input based on movement of the pen tip of the stylus pen on the display surface of the liquid crystal panel 140 and displays an image based on the accepted handwriting input. 100.

  Specifically, the execution unit 18 sends an instruction to permit the acceptance of handwriting input to the handwriting input reception unit 31. When the handwriting input receiving unit 31 receives the instruction, the handwriting input receiving unit 31 displays a display layer (see FIG. 21) for handwriting input and a display frame of a recognized character included in the layer ( 21 (see FIG. 21) is sent to the display control unit 11 (hereinafter referred to as “layer display instruction”). The handwriting input reception unit 31 receives handwriting input via the liquid crystal panel 140. As a handwriting input accepting operation, the handwriting input accepting unit 31 extracts stroke information indicating handwritten graphics and characters from the scan data sent from the scanning unit 12 after accepting the instruction. The handwriting input reception unit 31 sends the extracted stroke information to the display control unit 11 and the character recognition unit 32.

  When the display control unit 11 receives a layer display instruction from the handwriting input reception unit 31, the display control unit 11 displays a layer for handwriting input and a display frame for recognized characters included in the layer. After displaying the layer, the display control unit 11 displays a handwritten figure or character on the display surface of the liquid crystal panel 140 based on the stroke information.

  The character recognition unit 32 performs character recognition based on the stroke information. The character recognition unit 32 sends the recognition result to the display control unit 11. The character recognition unit 32 sends, for example, a character code to the display control unit 11 as a recognition result. The display control unit 11 displays characters based on the recognition result. In the present embodiment, the characters also include numbers and figures stored as codes (for example, JIS (Japan Industrial Standard) code or Unicode) by electronic device 100.

(Third operation mode)
In the third operation mode, the execution unit 18 causes the electronic device 100 to execute handwriting input based on the movement of the pen tip of the stylus pen on the display surface of the liquid crystal panel 140. In addition, the execution unit 18 causes the electronic device 100 to display an image based on the accepted handwritten input. Further, the execution unit 18 causes the electronic device 100 to execute association of the information based on the displayed image with the designated object.

  Specifically, when the electronic device 100 receives an input process for the specified object from the user via the input device, the execution unit 18 receives the handwritten input, displays the image, and specifies the specified The electronic device 100 is caused to execute association with the object.

  When the specified object is an icon indicating a file, examples of the input process include a process of opening the file. In addition, as the input process, for example, an instruction to change the name of the file can be given. Further, as the input process, for example, a process of displaying the property of the file can be mentioned. Note that the case where the specified object is an icon indicating a folder is the same as the case where the object is an icon indicating a file, and thus the description of the example of the input process will not be repeated.

When the electronic device 100 accepts the input process, the execution unit 18 sends an instruction for permitting acceptance of handwriting input to the handwriting input receiving unit 31. When the handwriting input receiving unit 31 receives the instruction, the handwriting input receiving unit 31 displays a layer for handwriting input and a display frame (see FIG. 21) of recognized characters included in the layer. Instruction (hereinafter referred to as “layer display instruction”) is sent to the display control unit 11. The handwriting input reception unit 31 receives handwriting input via the liquid crystal panel 140. As a handwriting input accepting operation, the handwriting input accepting unit 31 extracts stroke information indicating handwritten graphics and characters from the scan data sent from the scanning unit 12 after accepting the instruction. The handwriting input reception unit 31 sends the extracted stroke information to the display control unit 11, the character recognition unit 32, and the execution unit 18.

  When the display control unit 11 receives a layer display instruction from the handwriting input reception unit 31, the display control unit 11 displays a layer for handwriting input and a display frame for recognized characters included in the layer. After displaying the layer, the display control unit 11 displays a handwritten figure or character on the display surface of the liquid crystal panel 140 based on the stroke information.

  The character recognition unit 32 performs character recognition based on the stroke information. The character recognition unit 32 sends the recognition result to the display control unit 11. The character recognition unit 32 sends, for example, a character code as a recognition result to the display control unit 11 and the character input processing unit 41. The display control unit 11 displays characters based on the recognition result.

  The character input processing unit 41 associates the recognized character with the specified object. When the designated object is an icon indicating a file and the electronic device 100 accepts a process for opening the file as the input process, the character input processing unit 41 inputs the recognized character in the opened file. When the electronic device 100 accepts an instruction to change the file name as the input process, the character input processing unit 41 overwrites the contents of the memory area indicating the file name with recognized characters. Note that the case where the specified object is an icon indicating a folder is the same as the case where the object is an icon indicating a file, and thus the description of the example of the input process will not be repeated.

  The execution unit 18 sends the stroke information received from the handwriting input reception unit 31 to the association unit 42. The associating unit 42 stores the stroke information in the storage device 90 in association with the designated object.

  As described above, the first operation mode described above can be said to be a mode for accepting a so-called screen operation. Further, it can be said that the second operation mode and the third operation mode described above are modes for accepting handwriting input.

<Example of scanned image>
FIG. 15 is a diagram illustrating a scan image when the pen tip of the stylus pen is scanned. Referring to FIG. 15, electronic device 100 obtains pen tip image 401 of the stylus pen. The shape of the pen tip image 401 is circular.

  When a scan image as shown in FIG. 15 is obtained, electronic device 100 determines that there is no shape that matches the first shape and there is a shape that matches the second shape. When the scan image illustrated in FIG. 15 is obtained, the electronic device 100 sets the operation mode to the first operation mode.

FIG. 16 is a diagram showing a scan image when the stylus pen tip and the palm of the hand (near the little finger ball) are scanned. More specifically, FIG. 16 is a scan image when the user scans a state where the user holds the stylus pen with the right hand. Referring to FIG. 16, electronic device 100 obtains pen tip image 401 and hand palm image 402 from the scan data. The shape of the palm image 402 of the hand is a shape as shown in FIG. 16 because the hand is holding the stylus pen.

  When a scan image as shown in FIG. 16 is obtained, electronic device 100 determines that there is a shape that matches the first shape and a shape that matches the second shape. When the scan image shown in FIG. 16 is obtained, electronic device 100 sets the operation mode to the second operation mode or the third operation mode depending on whether or not an object is specified.

<Control structure>
FIG. 17 is a flowchart illustrating a processing flow in the electronic device 100. Referring to FIG. 17, in step S <b> 2, electronic device 100 detects that at least one object has touched the display surface of liquid crystal panel 140. In step S4, the electronic device 100 specifies the shape of the contact area on the display surface for each contact area. In step S <b> 6, electronic device 100 determines whether or not a shape that matches the pen tip shape (second shape) of the stylus pen exists among the specified shapes.

  When electronic device 100 determines that there is a shape that matches the pen tip shape (YES in step S6), in step S8, electronic device 100 determines the palm shape of the hand (first shape) among the specified shapes. It is determined whether or not there is a shape that matches. On the other hand, when electronic device 100 determines that there is no shape that matches the pen tip shape (NO in step S6), electronic device 100 ends the series of processing.

  When electronic device 100 determines that there is a shape that matches the shape of the palm of the hand (YES in step S8), in step S10, electronic device 100 has received an input for designating an object displayed on the display surface. Judging. On the other hand, when electronic device 100 determines that there is no shape that matches the shape of the palm of the hand (NO in step S8), in step S12, electronic device 100 changes the operation mode of electronic device 100 to the first operation mode. Set to.

  If it is determined that no object designation has been received (NO in step S10), in step S14, electronic device 100 sets the operation mode of electronic device 100 to the second operation mode. On the other hand, when it is determined that the object designation has been accepted (YES in step S10), in step S16, electronic device 100 sets the operation mode of electronic device 100 to the third operation mode.

<Example of screen display>
FIG. 18 is a diagram illustrating an example of display contents on the display surface of the liquid crystal panel 140. Referring to FIG. 18, electronic device 100 displays icons 501 to 503 indicating files, icons 531 and 532 indicating folders, and icon 551 indicating a trash can on the display surface. Hereinafter, the display contents of the display surface when the user touches at least one of the pen tip and the palm of the hand with the display surface of the liquid crystal panel 140 in the state illustrated in FIG. 18 will be described.

  Note that the name of the file specified by the icon 501 is “Fi-A”. The name of the file specified by the icon 502 is “Fi-B”. The name of the file specified by the icon 503 is “Fi-C”. The name of the folder specified by the icon 531 is “Fo-X”. The name of the folder specified by the icon 532 is “Fo-Y”.

FIG. 19 is a diagram illustrating a state in which the user touches the display surface of the liquid crystal panel 140 with the pen tip of the stylus pen 600. In the state shown in FIG. 19, electronic device 100 sets the operation mode to the first mode. FIG. 19 is a diagram showing a state in which an icon 501 that is an object is designated by the stylus pen 600. Referring to FIG. 19, electronic device 100 displays icon 501 as a selected state.

  FIG. 20 is a diagram illustrating a state immediately before the user touches the display surface of the liquid crystal panel 140 with the pen tip of the stylus pen 600 and the palm of the right hand 700. Referring to FIG. 20, the position of the pen tip of stylus pen 600 is not the display position of the object displayed on the display surface. For this reason, when the pen tip and the palm contact the display surface, the electronic device 100 sets the operation mode to the second mode.

  FIG. 21 is a diagram illustrating display contents on the display surface when the electronic device 100 sets the operation mode to the second operation mode. Referring to FIG. 21, electronic device 100 displays an image using liquid crystal panel 140 using layers 1401 and 1402. A layer 1402 is a layer for handwriting input. The layer 1402 is a translucent layer. The layer 1402 is an upper layer of the layer 1401. Furthermore, electronic device 100 displays a recognition character display frame 1421 in layer 1402.

  FIG. 22 is a diagram illustrating a state after the user performs handwriting input using the stylus pen in the state of FIG. 21. More specifically, FIG. 22 is a diagram illustrating a state after the user moves the right hand 700 in a state where the pen tip is in contact with the display surface. Referring to FIG. 22, electronic device 100 displays character 1451 corresponding to the locus of the pen tip on layer 1402.

  FIG. 23 is a diagram showing the display contents on the display surface after the character recognition process is performed in the state of FIG. For example, when the user presses a predetermined button (including a soft key) provided in electronic device 100, electronic device 100 starts a character recognition process. Referring to FIG. 23, electronic device 100 displays a character 1455 as a recognition result of character recognition in display frame 1421 of layer 1402.

  FIG. 24 is a diagram showing a state immediately before the user touches the display surface of the liquid crystal panel 140 with the pen tip of the stylus pen 600 and the palm of the right hand 700. Referring to FIG. 24, the position of the pen tip of stylus pen 600 is the display position of icon 501 that is an object. For this reason, when the pen tip and the palm contact the display surface, the electronic device 100 sets the operation mode to the third mode.

  FIG. 25 is a diagram showing the display content on the display screen when electronic device 100 sets the operation mode to the third operation mode. Referring to FIG. 25, electronic device 100 displays a menu screen 570 related to icon 501 on the display surface. Menu screen 570 includes, for example, three selectable items 571, 572, and 573.

  FIG. 26 is a diagram showing a state immediately before the item 572 is selected by the stylus pen 600 in the case shown in FIG. Item 572 is an item indicating that a file is to be opened.

  FIG. 27 shows a state after item 572 is selected by stylus pen 600. Referring to FIG. 27, electronic device 100 displays an image using liquid crystal panel 140 using layer 1401 and layer 1402. The electronic device 100 displays on the layer 1401 an image 501A showing a state where the file specified by the icon 501 is opened.

FIG. 28 is a diagram showing a state after the user performs handwriting input using the stylus pen in the case shown in FIG. More specifically, FIG. 28 is a diagram illustrating a state after the user moves the right hand 700 in a state where the pen tip is in contact with the display surface. Referring to FIG. 28, electronic device 100 displays character 1461 corresponding to the locus of the pen tip on layer 1402.

  FIG. 29 is a diagram showing the display contents on the display surface after the character recognition process is performed in the case shown in FIG. Referring to FIG. 29, electronic device 100 displays a character 1465 as a recognition result of character recognition in display frame 1421 of layer 1402.

  FIG. 30 is a diagram illustrating a state after the electronic device 100 receives an instruction to input the recognized character at the display position of the cursor 1501 of the image 501A in the case illustrated in FIG. 29 from the user. Referring to FIG. 30, electronic device 100 hides translucent layer 1402. Electronic device 100 further displays recognized character 1465 with the display position of cursor 1501 (see FIGS. 27 to 29) as the head.

  Electronic device 100 stores character 1461 (see FIG. 29), which is stroke information, in storage device 90 in association with the file specified by icon 501. The electronic device 100 may store the stroke information after a predetermined time has elapsed since the electronic device 100 detects that the palm of the right hand 700 and the pen tip have left the display surface, for example. .

  FIG. 31 is a diagram showing a state immediately before the user touches the display surface of the liquid crystal panel 140 with the pen tip of the stylus pen 600 and the palm of the right hand 700. Referring to FIG. 31, the position of the pen tip of stylus pen 600 is the display position of icon 531 which is an object. For this reason, when the pen tip and the palm contact the display surface, the electronic device 100 sets the operation mode to the third mode.

  FIG. 32 is a diagram showing display contents on the display screen when electronic device 100 sets the operation mode to the third operation mode. Referring to FIG. 32, since the pen tip of stylus pen 600 is the display position of icon 531, electronic device 100 displays menu screen 570 related to icon 531 on the display surface.

  FIG. 33 is a diagram showing a state immediately before the item 571 is selected by the stylus pen 600 in the case shown in FIG. Item 571 is an item indicating that the name of the folder is to be changed.

  FIG. 34 is a diagram showing a state after the item 571 is selected by the stylus pen 600. Referring to FIG. 34, electronic device 100 displays an image using liquid crystal panel 140 using layers 1401 and 1402.

  FIG. 35 is a diagram illustrating a state after the user performs handwriting input using the stylus pen in the case illustrated in FIG. 34. More specifically, FIG. 35 is a diagram illustrating a state after the user moves the right hand 700 in a state where the pen tip is in contact with the display surface. Referring to FIG. 35, electronic device 100 displays characters 1471 corresponding to the locus of the pen tip on layer 1402.

  FIG. 36 is a diagram showing the display contents on the display surface after the character recognition process is performed in the case shown in FIG. Referring to FIG. 36, electronic device 100 displays a character 1475 that is a recognition result of character recognition in display frame 1421 of layer 1402.

FIG. 37 is a diagram illustrating a state after the electronic device 100 has received an input instruction for changing the folder name to a recognized character from the user in the case illustrated in FIG. 36. Referring to FIG. 36, electronic device 100 hides translucent layer 1402. Electronic device 100 further changes the name of the folder specified by icon 531 to characters 1475 recognized from “Fo-X” (ie, “Fo-W”).

  Electronic device 100 stores character 1471 (see FIG. 35), which is stroke information, in storage device 90 in association with the folder specified by icon 531. The electronic device 100 may store the stroke information after a predetermined time has elapsed since the electronic device 100 detects that the palm of the right hand 700 and the pen tip have left the display surface, for example. .

  Next, a usage mode of the stroke information stored in the storage device 90 in association with the object will be described. When the electronic device 100 sets the operation mode to the third operation mode again and the stroke information is already associated with the file or folder specified by the specified object, the electronic device 100 performs the following processing. To do.

  That is, when electronic device 100 accepts processing for selecting any one item from items 571, 572, and 573 in menu screen 570, electronic device 100 displays stroke information on the display surface of liquid crystal panel 140. . When displaying the stroke information, the electronic device 100 preferably displays the two layers 1401 and 1402 and displays the stroke information on the layer 1402. Electronic device 100 is preferably configured to accept editing of displayed stroke information. In addition, the electronic device 100 displays the stroke information on the liquid crystal panel 140 only when the item selected when the stroke information is generated (for example, the item 571 in FIG. 33) is selected again. May be displayed on the display surface.

<Summary of Electronic Device 100>
(1) As described above, the storage device 90 stores the first shape data indicating the first shape and the second shape data indicating the second shape. The detection unit 13 detects that at least one object has contacted the display surface of the liquid crystal panel 140. When the contact is detected, the specifying unit 14 specifies the shape of the contact area for each contact area. Based on the first shape data, the first determination unit 15 determines whether there is a shape that matches the first shape (two-dimensional shape of the palm of the hand) among the specified shapes. The first determination unit 15 further determines whether there is a shape that matches the second shape (two-dimensional shape of the nib) among the specified shapes based on the second shape data. When it is determined that there is a shape that matches the second shape, the execution unit 18 causes the electronic device 100 to perform different operations depending on whether or not there is a shape that matches the first shape.

  Therefore, when the pen tip is in contact with the display surface, the electronic device 100 performs different operations depending on whether the palm of the hand is in contact with the display surface or not. Therefore, the user can easily and reliably perform the operation switching operation of the electronic device 100. Further, in electronic device 100, unless the user unintentionally touches the display surface of the palm of the hand, the operation that is not intended by the user is not switched.

  (2) When it is determined that there is a shape that matches the first shape and a shape that matches the second shape, the second determination unit 16 responds to the position of the contact region having a shape that matches the second shape. Then, it is determined whether or not an input for designating an object displayed on the display surface has been received. The execution unit 18 causes the electronic device 100 to execute different operations when it is determined that an input designating an object is received and when it is determined that the input is not received.

  Therefore, the user can input to the electronic device 100 by designating the object by placing the pen tip at the object display position. Also, the user can cause the electronic device 100 to perform different operations depending on whether the pen tip is placed at the object display position or at a position other than the object display position.

  (3) If there is no shape that matches the first shape, the third determination unit 17 has received an input for designating an object displayed on the display surface according to the contact position of the pen tip of the stylus pen 600. Determine whether. When it is determined that an input for designating an object has been received, the execution unit 18 causes the electronic device 100 to perform an operation based on the designation of the object.

  Therefore, when the user touches only the pen tip to the display surface without bringing the palm of the hand into contact with the display surface, the user places an input for designating the object by placing the pen tip at the object display position. This can be performed on the device 100. Further, when the user places the pen tip at the display position of the object, the user can cause the electronic device 100 to perform an operation based on the designation of the object. When the object is an icon indicating a file, the operation based on the designation of the object is, for example, an operation for setting the file to a selected state.

  (4) When it is determined that there is a shape that matches the first shape and a shape that matches the second shape, when it is determined that an input for designating an object is not received, the execution unit 18 displays the display surface. The electronic device 100 is caused to accept handwriting input based on the movement of the pen tip. Further, the execution unit 18 causes the electronic device 100 to display an image based on the accepted handwritten input.

  Therefore, when the user brings the pen tip and the palm of the hand into contact with the display surface and places the pen tip at a position other than the display position of the object, the user can perform handwriting input. The user can also visually recognize the result of handwriting input.

  (5) When it is determined that there is a shape that matches the first shape and a shape that matches the second shape, when it is determined that an input for designating an object has been received, the execution unit 18 The electronic device 100 is caused to accept handwriting input based on the movement of the pen tip. Further, the execution unit 18 causes the electronic device 100 to display an image based on the accepted handwritten input. Furthermore, the execution unit 18 causes the electronic device 100 to execute association of the information (stroke information) based on the displayed image with the designated object.

  Therefore, when the user brings the pen tip and the palm of the hand into contact with the display surface and places the pen tip at the display position of the object, the user can perform handwriting input. The user can also visually recognize the result of handwriting input. Further, the user can store the stroke information in the storage device 90 of the electronic device 100 in association with the designated object.

  The character recognition unit 32 recognizes the received handwritten input. When the electronic device 100 receives an instruction to open the file when the object is an icon indicating a file, the execution unit 18 causes the electronic device 100 to execute a process to open the file. Further, the execution unit 18 causes the electronic device 100 to execute the recognized character input process for the input area of the opened file. Therefore, after the electronic device 100 opens the file, the user can input a character, which is a recognition result of the handwriting input, in the input area of the file by performing handwriting input.

If the object is an icon indicating a file or an icon indicating a folder,
When the electronic device 100 receives an instruction to change the name of the object, the execution unit 18 causes the electronic device 100 to execute a process of changing the name of the object using the recognized character. Therefore, when electronic device 100 accepts an instruction to change the name of the object, the user can change the name of the object using the recognized character by performing handwriting input.

  When the contact position of the pen tip is the position where the object is displayed, the second determination unit 16 determines that an input specifying the object has been received. Therefore, the user can input to the electronic device 100 by designating the object by bringing the palm of the hand into contact with the display surface and placing the pen tip at the object display position.

<Modification of Electronic Device 100>
(1) In the above, the user specified an object with the pen tip of the stylus pen 600. However, the object designation method is not limited to the method using the pen tip. For example, the electronic device 100 may be changed to a configuration in which an object can be specified by another object such as a user's fingertip as described below.

  After detecting that the tip of the stylus pen 600 and the palm of the right hand 700 are in contact with the display surface, the electronic device 100 determines whether or not an object designation by another object has been received within a predetermined time. to decide. When the electronic device 100 accepts an object specification within a predetermined time, the electronic device 100 sets the operation mode to the third operation mode. Note that if the electronic device 100 has not received an object designation within a predetermined time, the electronic device 100 sets the operation mode to the second operation mode.

  FIG. 38 is a diagram showing a state immediately before the icon 531 is designated with the fingertip of the left hand 701 as the other object after the pen tip of the stylus pen 600 and the palm of the right hand 700 are brought into contact with the display surface. . In the state shown in FIG. 38, when the user touches the fingertip of the left hand 701 with the position where the icon 531 is displayed, the electronic device 100 changes the display content on the display surface to the state shown in FIG.

  In this way, by configuring the electronic device 100 so that an object can be specified by an object such as a finger, the user does not need to place the pen tip at the position of the object. It should be noted that the configuration of electronic device 100 may be changed so that the object can be specified using any of the pen tip of stylus pen 600 and the other object.

  (2) In the above, the electronic device 100 determines whether a shape that matches the pen tip shape exists in the specified shape, and further matches the shape of the palm of the hand in the specified shape. The configuration for determining whether or not a shape exists has been described. However, the present invention is not limited to this configuration, and the configuration of the electronic device 100 may be as follows, for example.

  The electronic device 100 determines whether or not there is a shape that matches the shape of the palm of the hand in the specified shape without determining whether or not the shape that matches the pen tip shape exists in the specified shape. to decide. The execution unit 18 of the electronic device 100 causes the electronic device 100 to perform different operations depending on whether or not there is a shape that matches the shape of the palm of the hand.

  Therefore, electronic device 100 performs different operations depending on whether the palm of the hand is in contact with the display surface or not. Therefore, the user can easily and reliably perform the operation switching operation of the electronic device 100. Further, in electronic device 100, unless the user unintentionally touches the display surface of the palm of the hand, the operation that is not intended by the user is not switched.

  The different operations include an operation for processing an input to the electronic device 100 by the pen tip after determining whether or not a shape that matches the first shape exists as a first command, and a second input for the second operation. It is an operation to process as a command.

  (3) In the above, taking into account that the user performs handwriting input using the stylus pen 600, the case where the first shape is a shape near the little finger ball has been described as an example. However, the first shape is not limited to the shape near the little ball. For example, the first shape may be a shape near the wrist of the palm of the hand. Thus, when the first shape is a shape near the wrist, it is suitable for handwriting input with a fingertip (for example, the fingertip of a forefinger). The first shape is not limited to the shape of the palm of the hand, and may be the shape of the back of the hand, for example. Furthermore, the first shape is not limited to the shape of a part of the hand, and may be the shape of another object. Also, the second shape is not limited to the shape of the pen tip of the stylus pen 600 or the shape of the fingertip. The second shape may be any shape different from the first shape.

  (4) It is preferable that the electronic device 100 can register the first shape data and the second shape data in the storage device 90 by separately scanning two objects using the liquid crystal panels 140 and 240. According to this, the electronic device 100 can be customized to a configuration according to the user's usage mode.

  (5) The electronic device 100 includes the two liquid crystal panels 140 and 240. However, in the specific implementation example described above, the electronic device 100 only needs to include at least one liquid crystal panel. In addition, the first housing 100A may include a touch panel capable of multipoint input instead of the liquid crystal panel 140 with a built-in optical sensor.

  (6) The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  DESCRIPTION OF SYMBOLS 10 Control part, 11 Display control part, 12 Scan part, 13 Detection part, 14 Identification part, 15 1st judgment part, 16 2nd judgment part, 17 3rd judgment part, 18 Execution part, 21 Selection processing part, 31 hand Document input receiving unit, 32 character recognition unit, 41 character input processing unit, 42 association unit, 90 storage device, 100 electronic device, 140 liquid crystal panel with built-in optical sensor, 600 stylus pen, 700 right hand, 701 left hand, 900 fingers.

Claims (13)

An electronic device equipped with a touch panel,
A storage device for storing first shape data indicating a first shape and second shape data indicating a second shape;
Detecting means for detecting that at least one object is in contact with the display surface of the touch panel;
Based on the detection, a specifying means for specifying the shape of the contact area for each contact area;
Based on the first shape data, it is determined whether there is a shape that matches the first shape among the identified shapes, and the specified shape is determined based on the second shape data. First determination means for determining whether or not a shape that matches the second shape exists in each shape;
If there is a shape that matches the second shape and there is no shape that matches the first shape, the electronic device is caused to execute a first operation, and the shape that matches the second shape and the first shape Execution means for causing the electronic device to execute a second operation different from the first operation without causing the electronic device to execute the first operation when there is a shape that matches one shape; Electronic equipment provided. Based on the determination that the electronic device has a shape that matches the first shape and a shape that matches the second shape, according to the position of the contact region that has the shape that matches the second shape. , Further comprising second determination means for determining whether or not an input designating an object displayed on the display surface has been received,
The electronic device according to claim 1, wherein the execution unit causes the electronic device to perform different operations when it is determined that an input designating the object is received and when it is determined that the input is not received. . A contact region having a shape that matches the second shape is formed when the first object contacts the display surface;
Whether the electronic device has received an input for designating an object displayed on the display surface according to a contact position of the first object, based on the absence of a shape that matches the first shape. A third judging means for judging,
The electronic device according to claim 2, wherein when it is determined that an input specifying the object is received, the execution unit causes the electronic device to execute an operation based on the specification of the object. The object is an icon indicating a file,
The electronic device according to claim 3, wherein the operation based on the designation of the object is an operation for setting the file in a selected state. A contact region having a shape that matches the second shape is formed when the first object contacts the display surface;
When it is determined that the input designating the object is not accepted, the execution means accepts the handwritten input based on the movement of the first object on the display surface and the image based on the accepted handwritten input. The electronic device according to claim 2, wherein display is executed by the electronic device. A contact region having a shape that matches the second shape is formed when the first object contacts the display surface;
If it is determined that an input designating the object is received, the execution means
Accepting handwriting input based on movement of the first object on the display surface;
Displaying an image based on the accepted handwritten input;
The electronic device according to claim 2, wherein the electronic device is caused to execute association of the information based on the displayed image with the designated object. The electronic device further comprises a recognition means for recognizing the accepted handwritten input,
When the electronic device receives an instruction to open the file when the object is an icon indicating a file, the execution means
Opening the file;
The electronic device according to claim 6, wherein the electronic device causes the electronic device to execute the recognized character input process on the input area of the opened file. The electronic device further comprises a recognition means for recognizing the accepted handwritten input,
When the electronic device accepts an instruction to change the name of the object when the object is an icon indicating a file or an icon indicating a folder, the execution means uses the recognized character as the name of the object. The electronic device according to claim 6, wherein the electronic device causes the electronic device to execute processing to be changed.   9. The method according to claim 6, wherein when the contact position of the first object is a position where the object is displayed, the second determination unit determines that an input designating the object is received. The electronic device described. A contact region having a shape that matches the first shape is formed by a second object contacting the display surface;
The electronic device according to claim 3, wherein the first object is a stylus pen or a finger, and the second object is a palm of a hand.   The electronic device according to any one of claims 1 to 10, wherein the touch panel includes a plurality of photosensors built in along the display surface. A display control method in an electronic device including a touch panel,
Detecting that at least one object is in contact with the display surface of the touch panel;
If the contact is detected, identifying the shape of the contact area for each contact area;
Determining whether or not there is a shape that matches the first shape among the identified shapes based on first shape data indicating the first shape;
Determining whether there is a shape that matches the second shape among the identified shapes based on second shape data indicating the second shape;
If there is a shape that matches the second shape and there is no shape that matches the first shape, the electronic device is caused to execute a first operation, and the shape that matches the second shape and the first shape And a step of causing the electronic device to execute a second operation different from the first operation without causing the electronic device to execute the first operation when there is a shape that matches one shape. , Display control method. A program for controlling an electronic device including a touch panel,
The program is stored in the electronic device.
Detecting that at least one object is in contact with the display surface of the touch panel;
If the contact is detected, identifying the shape of the contact area for each contact area;
Determining whether or not there is a shape that matches the first shape among the identified shapes based on first shape data indicating the first shape;
Determining whether there is a shape that matches the second shape among the identified shapes based on second shape data indicating the second shape;
If there is a shape that matches the second shape and there is no shape that matches the first shape, the electronic device is caused to execute a first operation, and the shape that matches the second shape and the first shape When there is a shape that matches one shape, the step of causing the electronic device to execute a second operation different from the first operation without causing the electronic device to execute the first operation. Let the program.

Images