JP2010140329A - Display device, display method, and display program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a display device for displaying an image to be outputted in a display mode, in which a user easily looks at the image, even though the user does not consciously input an instruction to change the display mode. <P>SOLUTION: The display device 100 is provided with a first display panel 140 and a second display panel 240. The display device is provided with a display control means 116 for outputting an output image to the first display panel in a first mode, and outputting the output image to the second display panel in a second mode, a first determining means 113 for determining whether at least one of first and second input images satisfies a first condition, a second determining means 114 for determining whether at least the other of the first and second input images satisfies a second condition, and a switching means 115 for switching to the second mode when the first condition is satisfied and switching to the first mode when the second condition is satisfied. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a display device, a display method, and a display program that can display an image to be output in a display mode that is easy for a user to view, and in particular, an image to be output according to how the user holds the display device. The present invention relates to a display device, a display method, and a display program that can be displayed in a display mode that is easy for a user to see.

  Information processing apparatuses that display information via a display panel and receive information via the display panel are known. Examples of such a display panel include a liquid crystal panel with a built-in optical sensor. The liquid crystal panel with a built-in optical sensor includes an optical sensor circuit that generates an input signal based on incident light and a pixel circuit that emits visible light for forming an image based on the output signal.

For example, Japanese Patent Laid-Open No. 8-76926 (Patent Document 1) discloses an image display device. According to Japanese Patent Application Laid-Open No. 8-76926 (Patent Document 1), an image display device displays a liquid crystal display that displays image information screen by screen, and a signal that is arranged on the screen of the liquid crystal display and that corresponds to a touch operation such as a finger. And a touch panel for outputting. Touching the surface of the touch panel with a finger by a predetermined amount of movement causes the touch panel to detect the direction of finger movement, and the screen data displayed on the liquid crystal display based on the direction of movement is displayed on the previous page or You can change to the next page. Furthermore, the number of pages to be turned on the screen displayed on the liquid crystal display can be changed according to the number of fingers touching the touch panel.
JP-A-8-76926

  However, in the conventional display device, in order to change the display mode of the image to be output, it is necessary for the user to input a command for changing the display mode to the display device. Then, the user needs to input such a command consciously. Therefore, there is a need for a display device, a display method, and a display program in which an output image is displayed in a display mode that is easy for the user to view even if the user does not consciously input a command.

  The present invention has been made to solve such a problem, and an object of the present invention is to display an output image that is easy for a user to view without inputting a command to change the display mode with the user consciously aware. It is providing the display apparatus displayed in an aspect, the display method, and a display program.

  According to one aspect of the present invention, a display device is provided. The display device includes a first display panel. The first display panel includes a first photosensor circuit that receives first incident light and a first pixel circuit that outputs an output image. The display device further includes a second display panel disposed on the back side of the first display panel. The second display panel includes a second photosensor circuit that receives the second incident light and a second pixel circuit that outputs an output image. The display device has a display control unit that outputs an output image to the first display panel in the first mode, and outputs an output image to the second display panel in the second mode, and the first incident light. At least one of a first generation means for generating a first input image, a second generation means for generating a second input image based on the second incident light, and a first input image and a second input image. First determination means for determining whether or not a preset first condition is satisfied, and whether or not at least one of the first and second input images satisfies a preset second condition Switching to the second mode when at least one of the first determination means and at least one of the first and second input images satisfies the first condition, and at least the other of the first and second input images is the second Switch to the first mode when the condition is met And means.

  Preferably, the first pixel circuit outputs one of the first and second output images. The second pixel circuit outputs the other of the first and second output images. The display control means causes the first display panel to output the first output image and the second display panel to output the second output image in the first mode. In the second mode, the display control means outputs the first output image to the first display image. The second output image is output to one display panel and the first output image is output to the second display panel.

  Preferably, the first determination unit obtains the number of fingers in contact with the second display panel based on the second input image, so that the first condition is satisfied based on the number of fingers. Judge whether or not. The second determining means obtains the number of fingers touching the first display panel based on the first input image, and determines whether or not the second condition is satisfied based on the number of fingers. to decide.

  Preferably, the display device further includes a storage unit that stores a first pattern image indicating a thumb. The second determination means determines whether the thumb is in contact with the first display panel based on the first input image and the first pattern image in the second mode. It is determined whether or not condition 2 is satisfied. The first determination means determines whether the thumb is in contact with the second display panel based on the second input image and the first pattern image in the first mode. It is determined whether or not condition 1 is satisfied.

  According to another aspect of the present invention, the display device includes a display panel. The display panel includes an optical sensor circuit that receives incident light and a pixel circuit that outputs an output image. The display device includes a display control unit that causes the display panel to output an output image in the vertical display mode in the vertical mode, and outputs an output image in the horizontal display mode to the display panel in the horizontal mode, and incident light. Generating means for generating an input image based on the image, vertical determining means for determining whether the input image satisfies a preset vertical condition, and determining whether the input image satisfies a preset horizontal condition Switching means for switching to the vertical mode when the input determining unit determines that the input image satisfies the vertical condition, and switching to the vertical mode when the horizontal determining unit determines that the input image satisfies the horizontal condition. Is further provided.

  According to another aspect of the present invention, the display device includes a display panel having four sides. The display panel includes an optical sensor circuit that receives incident light and a pixel circuit that outputs an output image. In the first vertical mode, the display device displays an output image in a display mode with the upper side of the display panel on the display panel, and in the second vertical mode, the lower side of the display panel is set on the display panel. The output image is displayed in the display mode, the output image is displayed in the display mode with the right side of the display panel on the display panel in the first horizontal mode, and the display panel is displayed on the display panel in the second horizontal mode. Display control means for displaying the output image in a display mode with the left side of the display side upward, generation means for generating the input image based on the incident light, and the first and second longitudinal modes and the first based on the input image Selecting means for selecting one of the first and second transverse modes.

  According to another aspect of the present invention, a display method for displaying an output image in a display device having a first display panel, a second display panel arranged on the back side of the first display panel, and an arithmetic processing unit. Is provided. The first display panel includes a first photosensor circuit that receives first incident light and a first pixel circuit that outputs an output image. The second display panel includes a second photosensor circuit that receives the second incident light and a second pixel circuit that outputs an output image. In the display method, the operation processing unit causes the first display panel to output an output image in the first mode, and the operation processing unit causes the second display panel to output the output image in the second mode. And a step in which the arithmetic processing unit generates a first input image based on the first incident light, a step in which the arithmetic processing unit generates a second input image based on the second incident light, A step of determining whether or not at least one of the first and second input images satisfies a first condition set in advance, and the arithmetic processing unit The step of switching to the second mode when at least one satisfies the first condition, and whether the arithmetic processing unit satisfies at least the other of the first and second input images satisfies a preset second condition Steps to determine , The arithmetic processing unit, at least the other of the first and second input image and a step of switching to the first mode when the second condition is satisfied.

  According to another aspect of the present invention, a display program for displaying an output image on a display device having a first display panel, a second display panel disposed on the back side of the first display panel, and an arithmetic processing unit. Is provided. The first display panel includes a first photosensor circuit that receives first incident light and a first pixel circuit that outputs an output image. The second display panel includes a second photosensor circuit that receives the second incident light and a second pixel circuit that outputs an output image. The display program causes the arithmetic processing unit to output an output image to the first display panel in the first mode, to output an output image to the second display panel in the second mode, At least one of the step of generating the first input image based on the incident light, the step of generating the second input image based on the second incident light, and the first and second input images is preset. Determining whether or not the first condition is satisfied, switching to the second mode when at least one of the first and second input images satisfies the first condition, and the first and second Determining whether at least the other of the input images satisfies a preset second condition, and when at least the other of the first and second input images satisfies the second condition, And a step of switching to de.

  As described above, according to the present invention, there is provided a display device in which an output image is displayed in a display mode that is easy for the user to view without inputting a command for changing the display mode consciously by the user. .

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[Embodiment 1]
<Overall Configuration of Electronic Device 100>
First, an overall configuration of electronic device 100 that is an example of the display device according to the present embodiment will be described. FIG. 1 is a schematic diagram showing an electronic device 100 according to the present embodiment. More specifically, FIG. 1A is an image diagram showing the surface (first liquid crystal panel 140 side) of electronic device 100 in a state where the user holds first liquid crystal panel 140 as the surface. FIG. 1B is an image diagram showing the back surface (second liquid crystal panel 240 side) of electronic device 100 in a state where the user holds first liquid crystal panel 140 as the front surface. FIG. 1C is an image diagram showing the surface (second liquid crystal panel 240 side) of electronic device 100 in a state where the user holds second liquid crystal panel 240 as the surface. FIG. 1D is an image diagram showing the back surface (first liquid crystal panel 140 side) of electronic device 100 in a state where the user holds second liquid crystal panel 240 as the front surface.

  1A to 1D, electronic device 100 includes a first photosensor built-in liquid crystal panel 140 (also referred to as first liquid crystal panel 140). The electronic device 100 includes a second photosensor built-in liquid crystal panel 240 (also referred to as a second liquid crystal panel 240) on the side opposite to the liquid crystal panel 140. As described above, electronic device 100 according to the present embodiment includes two liquid crystal panels with a built-in optical sensor. However, the electronic device 100 may be such that the first and second liquid crystal panels 140 and 240 are fixed to the front and back of the housing in advance as in the present embodiment, and both are outside. It may be deformed by the user into a state of facing or a state of both facing inward. The electronic device 100 is configured as a portable device having a display function such as a PDA (Personal Digital Assistant), a notebook personal computer, a portable telephone, and an electronic dictionary.

<Outline of Operation of Electronic Device 100>
Next, with reference to FIG. 1A to FIG. 1D, an outline of operation of the electronic device 100 according to the present embodiment will be described.

  Referring to FIG. 1A and FIG. 1B, the user makes the thumb of right hand 99R contact first liquid crystal panel 140 and the other finger of right hand 99R makes contact with second liquid crystal panel 240. By doing so, the electronic device 100 is held. At this time, there is a high possibility that the user is looking at the first liquid crystal panel 140 that is in contact with the thumb. Therefore, electronic device 100 according to the present embodiment determines that first liquid crystal panel 140 in contact with one finger is the front surface, and causes first liquid crystal panel 140 to display the first output image. Then, electronic device 100 determines that second liquid crystal panel 240 in contact with a plurality of fingers is the back surface, and causes second liquid crystal panel 240 to display a second output image.

  Alternatively, the electronic device 100 stores a pattern image indicating a thumb or other finger, determines that the first liquid crystal panel 140 in contact with the thumb is the front surface, and causes the first liquid crystal panel 140 to receive a first image. An output image may be displayed. Then, the electronic device 100 may determine that the second liquid crystal panel 240 that is not in contact with the thumb is the back surface, and display the second output image on the second liquid crystal panel 240.

  With reference to FIG. 1C and FIG. 1D, the user changes the electronic device 100. For example, the electronic device 100 is held by bringing the thumb of the left hand 99L into contact with the second liquid crystal panel 240 and bringing the other fingers of the left hand 99L into contact with the second liquid crystal panel 240. At this time, there is a high possibility that the user is looking at the second liquid crystal panel 240 that is in contact with the thumb. Electronic device 100 according to the present embodiment determines that second liquid crystal panel 240 in contact with one finger is the front surface, and displays the first output image on second liquid crystal panel 240. Then, electronic device 100 determines that first liquid crystal panel 140 in contact with a plurality of fingers is the back surface, and causes second liquid crystal panel 140 to display a second output image.

  In FIG. 1, the user changes the electronic device 100 from the right hand 99R to the left hand 99L, but the output image is switched in the same way even if the electronic device 100 is changed while holding the electronic device 100 with the right hand 99R.

  In FIG. 1, when the electronic device 100 displays a three-dimensional image, the front view of the object is displayed as the first output image, and the rear view of the object is displayed as the second output image. However, for example, the first output image may be a two-dimensional photographic image, and the second output image may be a wallpaper image designated in advance by the user.

  Furthermore, the electronic device 100 may be one in which the liquid crystal panel that displays the output image of the first and second liquid crystal panels 140 and 240 is switched depending on how the user holds the electronic device 100. That is, the electronic device 100 displays the output image on the first liquid crystal panel 140 by determining that the first liquid crystal panel 140 in contact with one finger is the front surface, and the plurality of fingers are in contact with each other. The second liquid crystal panel 240 may stop displaying images. Alternatively, the electronic device 100 displays the output image on the first liquid crystal panel 140 by determining that the first liquid crystal panel 140 in contact with the thumb is the front surface, and the second liquid crystal panel is in contact with the other finger. The liquid crystal panel 240 may not display an image.

  As described above, the electronic device 100 according to the present embodiment is based on the input image acquired through at least one of the first and second liquid crystal panels 140 and 240 and the user holds the electronic device 100 ( The electronic device 100 is held by a finger), and at least one of the first and second liquid crystal panels 140 and 240 is controlled so that the user can easily see the output image according to the holding method. . As a result, the output image is displayed in a display mode that is easy for the user to view without inputting a command to change the display mode consciously by the user. Hereinafter, a configuration for realizing such a function will be described in detail.

<Hardware Configuration of Electronic Device 100>
Next, an aspect of a specific configuration of the electronic device 100 will be described. FIG. 2 is a block diagram illustrating a hardware configuration of electronic device 100. Referring to FIG. 2, electronic device 100 includes a main device 101, a first display device 102, and a second display device 103 as main components.

  The main unit 101 includes a CPU (Central Processing Unit) 110, a RAM (Random Access Memory) 171, a ROM (Read-Only Memory) 172, a memory card reader / writer 173, a communication unit 174, a microphone 175, and a speaker. 176 and operation keys 177 are included. Each component is connected to each other by a data bus DB1. A memory card 1731 is attached to the memory card reader / writer 173.

  CPU 110 executes a program. The operation key 177 receives an instruction input from the user of the electronic device 100. The RAM 171 stores data generated by the execution of the program by the CPU 110 or data input via the operation keys 177 in a volatile manner. The ROM 172 stores data in a nonvolatile manner. The ROM 172 is a ROM capable of writing and erasing data such as an EPROM (Erasable Programmable Read-Only Memory) and a flash memory. The communication unit 174 performs wireless communication with other electronic devices (not shown). Although not illustrated in FIG. 2, the electronic device 100 may include an interface (IF) for connecting to another electronic device by wire.

  The display device 102 includes a driver 130, a first optical sensor built-in liquid crystal panel 140, an internal IF 178, a backlight 179, and an image processing engine 180.

  The driver 130 is a drive circuit for driving the liquid crystal panel 140 and the backlight 179. Various drive circuits included in the driver 130 will be described later.

  The liquid crystal panel 140 is a device having a liquid crystal display function and a photosensor function. That is, the liquid crystal panel 140 can perform image display using liquid crystal and sensing using an optical sensor. Details of the liquid crystal panel 140 will be described later.

  An internal IF (Interface) 178 mediates exchange of data between the main device 101 and the display device 102.

  The backlight 179 is a light source disposed on the back surface of the liquid crystal panel 140. The backlight 179 irradiates the back surface with uniform light.

  The image processing engine 180 controls the operation of the liquid crystal panel 140 via the driver 130. Here, the control is performed based on various data sent from the main apparatus 101 via the internal IF 178. Note that the various data includes commands to be described later. Further, the image processing engine 180 processes data output from the liquid crystal panel 140 and sends the processed data to the main apparatus 101 via the internal IF 178. Further, the image processing engine 180 includes a driver control unit 181, a timer 182, and a signal processing unit 183.

  The driver control unit 181 controls the operation of the driver 130 by sending a control signal to the driver 130. In addition, the driver control unit 181 analyzes a command transmitted from the main device 101. Then, the driver control unit 181 sends a control signal based on the analysis result to the driver 130. Details of the operation of the driver 130 will be described later.

The timer 182 generates time information and sends the time information to the signal processing unit 183.
The signal processing unit 183 receives data output from the optical sensor. Here, since the data output from the optical sensor is analog data, the signal processing unit 183 first converts the analog data into digital data. Further, the signal processing unit 183 performs data processing on the digital data in accordance with the content of the command sent from the main device 101. Then, the signal processing unit 183 sends data (hereinafter referred to as response data) including data after the above data processing and time information acquired from the timer 182 to the main unit 101. The signal processing unit 183 includes a RAM (not shown) that can store a plurality of scan data, which will be described later, continuously.

  The command includes a sensing command for instructing sensing by the optical sensor. Details of the sensing command and the response data will be described later (FIGS. 7 to 9).

  Note that the timer 182 is not necessarily provided in the image processing engine 180. For example, the timer 182 may be provided outside the image processing engine 180 in the display device 102. Alternatively, the timer 182 may be provided in the main body device 101. Further, the microphone 175 and the speaker 176 are not always provided in the electronic device 100, and may be configured so as not to include either or both of the microphone 175 and the speaker 176 depending on the embodiment of the electronic device 100.

  Here, the display device 102 includes a system liquid crystal. The system liquid crystal is a device obtained by integrally forming peripheral devices of the liquid crystal panel 140 on the glass substrate of the liquid crystal panel 140. In the present embodiment, the driver 130 (excluding the circuit that drives the backlight 179), the internal IF 178, and the image processing engine 180 are integrally formed on the glass substrate of the liquid crystal panel 140. Note that the display device 102 is not necessarily configured using a system liquid crystal, and the driver 130 (excluding a circuit that drives the backlight 179), the internal IF 178, and the image processing engine 180 are included in the glass substrate. Other substrates may be configured.

  The display device 103 includes a driver 230, a second optical sensor built-in liquid crystal panel 240, an internal IF 278, a backlight 279, and an image processing engine 280. The image processing engine 280 includes a driver control unit 281, a timer 282, and a signal processing unit 283.

  The display device 103 has the same configuration as the display device 102. That is, the driver 230, the liquid crystal panel 240, the internal IF 278, the backlight 279, and the image processing engine 280 have the same configuration as the driver 130, the liquid crystal panel 140, the internal IF 178, the backlight 179, and the image processing engine 180 in the display device 102. Have each. The driver control unit 281, the timer 282, and the signal processing unit 283 have the same configurations as the driver control unit 181, the timer 182, and the signal processing unit 183 in the display device 102. Therefore, description of each functional block included in display device 103 will not be repeated.

  By the way, the processing in the electronic device 100 is realized by each hardware and software executed by the CPU 110. Such software may be stored in the ROM 172 in advance. The software may be stored in a memory card 1731 or other storage medium and distributed as a program product. Alternatively, the software may be provided as a program product that can be downloaded by an information provider connected to the so-called Internet. Such software is read from the storage medium by the memory card reader / writer 173 or other reading device, or downloaded via the communication unit 174 or communication IF (not shown), and then temporarily stored in the ROM 172. The The software is read from the ROM 172 by the CPU 110 and stored in the RAM 171 in the form of an executable program. CPU 110 executes the program.

  Each component constituting the main device 101 of the electronic device 100 shown in FIG. 2 is a general one. Therefore, it can be said that the essential part of the present invention is the software stored in the RAM 171, the ROM 172, the memory card 1731 and other storage media, or the software downloadable via the network. Since the hardware operation of main device 101 of electronic device 100 is well known, detailed description will not be repeated.

  The storage medium is not limited to a memory card, but is a CD-ROM, FD (Flexible Disk), hard disk, magnetic tape, cassette tape, optical disk (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile). Disc)), IC (Integrated Circuit) cards (excluding memory cards), optical cards, mask ROM, EPROM, EEPROM (Electronically Erasable Programmable Read-Only Memory), flash ROM, and other semiconductor memories, etc. It may be a medium to be used.

  The program here includes not only a program directly executable by the CPU but also a program in a source program format, a compressed program, an encrypted program, and the like.

<Configuration and Drive of Photosensor Built-in Liquid Crystal Panel 140>
Next, the configuration of the liquid crystal panel 140 and the configuration of peripheral circuits of the liquid crystal panel 140 will be described. FIG. 3 is a diagram showing a configuration of the liquid crystal panel 140 and peripheral circuits of the liquid crystal panel 140.

  Referring to FIG. 3, the liquid crystal panel 140 includes a pixel circuit 141, an optical sensor circuit 144, a scanning signal line Gi, a data signal line SRj, a data signal line SGj, a data signal line SBj, and a sensor signal line. SSj, sensor signal line SDj, read signal line RWi, and reset signal line RSi are included. Note that i is a natural number satisfying 1 ≦ i ≦ m, and j is a natural number satisfying 1 ≦ j ≦ n.

  2 includes a scanning signal line driving circuit 131, a data signal line driving circuit 132, an optical sensor driving circuit 133, a switch 134, as peripheral circuits of the liquid crystal panel 140. And an amplifier 135.

  The scanning signal line drive circuit 131 receives the control signal TC1 from the driver control unit 181 shown in FIG. The scanning signal line drive circuit 131 applies a predetermined voltage in order from the scanning signal line G1 to each scanning signal line (G1 to Gm) based on the control signal TC1. More specifically, the scanning signal line driving circuit 131 sequentially selects one scanning signal line from the scanning signal lines (G1 to Gm) per unit time, and a TFT (to be described later) with respect to the selected scanning signal line. A voltage capable of turning on the gate of the thin film transistor 142 is applied (hereinafter referred to as a high level voltage). Note that a low level voltage is applied to a scanning signal line that is not selected without applying a high level voltage.

  The data signal line driving circuit 132 receives image data (DR, DG, DB) from the driver control unit 181 shown in FIG. The data signal line driving circuit 132 applies a voltage corresponding to one row of image data to the 3n data signal lines (SR1 to SRn, SG1 to SGn, SB1 to SBn) for each unit time. Apply sequentially.

  Note that although a driving method called a so-called line-sequential method has been described here, the driving method is not limited to this.

  The pixel circuit 141 is a circuit for setting the luminance (transmittance) of one pixel. Further, m × n pixel circuits 141 are arranged in a matrix. More specifically, m pixel circuits 141 are arranged in the vertical direction in FIG. 3 and n pixel circuits in the horizontal direction.

  The pixel circuit 141 includes an R subpixel circuit 141r, a G subpixel circuit 141g, and a B subpixel circuit 141b. Each of these three circuits (141r, 141g, 141b) includes a TFT 142, a pair of electrode pairs 143 including a pixel electrode and a counter electrode, and a capacitor (not shown).

  In addition, CMOS (Complementary Metal Oxide Semiconductor) capable of forming n-type transistors and p-type transistors can be realized, and the movement speed of carriers (electrons or holes) is several hundreds compared to amorphous silicon thin film transistors (a-Si TFTs). For example, a polycrystalline silicon thin film transistor (p-Si TFT) is used as the TFT 142 in the display device 102 because it is twice as fast. Note that the TFT 142 will be described as an n-channel field effect transistor. However, the TFT 142 may be a p-type channel field effect transistor.

  The source of the TFT 142 in the R subpixel circuit 141r is connected to the data signal line SRj. The gate of the TFT 142 is connected to the scanning signal line Gi. Further, the drain of the TFT 142 is connected to the pixel electrode of the electrode pair 143. A liquid crystal is disposed between the pixel electrode and the counter electrode. Note that the G subpixel circuit 141g and the B subpixel circuit 141b have the same configuration as the pixel circuit 141r except that the data signal line to which the source of each TFT 142 is connected is different. Therefore, description of these two circuits (141g, 141b) will not be repeated.

  Here, setting of luminance in the pixel circuit 141 will be described. First, the high level voltage is applied to the scanning signal line Gi. By the application of the high level voltage, the gate of the TFT 142 is turned on. In this manner, with the gate of the TFT 142 turned on, a specified voltage (voltage corresponding to image data for one pixel) is applied to each data signal line (SRj, SGj, SBj). Thereby, a voltage based on the designated voltage is applied to the pixel electrode. As a result, a potential difference is generated between the pixel electrode and the counter electrode. Based on this potential difference, the liquid crystal responds and the luminance of the pixel is set to a predetermined luminance. Note that the potential difference is held by the capacitor (auxiliary capacitor) (not shown) until the scanning signal line Gi is selected in the next frame period.

  The optical sensor drive circuit 133 receives the control signal TC2 from the driver control unit 181 shown in FIG.

  Then, the optical sensor drive circuit 133 sequentially selects one signal line from the reset signal lines (RS1 to RSm) for each unit time based on the control signal TC2, and determines in advance for the selected signal line. At a given timing, the voltage VDDR that is higher than usual is applied. Note that a voltage VSSR lower than the voltage applied to the selected reset signal line is kept applied to the unselected reset signal line. For example, the voltage VDDR may be set to 0V and the voltage VSSR may be set to −5V.

  Further, the optical sensor driving circuit 133 sequentially selects one signal line from the readout signal lines (RW1 to RWm) for each unit time based on the control signal TC2, and determines in advance for the selected signal line. At a given timing, a voltage VDD higher than usual is applied. Note that the voltage VSSR is applied to the read signal line that is not selected. For example, the value of VDD may be set to 8V.

  The timing for applying the voltage VDDR and the timing for applying the voltage VDD will be described later.

  The optical sensor circuit 144 includes a photodiode 145, a capacitor 146, and a TFT 147. In the following description, it is assumed that the TFT 147 is an n-type channel field effect transistor. However, the TFT 147 may be a p-type channel field effect transistor.

  The anode of the photodiode 145 is connected to the reset signal line RSi. On the other hand, the cathode of the photodiode 145 is connected to one electrode of the capacitor 146. The other electrode of the capacitor 146 is connected to the read signal line RWi. Hereinafter, a connection point between the photodiode 145 and the capacitor 146 is referred to as a node N.

  The gate of the TFT 147 is connected to the node N. The drain of the TFT 147 is connected to the sensor signal line SDj. Further, the source of the TFT 147 is connected to the sensor signal line SSj. Details of sensing using the optical sensor circuit 144 will be described later.

  The switch 134 is a switch provided to switch whether or not to apply a predetermined voltage to the sensor signal lines (SD1 to SDn). The switching operation of the switch 134 is performed by the optical sensor driving circuit 133. Note that the voltage applied to the sensor signal lines (SD1 to SDn) when the switch 134 is turned on will be described later.

  The amplifier 135 amplifies the voltage output from each sensor signal line (SS1 to SSn). The amplified voltage is sent to the signal processing unit 183 shown in FIG.

  The image processing engine 180 controls the timing for displaying an image on the liquid crystal panel 140 using the pixel circuit 141 and the timing for sensing using the optical sensor circuit 144.

  FIG. 4 is a cross-sectional view of the liquid crystal panel 140 and the backlight 179. Referring to FIG. 4, liquid crystal panel 140 includes an active matrix substrate 151A, a counter substrate 151B, and a liquid crystal layer 152. The counter substrate 151B is disposed to face the active matrix substrate 151A. The liquid crystal layer 152 is sandwiched between the active matrix substrate 151A and the counter substrate 151B. The backlight 179 is disposed on the opposite side of the liquid crystal layer 152 with respect to the active matrix substrate 151A.

  The active matrix substrate 151 </ b> A includes a polarizing filter 161, a glass substrate 162, a pixel electrode 143 a that constitutes the electrode pair 143, a photodiode 145, a data signal line 157, and an alignment film 164. Further, although not shown in FIG. 4, the active matrix substrate 151A includes the capacitor 146, the TFT 147, the TFT 142, and the scanning signal line Gi shown in FIG.

  In the active matrix substrate 151A, the polarizing filter 161, the glass substrate 162, the pixel electrode 143a, and the alignment film 164 are arranged in this order from the backlight 179 side. The photodiode 145 and the data signal line 157 are formed on the liquid crystal layer 152 side of the glass substrate 162.

  The counter substrate 151B includes a polarizing filter 161, a glass substrate 162, a light shielding film 163, color filters (153r, 153g, 153b), a counter electrode 143b constituting the electrode pair 143, and an alignment film 164.

  In the counter substrate 151B, the alignment film 164, the counter electrode 143b, the color filters (153r, 153g, 153b), the glass substrate 162, and the polarizing filter 161 are arranged in this order from the liquid crystal layer 152 side. The light shielding film 163 is formed in the same layer as the color filters (153r, 153g, 153b).

  The color filter 153r is a filter that transmits light having a red wavelength. The color filter 153g is a filter that transmits light having a green wavelength. The color filter 153b is a filter that transmits light having a blue wavelength. Here, the photodiode 145 is arranged at a position facing the color filter 153b.

  The liquid crystal panel 140 displays an image by blocking external light or light emitted from a light source such as a backlight 179 or transmitting the light. Specifically, the liquid crystal panel 140 changes the direction of liquid crystal molecules in the liquid crystal layer 152 by applying a voltage between the pixel electrode 143a and the counter electrode 143b, thereby blocking or transmitting the light. However, since the light cannot be completely blocked by the liquid crystal alone, a polarizing filter 161 that transmits only light having a specific polarization direction is provided.

  Note that the position of the photodiode 145 is not limited to the above position, and may be provided at a position facing the color filter 153r or a position facing the color filter 153g.

  Here, the operation of the optical sensor circuit 144 will be described. FIG. 5 is a diagram showing a timing chart when the optical sensor circuit 144 is operated. In FIG. 5, a voltage VINT indicates a potential at the node N in the photosensor circuit 144. A voltage VPIX is an output voltage from the sensor signal line SSj shown in FIG. 3 and is a voltage before being amplified by the amplifier 135.

  The following description will be divided into a reset period for resetting the optical sensor circuit 144, a sensing period for sensing light using the optical sensor circuit 144, and a readout period for reading the sensed result.

  First, the reset period will be described. In the reset period, the voltage applied to the reset signal line RSi is instantaneously switched from the low level (voltage VSSR) to the high level (voltage VDDR). On the other hand, the voltage applied to the read signal line RWi is kept at the low level (voltage VSSR). As described above, by applying the high-level voltage to the reset signal line RSi, a current starts to flow in the forward direction (from the anode side to the cathode side) of the photodiode 145. As a result, the voltage VINT which is the potential of the node N has a value represented by the following expression (1). In Equation (1), the forward voltage drop amount in the photodiode 145 is Vf.

VINT = VSSR + | VDDR−VSSR | −Vf (1)
Therefore, the potential of the node N is a value smaller by Vf than the voltage VDDR as shown in FIG.

  Here, since the voltage VINT is not more than the threshold value for turning on the gate of the TFT 147, there is no output from the sensor signal line SSj. For this reason, the voltage VPIX does not change. Further, a difference corresponding to the voltage VINT occurs between the electrodes of the capacitor 146. For this reason, the capacitor 146 accumulates charges corresponding to the difference.

  Next, the sensing period will be described. In the sensing period following the reset period, the voltage applied to the reset signal line RSi instantaneously switches from the high level (voltage VDDR) to the low level (voltage VSSR). On the other hand, the voltage applied to the read signal line RWi is kept at the low level (voltage VSSR).

  Thus, by changing the voltage applied to the reset signal line RSi to the low level, the potential of the node N becomes higher than the voltage of the reset signal line RSi and the voltage of the read signal line RWi. For this reason, in the photodiode 145, the voltage on the cathode side becomes higher than the voltage on the anode side. That is, the photodiode 145 is in a reverse bias state. In such a reverse bias state, when the photodiode 145 receives light from the light source, current starts to flow from the cathode side to the anode side of the photodiode 145. As a result, as shown in FIG. 5, the potential of the node N (that is, the voltage VINT) becomes lower with the passage of time.

  Since the voltage VINT continues to decrease in this way, the gate of the TFT 147 does not turn on. Therefore, there is no output from the sensor signal line SSj. For this reason, the voltage VPIX does not change.

  Next, the reading period will be described. In the readout period following the sensing period, the voltage applied to the reset signal line RSi is kept at the low level (voltage VSSR). On the other hand, the voltage applied to the read signal line RWi is instantaneously switched from the low level (voltage VSSR) to the high level (voltage VDD). Here, the voltage VDD is higher than the voltage VDDR.

  Thus, by applying a high level voltage instantaneously to the read signal line RWi, the potential of the node N is raised through the capacitor 146 as shown in FIG. Note that the increase width of the potential of the node N is a value corresponding to the voltage applied to the read signal line RWi. Here, since the potential of the node N (that is, the voltage VINT) is raised to a threshold value that turns on the gate of the TFT 147, the gate of the TFT 147 is turned on.

  At this time, if a constant voltage is applied in advance to the sensor signal line SDj (see FIG. 3) connected to the drain side of the TFT 147, the sensor signal line SSj connected to the source side of the TFT 147 will cause the VPIX in FIG. As shown in the graph, a voltage corresponding to the potential of the node N is output.

  Here, when the amount of light received by the photodiode 145 (hereinafter referred to as the amount of received light) is small, the slope of the straight line shown in the VINT graph of FIG. 5 becomes gentle. As a result, the voltage VPIX is higher than when the amount of received light is large. As described above, the optical sensor circuit 144 changes the value of the voltage output to the sensor signal line SSj in accordance with the amount of light received by the photodiode 145.

  By the way, in the above, the operation | movement was demonstrated paying attention to one optical sensor circuit 144 among the m * n optical sensor circuits which exist. Below, operation | movement of each photosensor circuit in the liquid crystal panel 140 is demonstrated.

  First, the optical sensor drive circuit 133 applies a predetermined voltage to all n sensor signal lines (SD1 to SDn). Next, the photosensor drive circuit 133 applies a voltage VDDR that is higher than normal to the reset signal line RS1. The other reset signal lines (RS2 to RSm) and read signal lines (RW1 to RWm) are kept in a state where a low level voltage is applied. As a result, the n photosensor circuits in the first row in FIG. 3 enter the reset period described above. Thereafter, the n photosensor circuits in the first row enter a sensing period. Further, thereafter, the n photosensor circuits in the first row enter a reading period.

  Note that the timing at which a predetermined voltage is applied to all n sensor signal lines (SD1 to SDn) is not limited to the above timing, and may be any timing that is applied at least before the readout period. .

  When the readout period of the n photosensor circuits in the first row is completed, the photosensor drive circuit 133 applies a voltage VDDR that is higher than usual to the reset signal line RS2. That is, the reset period of the n photosensor circuits in the second row starts. When the reset period ends, the n photosensor circuits in the second row enter a sensing period, and thereafter enter a reading period.

  Thereafter, the processing described above is sequentially performed on the n photosensor circuits in the third row, the n photosensor circuits in the fourth row,..., The n photosensor circuits in the m row. As a result, the sensing result of the first row, the sensing result of the second row,..., The sensing result of the m-th row are output in this order from the sensor signal lines (SS1 to SSn).

  In the display device 102, sensing is performed for each row as described above, and a sensing result is output from the liquid crystal panel 140 for each row. For this reason, hereinafter, the data after the signal processing unit 183 performs the above-described data processing on the data regarding the voltage for m rows from the first row to the m-th row output from the liquid crystal panel 140, This is called “scan data”. That is, scan data refers to image data obtained by scanning a scan target (for example, a user's finger). An image displayed based on the scan data is referred to as a “scanned image”. Further, in the following, sensing is referred to as “scan”.

  Moreover, in the above, although the structure which scans using all the m * n photosensor circuits was mentioned as an example, it is not limited to this. Scanning may be performed on a partial region of the surface of the liquid crystal panel 140 using a photosensor circuit selected in advance.

  In the following, it is assumed that the electronic device 100 can take either of the two configurations. Further, switching between the components is assumed to be performed by a command sent from the main device 101 based on an input via the operation key 177 or the like. Note that when scanning is performed on a partial area on the surface of the liquid crystal panel 140, the image processing engine 180 sets a scan target area. The setting of the area may be configured to be specified by the user via the operation key 177.

  As described above, when scanning is performed on a partial region of the surface of the liquid crystal panel 140, there are the following modes of use for displaying an image. The first is a mode in which an image is displayed in a surface area other than the partial area (hereinafter referred to as a scan area). The second is a mode in which no image is displayed in the surface area other than the scan area. Which mode is used is based on a command sent from the main apparatus 101 to the image processing engine 180.

  FIG. 6 is a cross-sectional view of the liquid crystal panel 140 and the backlight 179, showing a configuration in which the photodiode 145 receives light from the backlight 179 during scanning.

  Referring to FIG. 6, when the user's finger 900 is in contact with the surface of the liquid crystal panel 140, a part of the light emitted from the backlight 179 is part of the user's finger 900 (abbreviated in the contacted area). Reflected on the plane). The photodiode 145 receives the reflected light.

  Even in a region where the finger 900 is not in contact, part of the light emitted from the backlight 179 is reflected by the user's finger 900. Even in this case, the photodiode 145 receives the reflected light. However, since the finger 900 is not in contact with the surface of the liquid crystal panel 140 in the region, the amount of light received by the photodiode 145 is smaller than the region in which the finger 900 is in contact. Note that the photodiode 145 cannot receive most of the light emitted from the backlight 179 that does not reach the user's finger 900.

  Here, by turning on the backlight 179 at least during the sensing period, the optical sensor circuit 144 can output a voltage corresponding to the amount of light reflected by the user's finger 900 from the sensor signal line SSj. it can. In this manner, by controlling the backlight 179 to be turned on and off, the liquid crystal panel 140 has a contact position of the finger 900, a range in which the finger 900 is in contact (determined by the pressing force of the finger 900), The voltage output from the sensor signal lines (SS1 to SSn) varies depending on the direction of the finger 900 with respect to the surface of the sensor.

  As described above, the display device 102 can scan an image (hereinafter, also referred to as a reflected image) obtained by reflecting light with the finger 900.

Note that examples of the scan object other than the finger 900 include a stylus pen.
By the way, in the present embodiment, a liquid crystal panel is described as an example of the display device of electronic device 100, but another panel such as an organic EL (Electro-Luminescence) panel is used instead of the liquid crystal panel. Also good.

<About data>
Next, the sensing command will be described. In the display device 102, the image processing engine 180 analyzes the content of the sensing command and sends back data (that is, response data) according to the analysis result to the main body device 101.

  FIG. 7 is a diagram showing a schematic configuration of a sensing command. Referring to FIG. 7, the sensing command includes a header data area DA01, a timing data area DA02, a data type data area DA03, a reading method data area DA04, and image gradation data. An area DA05, a data area DA06 indicating resolution, and a spare data area DA07 are included.

  FIG. 8 is a diagram showing data values in each area of the sensing command and the meaning contents indicated by the values.

  Referring to FIG. 8, the sensing command in which “00” is set in the data area indicating the timing requests image processing engine 180 to transmit scan data at that time. That is, the sensing command requests transmission of scan data obtained by scanning using the optical sensor circuit 144 after the image processing engine 180 receives the sensing command. A sensing command in which “01” is set in the data area indicating the timing requests transmission of scan data when the scan result has changed. Furthermore, a sensing command in which “10” is set in the data area indicating the timing requests transmission of scan data at regular intervals.

  A sensing command in which “001” is set in the data area indicating the data type requests transmission of the coordinate value of the center coordinate in the partial image. In addition, a sensing command in which “010” is set in the data area indicating the data type requests transmission of only a partial image whose scan result has changed. Note that the change in the scan result indicates that the previous scan result is different from the current scan result. Further, a sensing command in which “100” is set in the data area indicating the data type requests transmission of the entire image.

  Here, the “entire image” is an image generated by the image processing engine 180 based on the voltage output from each photosensor circuit when scanning using m × n photosensor circuits. . The “partial image” is a part of the entire image. The reason why the partial image is requested to transmit only the partial image whose scan result has changed will be described later.

  The coordinate value and the partial image or the entire image may be requested at the same time. Further, in the case of a configuration in which scanning is performed on a partial area on the surface of the liquid crystal panel 140, the entire image is an image corresponding to the area to be scanned.

  A sensing command in which “00” is set in the data area indicating the reading method requests to scan with the backlight 179 on. A sensing command in which “01” is set in the data area indicating the reading method requests scanning with the backlight 179 off. A configuration for scanning with the backlight 179 off is described later (FIG. 11). Furthermore, a sensing command in which “10” is set in the data area indicating the reading method requests scanning using both reflection and transmission. Note that the combined use of reflection and transmission refers to scanning a scan object by switching between a method in which the backlight 179 is turned on for scanning and a method in which the backlight is turned off for scanning.

  A sensing command in which “00” is set in the data area indicating the image gradation requests monochrome image data of black and white. A sensing command in which “01” is set in the data area indicating the image gradation requests multi-gradation image data. Furthermore, a sensing command in which “10” is set in the data area indicating the image gradation requests RGB color image data.

  A sensing command in which “0” is set in the data area indicating the resolution requests image data with a high resolution. A sensing command in which “1” is set in the data area indicating the resolution requests image data with a low resolution.

  In addition to the data shown in FIG. 7 and FIG. 8, the sensing command includes designation of an area to be scanned (area of pixels that drive the optical sensor circuit 144), timing to perform scanning, and timing to turn on the backlight 179. Etc. are described.

  FIG. 9 is a diagram showing a schematic configuration of response data. The response data is data corresponding to the content of the sensing command, and is data that the image processing engine 180 of the display device 102 transmits to the main body device 101.

  Referring to FIG. 9, the response data includes a header data area DA11, a coordinate data area DA12, a time data area DA13, and an image data area DA14. Here, the value of the center coordinates of the partial image is written in the data area DA12 indicating the coordinates. In addition, time information acquired from the timer 182 of the image processing engine 180 is written in the data area indicating the time. Further, image data (that is, scan data) after being processed by the image processing engine 180 is written in the data area indicating the image.

  FIG. 10 is a diagram illustrating an image obtained by scanning the finger 900 (that is, a scanned image). Referring to FIG. 10, an image of a region W1 surrounded by a thick solid line is an entire image, and an image of a region P1 surrounded by a broken line is a partial image. The center point C1 of the cross indicated by a thick line is the center coordinate.

  In the present embodiment, a pixel (that is, a predetermined gradation or a pixel having a photosensor circuit which is a rectangular region and whose output voltage from the sensor signal line SSj is equal to or higher than a predetermined value). An area including all of the pixels having a luminance equal to or higher than a predetermined luminance is set as a partial image area.

  The center coordinates are coordinates determined in consideration of the gradation of each pixel in the partial image area. Specifically, for each pixel in the partial image, the center coordinate is determined by performing a weighting process based on the gradation of the pixel and the distance between the pixel and the rectangular center point (that is, the centroid). Is done. That is, the center coordinates do not necessarily match the centroid of the partial image.

  However, the position of the center coordinates is not necessarily limited to the above, and the center coordinates may be the coordinates of the centroid or the coordinates near the centroid.

  When “001” is set in the data area indicating the data type of the sensing command, the image processing engine 180 writes the value of the central coordinate in the data area DA12 indicating the coordinate. In this case, the image processing engine 180 does not write image data in the data area DA14 indicating an image. After writing the value of the center coordinate, the image processing engine 180 sends response data including the value of the center coordinate to the main body device 101. As described above, when “001” is set in the data area indicating the data type, the sensing command requests the output of the value of the center coordinate without requesting the output of the image data.

  Further, when “010” is set in the data area indicating the data type of the sensing command, the image processing engine 180 writes the image data of the partial image whose scan result has changed in the data area DA14 indicating the image. . In this case, the image processing engine 180 does not write the value of the center coordinate in the data area DA12 indicating the coordinate. The image processing engine 180 writes the image data of the partial image whose scan result has changed, and then sends response data including the image data of the partial image to the main body device 101. As described above, when “010” is set in the data area indicating the data type, the sensing command does not request the output of the value of the center coordinate, and the image data of the partial image whose scan result has changed. Request output.

  As described above, the reason for requesting transmission of only the partial image whose scan result has changed is that the scan data of the partial image area of the scan data is more important than the scan data of the other area. This is because the data is high, and the scan data in the region corresponding to the region of the partial image in the scan data is likely to change due to the contact state with the scan object such as the finger 900.

  When “011” is set in the data area indicating the data type of the sensing command, the image processing engine 180 writes the value of the central coordinate in the data area DA12 indicating the coordinate and also displays the data area indicating the image. The image data of the partial image whose scan result has changed is written in DA14. Thereafter, the image processing engine 180 sends response data including the value of the center coordinate and the image data of the partial image to the main body device 101. As described above, when “011” is set in the data area indicating the data type, the sensing command requests the output of the value of the center coordinate and the output of the image data of the partial image whose scan result has changed. To do.

  If “100” is set in the data area indicating the data type of the sensing command, the image processing engine 180 displays the image of the entire image in the data area DA14 indicating the image of the response data shown in FIG. Write data. In this case, the image processing engine 180 does not write the value of the center coordinate in the data area DA12 indicating the coordinate. After writing the image data of the entire image, the image processing engine 180 sends response data including the image data of the entire image to the main body device 101. As described above, when “100” is set in the data area indicating the data type, the sensing command requests the output of the image data of the entire image without requesting the output of the value of the center coordinate.

  When “101” is set in the data area indicating the data type of the sensing command, the image processing engine 180 writes the value of the central coordinate in the data area DA12 indicating the coordinate and also displays the data area indicating the image. Write the image data of the entire image to DA14. Thereafter, the image processing engine 180 sends response data including the value of the center coordinate and the image data of the entire image to the main body device 101. As described above, when “101” is set in the data area indicating the data type, the sensing command requests the output of the value of the center coordinate and the output of the image data of the entire image.

<Variation of scanning method>
Next, a scanning method different from the scanning method described with reference to FIG. 6 (that is, the method of scanning the reflected image) will be described with reference to FIG.

  FIG. 11 is a cross-sectional view showing a configuration in which a photodiode receives external light during scanning. Referring to FIG. 11, part of the outside light is blocked by finger 900. Therefore, the photodiode disposed under the surface region of the liquid crystal panel 140 that is in contact with the finger 900 can hardly receive external light. In addition, although the photodiodes disposed under the surface area where the shadow of the finger 900 is formed can receive a certain amount of external light, the amount of external light received is small compared to the surface area where no shadow is formed.

  Here, by turning off the backlight 179 at least during the sensing period, the optical sensor circuit 144 can output a voltage corresponding to the position of the finger 900 with respect to the surface of the liquid crystal panel 140 from the sensor signal line SSj. it can. In this manner, by controlling the backlight 179 to be turned on and off, the liquid crystal panel 140 has a contact position of the finger 900, a range in which the finger 900 is in contact (determined by the pressing force of the finger 900), and the liquid crystal panel 140. The voltage output from the sensor signal lines (SS1 to SSn) varies depending on the direction of the finger 900 with respect to the surface of the sensor.

  As described above, the display device 102 can scan an image (hereinafter also referred to as a shadow image) obtained by blocking external light with the finger 900.

  Further, the display device 102 may be configured to perform scanning by turning on the backlight 179 and then performing scanning again by turning off the backlight 179. Alternatively, the display device 102 may be configured to perform scanning by turning off the backlight 179 and then performing scanning again by turning on the backlight 179.

  In this case, since two scanning methods are used together, two scan data can be obtained. Therefore, it is possible to obtain a highly accurate result as compared with the case of scanning using only one scanning method.

<About display devices>
The operation of the display device 103 is controlled in accordance with a command (for example, a sensing command) from the main body device 101 as in the operation of the display device 102. The display device 103 has the same configuration as the display device 102. Therefore, when the display device 103 receives the same command as the display device 102 from the main body device 101, the display device 103 performs the same operation as the display device 102. For this reason, description of the configuration and operation of the display device 103 will not be repeated.

  Note that the main device 101 can send commands having different commands to the display device 102 and the display device 103. In this case, the display device 102 and the display device 103 perform different operations. Further, the main device 101 may send a command to either the display device 102 or the display device 103. In this case, only one display device performs an operation according to the command. Further, the main device 101 may send a command having the same command to the display device 102 and the display device 103. In this case, the display device 102 and the display device 103 perform the same operation.

  Note that the size of the liquid crystal panel 140 of the display device 102 and the size of the liquid crystal panel 240 of the display device 103 may be the same or different. Further, the resolution of the liquid crystal panel 140 and the resolution of the liquid crystal panel 240 may be the same or different.

<Functional configuration of electronic device 100 according to the present embodiment>
Hereinafter, a functional configuration of electronic device 100 according to the present embodiment will be described with reference to FIGS. 1, 2, and 12. FIG. 12 is a block diagram showing a functional configuration of electronic device 100 according to the present embodiment.

  Electronic device 100 according to the present embodiment includes first generation unit 111, second generation unit 112, first determination unit 113, second determination unit 114, switching unit 115, and display control. Part 116. In addition, as illustrated in FIG. 2, the electronic device 100 includes a RAM 171, a first liquid crystal panel 140 including a plurality of photosensor circuits 144 and a plurality of pixel circuits 141, a plurality of photosensor circuits 244 and a plurality of photosensor circuits 244. And a second liquid crystal panel 240 including the pixel circuit 241.

  Electronic device 100 according to the present embodiment has at least a first mode and a second mode, the display state of first liquid crystal panel 140 in the first mode, and the first mode in the second mode. The display state of the first liquid crystal panel 140 is different. The display state of the second liquid crystal panel 240 in the first mode is different from the display state of the second liquid crystal panel 240 in the second mode.

  First, each of the plurality of optical sensor circuits 144 receives incident light and generates an electrical signal corresponding to the incident light. The plurality of photosensor circuits 144 as a whole input an electrical signal corresponding to incident light to the first determination unit 113 via the image processing engine 180 (FIG. 2) or the like. Note that the plurality of optical sensor circuits 144 may read the position of the finger or the stylus pen with the backlight 179 turned off as shown in FIG.

  Each of the plurality of pixel circuits 141 emits visible light to the outside based on image data and text data from the display control unit 116. More specifically, the plurality of pixel circuits 141 as a whole are based on image data and text data (electrical signals) from the display control unit 116 via the image processing engine 180 (FIG. 2) and the like from the backlight 179. Display images and text while using the light.

  As described above, the plurality of photosensor circuits 144 and the image processing engine 180 according to the present embodiment implement an input unit as a whole. In addition, the plurality of pixel circuits 141 and the image processing engine 180 according to the present embodiment realize a display unit as a whole.

  Since the function of second liquid crystal panel 240 is the same as that of first liquid crystal panel 140, description thereof will not be repeated here.

  The RAM 171 stores pattern image data 171A indicating the shape and characteristics of each finger of the right hand and the left hand. The RAM 171 stores word processing software, an e-mail transmission / reception program, still image data, moving image data, and the like. The display control unit 116 reads still image data and moving image data from the RAM 171 and displays the still image and moving image on the first liquid crystal panel 140 and the second liquid crystal panel 240.

  The first generation unit 111, the second generation unit 112, the first determination unit 113, the second determination unit 114, the switching unit 115, and the display control unit 116 are functions realized by the CPU 110 and the like. More specifically, each function of the CPU 110 is a function realized by the CPU 110 executing a control program stored in the RAM 171 or the like to control each hardware shown in FIG.

  First, the first generation unit 111 receives the first input input to the first liquid crystal panel 140 based on the first electric signal input from the plurality of photosensor circuits 144 via the image processing engine 180. Accept image data. The first generation unit 111 stores the first input image data output from the image processing engine 180 and the first input image data generated by itself in the RAM 171. That is, the first generation unit 111 constantly updates the first input image data in the RAM 171 to the latest first input image data.

  Note that the first generation unit 111 may have a function realized by the CPU 110 and the plurality of photosensor circuits 144 of the liquid crystal panel 140. That is, the first generation unit 111 may be a concept indicating a functional block including a part of the function of the CPU 110 and the light receiving function of the first liquid crystal panel 140.

  The second generation unit 112 receives the second input image data input to the second liquid crystal panel 240 based on the second electrical signal input from the plurality of photosensor circuits 244 via the image processing engine 280. Accept. The second generation unit 112 stores the second input image data output from the image processing engine 280 and the second input image data generated by itself in the RAM 171. That is, the second generation unit 112 constantly updates the second input image data in the RAM 171 to the latest second input image data.

  Note that the second generation unit 112 may have a function realized by the CPU 110 and the plurality of photosensor circuits 244 of the liquid crystal panel 240. That is, the second generation unit 112 may be a concept showing a functional block including a part of the function of the CPU 110 and the light receiving function of the second liquid crystal panel 240.

  The first determination unit 113 reads the first and second input image data from the RAM 171 and determines whether or not at least one of the first and second input images satisfies a preset first condition. To do.

  More specifically, in the first mode, the first determination unit 113 acquires (calculates) the number of fingers in contact with the second liquid crystal panel 240 based on the second input image data. Whether the first condition is satisfied is determined based on the number of fingers. Alternatively, in the first mode, the first determination unit 113 obtains (calculates) the number of fingers in contact with the first liquid crystal panel 140 based on the first input image data, thereby Whether or not the first condition is satisfied is determined based on the number of. Alternatively, in the first mode, the first determination unit 113 determines the number of fingers in contact with the first and second liquid crystal panels 140 and 240 based on both the first and first input image data. By acquiring (calculating), it is determined whether or not the first condition is satisfied based on the number of fingers.

  Further, the first determination unit 113 according to the present embodiment uses the first coordinates of the user's finger (such as a thumb) that is in contact with the first liquid crystal panel 140 based on the first input image data. calculate. The first determination unit 113 calculates the first coordinates of the user's finger (for example, index finger) that is in contact with the second liquid crystal panel 240 based on the second input image data. Then, the first determination unit 113 generates a command indicating that an icon, text, or the like has been selected (command indicating that an object has been pinched) from the first coordinate and the second coordinate.

  More specifically, the first determination unit 113 makes contact with the user's finger at positions corresponding to each other on the first and second liquid crystal panels 140 and 240 based on the first coordinates and the second coordinates. The instruction is generated by judging whether or not. The first determination unit 113 delivers the command to another application such as a word processor or an OS (Operating System).

  The second determination unit 114 reads the first and second input image data from the RAM 171 and determines whether at least the other of the first and second input images satisfies a preset second condition. To do.

  More specifically, in the second mode, the second determination unit 114 acquires (calculates) the number of fingers in contact with the first liquid crystal panel 140 based on the first input image data. Whether the first condition is satisfied is determined based on the number of fingers. Alternatively, in the second mode, the second determination unit 114 obtains (calculates) the number of fingers that are in contact with the second liquid crystal panel 240 based on the second input image data. Whether or not the second condition is satisfied is determined based on the number of. Alternatively, in the second mode, the second determination unit 114 determines the number of fingers in contact with the first and second liquid crystal panels 140 and 240 based on both the first and second input image data. By acquiring (calculating), it is determined whether or not the second condition is satisfied based on the number of fingers.

  The switching unit 115 switches to the second mode when at least one of the first and second input image data satisfies the first condition in the first mode. On the other hand, in the second mode, the switching unit 115 switches to the first mode when at least the other of the first and second input image data satisfies the second condition.

  The display control unit 116 causes the first liquid crystal panel 140 to output an output image in the first mode, and causes the second liquid crystal panel 240 to output an output image in the second mode. More specifically, as shown in FIG. 1A, the display control unit 116 causes the first liquid crystal panel 140 to output the first output image in the first mode. As shown in FIG. 1B, the display control unit 116 causes the second liquid crystal panel 240 to output a second output image in the first mode. As shown in FIG. 1C, the display control unit 116 causes the first liquid crystal panel 140 to output a second output image in the second mode. As shown in FIG. 1D, the display control unit 116 causes the second liquid crystal panel 240 to output the first output image in the second mode.

<Display processing according to the present embodiment>
Next, display processing in electronic device 100 according to the present embodiment will be described with reference to FIGS. 1, 2, 12, and 13. FIG. 13 is a flowchart showing a processing procedure of display processing in electronic device 100 according to the present embodiment.

  First, the CPU 110 functioning as the first liquid crystal panel 140 and the first generation unit 111 acquires a first input image (step S102). Similarly, CPU 110 functioning as second liquid crystal panel 240 and second generation unit 112 acquires a second input image (step S104).

  CPU 110 functioning as switching unit 115 determines whether or not the current mode is the first mode (step S106). When the current mode is the first mode (YES in step S106), CPU 110 functioning as first determination unit 113 causes second liquid crystal panel 240 to be applied to second liquid crystal panel 240 based on the second input image. It is determined whether or not one finger is in contact (step S108). When one finger is not in contact with second liquid crystal panel 240 (NO in step S108), CPU 110 ends the display process (repeats the process from step S102).

  When one finger is in contact with second liquid crystal panel 240 (YES in step S108), CPU 110 functioning as first determination unit 113 selects the first input image based on the first input image. It is determined whether or not a plurality of fingers are in contact with one liquid crystal panel 140 (step S110). If a plurality of fingers are not in contact with first liquid crystal panel 140 (NO in step S110), CPU 110 ends the display process.

  If a plurality of fingers are in contact with first liquid crystal panel 140 (YES in step S110), CPU 110 functioning as switching unit 115 switches from the first mode to the second mode (step S112). Thereby, electronic device 100 shifts from the state shown in FIGS. 1A and 1B to the state shown in FIGS. 1C and 1D.

  On the other hand, when the current mode is not the first mode (NO in step S106), CPU 110 functioning as second determination unit 114, based on the first input image, first liquid crystal panel 140. It is determined whether or not one finger is touching (step S114). If one finger is not in contact with first liquid crystal panel 140 (NO in step S114), CPU 110 ends the display process.

  When one finger is in contact with first liquid crystal panel 140 (in the case of YES in step S114), CPU 110 functioning as second determination unit 114 performs the first operation based on the second input image. It is determined whether or not a plurality of fingers are in contact with the second liquid crystal panel 240 (step S116). If a plurality of fingers are not in contact with second liquid crystal panel 240 (NO in step S116), CPU 110 ends the display process.

  When a plurality of fingers are in contact with second liquid crystal panel 240 (YES in step S116), CPU 110 functioning as switching unit 115 switches from the second mode to the first mode (step S118). Accordingly, electronic device 100 shifts from the state shown in FIGS. 1C and 1D to the state shown in FIGS. 1A and 1B.

<Modification of judgment configuration>
Hereinafter, modified examples of the first determination unit 113B, the second determination unit 114B, and the switching unit 115B according to the present embodiment will be described with reference to FIGS. The electronic device 100 according to the first embodiment switches the mode based on the number of fingers, but the electronic device 100 according to the present modification switches the mode based on the finger shape (image). Is. FIG. 14 is a block diagram illustrating a functional configuration of the electronic device 100 according to the present modification. The description of the functional configuration other than the first determination unit 113B, the second determination unit 114B, and the switching unit 115B will not be repeated here.

  The first determination unit 113B reads out the second input image data and the pattern image data 171A indicating the thumb image from the RAM 171 and determines whether or not the second input image satisfies the first condition set in advance. to decide.

  In the first mode, the first determination unit 113B determines whether or not the thumb is in contact with the second liquid crystal panel 240 based on the second input image data and the pattern image data 171A. The first determination unit 113B determines whether or not the pattern image data 171A is included in the second input image data.

  More specifically, the first determination unit 113B determines that the thumb is in contact with the second liquid crystal panel 240 when determining that the second input image includes a thumb pattern image. When determining that the thumb is in contact with the second liquid crystal panel 240, the first determination unit 113B determines that the first condition is satisfied.

  The second determination unit 114B reads the first input image data and the pattern image data 171A indicating the thumb image from the RAM 171 and determines whether or not the first input image satisfies a preset second condition. to decide.

  In the second mode, the second determination unit 114B determines whether or not the thumb is in contact with the first liquid crystal panel 140 based on the first input image data and the pattern image data 171A. The second determination unit 114B determines whether or not the pattern image data 171A is included in the first input image data.

  More specifically, when determining that the second input image includes a thumb pattern image, the second determination unit 114B determines that the thumb is in contact with the first liquid crystal panel 140. When determining that the thumb is in contact with the first liquid crystal panel 140, the second determination unit 114B determines that the second condition is satisfied.

  In the first mode, the switching unit 115 switches to the second mode when the second input image data satisfies the first condition. On the other hand, in the second mode, the switching unit 115 switches to the first mode when the first input image data satisfies the second condition.

<Display processing according to this modification>
Next, display processing in the electronic device 100 according to the present modification will be described with reference to FIGS. 1, 2, 14, and 15. FIG. 15 is a flowchart showing a processing procedure of display processing in the electronic apparatus 100 according to this modification.

  First, the CPU 110 functioning as the switching unit 115 determines whether or not the current mode is the first mode (step S202). When the current mode is the first mode (YES in step S202), CPU 110 functioning as second liquid crystal panel 240 and second generation unit 112 acquires the second input image ( Step S204).

  The CPU 110 functioning as the second determination unit 114B reads the pattern image data 171A from the RAM 171 and determines whether or not a thumb pattern image is included in the second input image (step S206). If the second input image includes a thumb pattern image (YES in step S206), CPU 110 functioning as switching unit 115 switches from the first mode to the second mode (step S208). . Thereby, electronic device 100 shifts from the state shown in FIGS. 1A and 1B to the state shown in FIGS. 1C and 1D.

  If the input image does not include a thumb pattern image (NO in step S206), CPU 110 ends the display process (repeats the process from step S202).

  If the current mode is not the first mode (NO in step S202), CPU 110 functioning as first liquid crystal panel 140 and first generation unit 111 acquires the first input image (step). S210).

  The CPU 110 functioning as the first determination unit 113B reads the pattern image data 171A from the RAM 171 and determines whether or not a thumb pattern image is included in the first input image (step S212). When the first input image includes a thumb pattern image (YES in step S212), CPU 110 functioning as switching unit 115 switches from the second mode to the first mode (step S214). . Accordingly, electronic device 100 shifts from the state shown in FIGS. 1C and 1D to the state shown in FIGS. 1A and 1B.

  If the first input image does not include the thumb pattern image (NO in step S212), CPU 110 ends the display process.

[Embodiment 2]
Next, a second embodiment of the present invention will be described. In the electronic device 100 according to the first embodiment described above, the display content of the first liquid crystal panel 140 and the display content of the second liquid crystal panel 240 change depending on how the user holds (mode). . For example, the first liquid crystal panel 140 switches between a photographic image and a wallpaper image, and the second liquid crystal panel 240 switches between a wallpaper image and a photographic image depending on how the user holds. In other words, the electronic device 100 switches display contents by paying attention to how the user holds the front and back of the electronic device 100.

  On the other hand, in electronic device 100-2 according to the present embodiment, the display mode of first liquid crystal panel 140 (or second liquid crystal panel 240) changes depending on how the user holds. For example, the first liquid crystal panel 140 displays a photographic image in a portrait orientation or a landscape orientation depending on how the user holds it. In other words, the electronic device 100 switches the display form by paying attention to the way the user holds the device in the vertical and horizontal directions.

  The overall configuration of electronic device 100-2, which is an example of the display device according to the present embodiment, is similar to that of electronic device 100 according to Embodiment 1, and thus description thereof will not be repeated here. Below, the operation | movement outline | summary of the electronic device 100-2 which concerns on this Embodiment is demonstrated.

<Outline of Operation of Electronic Device 100-2 according to Embodiment>
FIG. 16 is a schematic diagram showing electronic apparatus 100-2 according to the present embodiment. More specifically, FIG. 16A is an image diagram showing the electronic device 100-2 in a state where the user holds the electronic device 100-2 in the first vertical state. FIG. 16B is an image diagram showing the electronic device 100-2 in a state where the user holds the electronic device 100-2 in the first horizontal state. FIG. 16C is an image diagram illustrating the electronic device 100-2 in a state where the user holds the electronic device 100-2 in the second vertical state. FIG. 16D is an image diagram illustrating the electronic device 100-2 in a state where the user holds the electronic device 100-2 in the second horizontal state.

<Outline of Operation of Electronic Device 100-2>
Next, with reference to FIG. 16A to FIG. 16D, an outline of operation of electronic device 100-2 according to the present embodiment will be described.

  Referring to FIG. 16A, the user holds electronic device 100-2 in the first vertical state using right hand 99R. At this time, the thumb of the right hand 99R reaches the inside of the first liquid crystal panel 140 from the right side 140R of the first liquid crystal panel 140. In other words, when the thumb of the right hand 99R reaches the inner side of the first liquid crystal panel 140 from the right side 140R, the user sets the electronic device 100 with the right side 140R on the right side and the upper side 140T on the upper side. -2 is likely to be held. In this case, electronic device 100-2 according to the present embodiment causes first liquid crystal panel 140 to display an image with upper side 140T as the upward direction.

  Referring to FIG. 16B, the user changes electronic device 100-2 to the first lateral state using right hand 99R. At this time, the thumb of the right hand 99R has reached the inside of the first liquid crystal panel 140 from the upper side 140T of the first liquid crystal panel 140. In other words, when the thumb of the right hand 99R reaches the inner side of the first liquid crystal panel 140 from the upper side 140T, the user places the electronic device 100 with the upper side 140T on the right side and the left side 140L on the upper side. -2 is likely to be held. In this case, electronic device 100-2 according to the present embodiment displays an image on first liquid crystal panel 140 with left side 140L as the upward direction.

  Referring to FIG. 16C, the user changes electronic device 100-2 to the second vertical state using right hand 99R. At this time, the thumb of the right hand 99R has reached the inside of the first liquid crystal panel 140 from the left side 140L of the first liquid crystal panel 140. In other words, when the thumb of the right hand 99R reaches the inner side of the first liquid crystal panel 140 from the left side 140L, the user places the electronic device 100 with the left side 140L on the right side and the lower side 140B on the upper side. -2 is likely to be held. In this case, electronic device 100-2 according to the present embodiment causes first liquid crystal panel 140 to display an image with lower side 140B as the upward direction.

  Referring to FIG. 16D, the user switches electronic device 100-2 to the second lateral state using right hand 99R. At this time, the thumb of the right hand 99R has reached the inside of the first liquid crystal panel 140 from the upper side 140T of the first liquid crystal panel 140. In other words, when the thumb of the right hand 99R reaches the inner side of the first liquid crystal panel 140 from the upper side 140T, the user places the electronic device 100 with the upper side 140T on the right side and the left side 140L on the upper side. -2 is likely to be held. In this case, electronic device 100-2 according to the present embodiment displays an image on first liquid crystal panel 140 with right side 140R as the upward direction.

  Alternatively, the electronic device 100-2 may be switched to two types of display forms depending on how the user holds the electronic device 100-2. That is, when the thumb reaches the inside of the first liquid crystal panel 140 from the right side 140R or the left side 140L, the electronic device 100-2 sets the upper side 140T as the upward direction and the right side 140R as the right direction, and the first liquid crystal panel. 140 may display an image. When the thumb reaches the inner side of the first liquid crystal panel 140 from the upper side 140T or the lower side 140B, the electronic device 100-2 sets the left side 140L as the upward direction and the upper side 140T as the right direction, and the first liquid crystal panel. 140 may display an image.

  As described above, the electronic device 100-2 according to the present embodiment is based on the input image acquired via the first liquid crystal panel 140, and the user holds the electronic device 100-2 (the electronic device 100 using a finger). -2 holding state), and controls the first liquid crystal panel 140 in a manner (direction) in which the user can easily view the image to be output, depending on how to hold the image. As a result, even if the user does not input a command to change the display mode consciously, the output image is displayed in a user-friendly display mode.

  Hereinafter, a configuration for realizing such a function will be described in detail. Note that the hardware configuration and the like of electronic device 100-2 according to the present embodiment are the same as those of electronic device 100 according to Embodiment 1 as shown in FIGS. Do not repeat.

<Functional configuration of electronic device 100-2 according to the present embodiment>
Hereinafter, a functional configuration of electronic apparatus 100-2 according to the present embodiment will be described with reference to FIGS. FIG. 17 is a block diagram showing a functional configuration of electronic device 100-2 according to the present embodiment. Electronic device 100-2 according to the present embodiment switches between two types of display modes depending on how the user holds.

  Electronic device 100-2 according to the present embodiment includes generation unit 111-2, horizontal determination unit 113-2, vertical determination unit 114-2, switching unit 115-2, and display control unit 116-2. including. In addition, as illustrated in FIG. 2, the electronic device 100-2 includes a RAM 171 and a first liquid crystal panel 140 including a plurality of photosensor circuits 144 and a plurality of pixel circuits 141. Note that the configurations of the plurality of photosensor circuits 144 and the plurality of pixel circuits 141 of the electronic device 100-2 according to this embodiment are the same as those according to Embodiment 1, and thus description thereof will not be repeated here. .

  The RAM 171 according to the present embodiment stores a pattern image of the thumb of the right hand, a pattern image of the thumb of the left hand, and the like. Then, after rotating each pattern image, the CPU 110 performs matching processing between the rotated pattern image and an input image described later.

  Alternatively, the RAM 171 includes a pattern image of the right hand thumb and a left thumb pattern image in the first portrait mode, a pattern image of the right hand thumb and a left thumb pattern in the first landscape mode, The pattern image of the thumb of the right hand and the pattern of the thumb of the left hand in the portrait mode and the pattern image of the thumb of the right hand and the pattern of the thumb of the left hand in the second landscape mode may be stored. Then, the CPU 110 may perform matching processing between each pattern image and an input image described later.

  The generation unit 111-2, the horizontal determination unit 113-2, the vertical determination unit 114-2, the switching unit 115-2, and the display control unit 116-2 are functions realized by the CPU 110 and the like. More specifically, each function of the CPU 110 is a function realized by the CPU 110 executing a control program stored in the RAM 171 or the like to control each hardware shown in FIG.

  The generation unit 111-2 according to the present embodiment receives an input input to the first liquid crystal panel 140 based on a first electrical signal input from the plurality of optical sensor circuits 144 via the image processing engine 180. Accept image data. Then, the generation unit 111-2 stores the input image data output from the image processing engine 180 or the input image data generated by itself in the RAM 171. That is, the generation unit 111-2 constantly updates the input image data in the RAM 171 to the latest input image data.

  Note that the generation unit 111-2 may be a function realized by the CPU 110 and the plurality of photosensor circuits 144 of the liquid crystal panel 140. That is, the generation unit 111-2 may be a concept indicating a functional block including a part of the function of the CPU 110 and the light receiving function of the first liquid crystal panel 140.

  The vertical determination unit 114-2 reads the input image data (and the pattern image data 171A) from the RAM 171 and determines whether the input image satisfies a predetermined vertical condition. More specifically, in the horizontal mode, the vertical determination unit 114-2 determines whether the vertical condition is satisfied based on the input image data (and the pattern image data 171A).

  For example, the vertical determination unit 114-2 determines whether the vertical condition is satisfied by determining whether the object in contact with the first liquid crystal panel 140 has reached the right side 140R or the left side 140L. Judging. Alternatively, the vertical determination unit 114-2 reads the right and left thumb pattern images from the RAM 171 and rotates the pattern images to determine whether the input image includes a vertical thumb pattern image. To do. Alternatively, the vertical determination unit 114-2 reads the pattern images of the right and left thumbs in the vertical mode from the RAM 171 and determines whether or not the input image includes the vertical thumb pattern image.

  The horizontal determination unit 113-2 reads the input image data (and the pattern image data 171A) from the RAM 171 and determines whether or not the input image satisfies a preset horizontal condition. More specifically, in the vertical mode, the horizontal determination unit 113-2 determines whether the horizontal condition is satisfied based on the input image data (and the pattern image data 171A).

  For example, the lateral determination unit 113-2 determines whether or not the lateral condition is satisfied by determining whether or not the object in contact with the first liquid crystal panel 140 has reached the upper side 140T or the lower side 140B. Judging. Alternatively, the lateral determination unit 113-2 reads the right and left thumb pattern images from the RAM 171 and rotates the pattern image to determine whether the input image includes a horizontal thumb pattern image. . Alternatively, the lateral determination unit 113-2 reads out the pattern images of the right and left thumbs in the lateral mode from the RAM 171 and determines whether or not the input image includes the horizontal thumb pattern image.

  The switching unit 115-2 switches to the horizontal mode when the input image data satisfies the horizontal condition in the vertical mode. On the other hand, in the horizontal mode, the switching unit 115-2 switches to the vertical mode when the input image data satisfies the vertical condition.

  The display control unit 116-2 causes the first liquid crystal panel 140 to output the output image in the portrait mode in the portrait mode, and causes the first liquid crystal panel 140 to output the output image in the landscape mode in the landscape mode. More specifically, as shown in FIG. 16A, the display control unit 116-2 causes the first liquid crystal panel 140 to output the output image in the portrait orientation in the portrait mode. As shown in FIG. 16B, the display control unit 116-2 causes the first liquid crystal panel 140 to output an output image in the horizontal direction in the horizontal mode.

<Display processing according to the present embodiment>
Next, with reference to FIG. 1, FIG. 2, FIG. 12, and FIG. 18, display processing in electronic device 100-2 according to the present embodiment will be described. FIG. 18 is a flowchart showing a processing procedure of display processing in electronic device 100-2 according to the present embodiment.

  First, the CPU 110 functioning as the first liquid crystal panel 140 and the generation unit 111-2 acquires an input image (step S302). Next, the CPU 110 functioning as the switching unit 115-2 determines whether or not the current mode is the vertical mode (step S304). When the current mode is the vertical mode (YES in step S304), CPU 110 functioning as horizontal determination unit 113-2 reads a horizontal thumb pattern image from RAM 171 (step S306).

  Then, CPU 110 determines whether or not the input image includes a horizontal thumb pattern image (step S308). If the input image does not include the horizontal thumb pattern image (NO in step S308), CPU 110 ends the display process (repeats the process from step S302).

  When the input image includes a horizontal thumb pattern image (YES in step S308), CPU 110 functioning as switching unit 115-2 switches from the vertical mode to the horizontal mode (step S310). Thereby, electronic device 100-2 (CPU 110 functioning as display control unit 116-2) shifts from the state shown in FIG. 16A to the state shown in FIG.

  On the other hand, when the current mode is not the vertical mode (NO in step S304), CPU 110 functioning as vertical determination unit 114-2 reads a vertical thumb pattern image from RAM 171 (step S312). CPU 110 determines whether or not a vertical thumb pattern image is included in the input image (step S314). If the input image does not include the vertical thumb pattern image (NO in step S314), CPU 110 ends the display process.

  If the input image includes a vertical thumb pattern image (YES in step S314), CPU 110 functioning as switching unit 115-2 switches from the horizontal mode to the vertical mode (step S316). As a result, the electronic device 100-2 shifts from the state illustrated in FIG. 16B to the state illustrated in FIG.

<Modification of judgment configuration>
Hereinafter, with reference to FIG. 1, FIG. 2, FIG. 17, modifications of the vertical determination unit 114-2, the horizontal determination unit 113-2, the switching unit 115-2, and the display control unit 116-2 according to the present embodiment. An example will be described. The electronic device 100-2 according to this modification is switched to four types of display modes depending on how the user holds the electronic device 100-2.

  The electronic device 100-2 according to the present modification includes a generation unit 111-2, a vertical determination unit 114-2, a horizontal determination unit 113-2, a selection unit 117 as a modification of the switching unit 115-2, and display control. Part 116-2.

  The selection unit 117 reads input image data from the RAM 171 and determines whether or not the input image satisfies a first vertical condition that is set in advance. For example, the selection unit 117 determines whether or not the first vertical condition is satisfied by determining whether or not the object in contact with the first liquid crystal panel 140 has reached the right side 140R. . Alternatively, the selection unit 117 determines whether the object is in contact from the right side 140R toward the upper left direction or whether the object is in contact from the left side 140L toward the upper right direction. If the condition is satisfied, it is determined that the first vertical condition is satisfied.

  Alternatively, the selection unit 117 reads the right and left thumb pattern images from the RAM 171 and rotates the pattern images to determine whether or not the input image includes the first vertical thumb pattern image. To do. Alternatively, the selection unit 117 reads the pattern images of the right and left thumbs in the first vertical mode from the RAM 171 and determines whether or not the pattern image is included in the input image.

  The selection unit 117 reads the input image data from the RAM 171 and determines whether or not the input image satisfies a preset first horizontal condition. For example, the selection unit 117 determines whether or not the first lateral condition is satisfied by determining whether or not the object in contact with the first liquid crystal panel 140 has reached the upper side 140T. . Alternatively, the selection unit 117 determines whether any of the conditions is satisfied by determining whether the object is in contact with the upper side 140T from the lower side 140T toward the lower left direction or whether the object is in contact with the lower side 140B from the upper left direction. If it is satisfied, it is determined that the first lateral condition is satisfied.

  Alternatively, the selection unit 117 reads the right and left thumb pattern images from the RAM 171 and rotates the pattern images to determine whether or not the input image includes the first horizontal thumb pattern image. . Alternatively, the selection unit 117 reads the pattern images of the right and left thumbs in the first lateral mode from the RAM 171 and determines whether or not the pattern image is included in the input image.

  The selection unit 117 reads the input image data from the RAM 171 and determines whether or not the input image satisfies the second vertical condition set in advance. For example, the selection unit 117 determines whether or not the second vertical condition is satisfied by determining whether or not the object in contact with the first liquid crystal panel 140 has reached the left side 140L. . Alternatively, the selection unit 117 determines whether or not the object is in contact from the left side 140L in the lower right direction, or whether or not the object is in contact from the right side 140R in the lower left direction. If the second condition is satisfied, it is determined that the second vertical condition is satisfied.

  Alternatively, the selection unit 117 reads the right and left thumb pattern images from the RAM 171 and rotates the pattern images to determine whether or not the input image includes the second vertical thumb pattern image. To do. Alternatively, the selection unit 117 reads the pattern images of the right and left thumbs in the second vertical mode from the RAM 171 and determines whether or not the pattern image is included in the input image.

  The selection unit 117 reads input image data from the RAM 171 and determines whether or not the input image satisfies a preset second horizontal condition. For example, the selection unit 117 determines whether or not the second lateral condition is satisfied by determining whether or not the object in contact with the first liquid crystal panel 140 has reached the lower side 140B. . Alternatively, the selection unit 117 determines whether the object is in contact from the lower side 140B toward the upper right direction or by determining whether the object is in contact from the upper side 140T toward the lower right direction. If the second condition is satisfied, it is determined that the second lateral condition is satisfied.

  Alternatively, the selection unit 117 reads the right and left thumb pattern images from the RAM 171 and rotates the pattern images to determine whether or not the input image includes the second horizontal thumb pattern image. . Alternatively, the selection unit 117 reads the pattern images of the right and left thumbs in the second lateral mode from the RAM 171 and determines whether or not the pattern image is included in the input image.

  When the input image data satisfies the first vertical condition, the switching unit 115-2 switches to the first vertical mode, that is, selects the first vertical mode. When the input image data satisfies the first horizontal condition, the switching unit 115-2 switches to the first horizontal mode, that is, selects the first horizontal mode. When the input image data satisfies the second vertical condition, the switching unit 115-2 switches to the second vertical mode, that is, selects the second vertical mode. When the input image data satisfies the second horizontal condition, the switching unit 115-2 switches to the second horizontal mode, that is, selects the second horizontal mode.

  In the first vertical mode, the display control unit 116-2 displays the output image on the first liquid crystal panel 140 in the first vertical direction (with the upper side 140T on the upper side) as shown in FIG. Output. In the first horizontal mode, the display control unit 116-2 outputs an output image in the first horizontal direction (with the left side 140L on the upper side) on the first liquid crystal panel 140, as shown in FIG. 16B. Let In the second vertical mode, the display control unit 116-2 outputs an output image on the first liquid crystal panel 140 in the second vertical direction (with the lower side 140B on the lower side) as shown in FIG. Is output. In the second horizontal mode, the display control unit 116-2 outputs an output image in the second horizontal direction (with the right side 140R on the upper side) on the first liquid crystal panel 140, as shown in FIG. Let

<Display processing according to this modification>
Next, display processing in the electronic apparatus 100-2 according to the present modification will be described with reference to FIG. 1, FIG. 2, FIG. 14, and FIG. FIG. 19 is a flowchart illustrating a processing procedure of display processing in the electronic device 100-2 according to the present modification.

  First, the CPU 110 functioning as the first liquid crystal panel 140 and the generation unit 111-2 acquires an input image (step S402).

  The CPU 110 functioning as the selection unit 117 reads out a thumb pattern image for the upper side 140T (for the first vertical condition) from the RAM 171 (step S404). The CPU 110 determines whether or not the input image includes a thumb pattern image for the upper side 140T (step S406). When the input image includes a thumb pattern image for upper side 140T (YES in step S406), CPU 110 functioning as switching unit 115-2 switches to the upper side mode (first vertical mode) (step). S408). As a result, the electronic device 100-2 (the CPU 110 functioning as the display control unit 116-2) shifts to the state illustrated in FIG.

  However, in steps S404 to S406, the CPU 110 determines whether or not an object is in contact with the first liquid crystal panel 140 in the upper left direction from the right side 140R based on the input image, or the left side of the first liquid crystal panel 140 It may be determined whether or not the object is in the upper right direction from 140L. If the object is in contact with the first liquid crystal panel 140 in the upper left direction from the right side 140R, or if the object is in contact with the first liquid crystal panel 140 in the upper right direction from the left side 140L, in step S408, You may transfer to the upper side mode.

  If the input image does not include the thumb pattern image for the upper side 140T (NO in step S406), the CPU 110 reads the thumb pattern image for the left side 140L (for the first lateral condition) from the RAM 171. (Step S410). The CPU 110 determines whether or not the input image includes a thumb pattern image for the left side 140L (step S412). When the input image includes a thumb pattern image for left side 140L (YES in step S412), CPU 110 switches to the left side mode (first horizontal mode) (step S414). As a result, the electronic device 100-2 shifts to a state illustrated in FIG.

  However, in steps S410 to S412, the CPU 110 determines whether or not an object is in contact with the first liquid crystal panel 140 in the lower left direction from the upper side 140T based on the input image, or the lower side of the first liquid crystal panel 140. It may be determined whether an object is in contact with the upper left direction from 140B. If the object is in contact with the first liquid crystal panel 140 in the lower left direction from the upper side 140T, or if the object is in contact with the first liquid crystal panel 140 in the upper left direction from the lower side 140B, in step S414, You may shift to the left side mode.

  If the input image does not include the thumb pattern image for the left side 140L (NO in step S412), the CPU 110 reads the thumb pattern image for the lower side 140B (for the second vertical condition) from the RAM 171. (Step S416). CPU 110 determines whether or not the input image includes a thumb pattern image for lower side 140B (step S418). If the input image includes a thumb pattern image for lower side 140B (YES in step S418), CPU 110 switches to the lower side mode (second vertical mode) (step S420). As a result, the electronic device 100-2 shifts to a state illustrated in FIG.

  However, in steps S416 to S418, the CPU 110 determines whether or not an object is in contact with the first liquid crystal panel 140 in the lower right direction from the left side 140L based on the input image, or on the first liquid crystal panel 140. It may be determined whether or not an object is in contact with the lower right direction from the right side 140R. If the object is in contact with the first liquid crystal panel 140 in the lower right direction from the left side 140L, or if the object is in contact with the first liquid crystal panel 140 in the lower left direction from the right side 140R, in step S420. The mode may be shifted to the lower side mode.

  If the input image does not include the thumb pattern image for the lower side 140B (NO in step S418), the CPU 110 reads the thumb pattern image for the right side 140R (for the second lateral condition) from the RAM 171. (Step S422). CPU 110 determines whether or not the input image includes a thumb pattern image for right side 140R (step S424). If the input image includes a thumb pattern image for right side 140R (YES in step S424), CPU 110 switches to the right side mode (second horizontal mode) (step S426). Thereby, the electronic device 100-2 shifts to a state illustrated in FIG.

  However, in steps S422 to S424, the CPU 110 determines whether or not an object is in contact with the first liquid crystal panel 140 in the upper right direction from the lower side 140B based on the input image, or the upper side of the first liquid crystal panel 140 It may be determined whether or not the object is in contact with the lower right direction from 140T. If the object is in contact with the first liquid crystal panel 140 in the upper right direction from the lower side 140B, or if the object is in contact with the first liquid crystal panel 140 in the lower right direction from the upper side 140T, in step S426. The mode may be shifted to the right side mode.

  On the other hand, if the input image does not include the thumb pattern image for right side 140R (NO in step S424), CPU 110 ends the display process (repeats the process from step S402).

<Other embodiments>
It goes without saying that the present invention can also be applied to a case where it is achieved by supplying a program to a system or apparatus. Then, a storage medium storing a program represented by software for achieving the present invention is supplied to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the program code stored in the storage medium It is possible to enjoy the effects of the present invention also by reading and executing.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

  As a storage medium for supplying the program code, for example, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, nonvolatile memory card (IC memory card), ROM (mask ROM, flash) EEPROM, etc.) can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) running on the computer based on the instruction of the program code However, it is needless to say that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Further, after the program code read from the storage medium is written to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 is a schematic diagram illustrating an electronic device according to a first embodiment. It is a block diagram showing the hardware constitutions of an electronic device. It is the figure which showed the structure of the liquid crystal panel, and the peripheral circuit of the said liquid crystal panel. It is sectional drawing of a liquid crystal panel and a backlight. It is the figure which showed the timing chart at the time of operating an optical sensor circuit. It is sectional drawing which showed the structure which a photodiode receives the light from a backlight in the case of a scan. It is the figure which showed schematic structure of the sensing command. It is the figure which showed the value of the data in each area | region of a sensing command, and the meaning content which the said value shows. It is the figure which showed schematic structure of the response data. It is the figure which showed the image obtained by scanning a finger | toe. It is sectional drawing which showed the structure which a photodiode receives external light in the case of a scan. 1 is a block diagram illustrating a functional configuration of an electronic device according to a first embodiment. FIG. 6 is a flowchart illustrating a processing procedure of display processing in the electronic device according to the first embodiment. 6 is a block diagram illustrating a functional configuration of an electronic device according to a modification of the first embodiment. FIG. FIG. 11 is a flowchart illustrating a processing procedure of display processing in the electronic device according to the modification example of the first embodiment. FIG. 6 is a schematic diagram illustrating an electronic apparatus according to a second embodiment. 6 is a block diagram illustrating a functional configuration of an electronic device according to a second embodiment. FIG. FIG. 10 is a flowchart illustrating a processing procedure of display processing in the electronic device according to the second embodiment. FIG. 10 is a flowchart illustrating a processing procedure of display processing in an electronic device according to a modification of the second embodiment.

Explanation of symbols

  100, 100-2 Electronic device, 101 Main device, 102 Display device, 110 CPU, 111 First generation unit, 111-1 generation unit, 112 Second generation unit, 113, 113B First determination unit, 113- 1 vertical determination unit, 114, 114B second determination unit, 114-2 horizontal determination unit, 115, 115B, 115-2 switching unit, 116, 116-2 display control unit, 117 selection unit, 130 driver, 131 scanning signal Line drive circuit, 132 data signal line drive circuit, 133 photosensor drive circuit, 134 switch, 135 amplifier, 140, 240 photosensor built-in liquid crystal panel (liquid crystal panel), 141, 241 pixel circuit, 141b, 141g, 141r subpixel circuit , 143 electrode pair, 143a pixel electrode, 143b counter electrode, 144, 244 photosensor Path, 145, 145b, 145g, 145r photodiode, 146 capacitor, 151A active matrix substrate, 151B counter substrate, 152 liquid crystal layer, 153b, 153g, 153r color filter, 157 data signal line, 161 polarization filter, 162 glass substrate, 163 Light shielding film, 164 orientation film, 171 RAM, 171A pattern image data, 173 memory card reader / writer, 174 communication device, 175 microphone, 176 speaker, 177 operation key, 179 backlight, 180 image processing engine, 181 driver control unit, 182 Timer, 183 Signal processor, 1731 Memory card, DB1, DB2 Data bus.

Claims (8)

A first display panel;
The first display panel includes:
A first photosensor circuit for receiving first incident light;
A first pixel circuit for outputting an output image,
A second display panel disposed on the back side of the first display panel;
The second display panel is
A second photosensor circuit for receiving second incident light;
A second pixel circuit that outputs the output image,
Display control means for causing the first display panel to output the output image in a first mode, and outputting the output image to the second display panel in a second mode;
First generation means for generating a first input image based on the first incident light;
Second generation means for generating a second input image based on the second incident light;
First determination means for determining whether at least one of the first and second input images satisfies a preset first condition;
Second determination means for determining whether at least the other of the first and second input images satisfies a preset second condition;
When at least one of the first and second input images satisfies the first condition, the mode is switched to the second mode, and at least the other of the first and second input images satisfies the second condition. A display device further comprising switching means for sometimes switching to the first mode. The first pixel circuit outputs one of the first and second output images,
The second pixel circuit outputs the other of the first and second output images,
In the first mode, the display control means causes the first display panel to output the first output image, causes the second display panel to output the second output image, and outputs the second output image to the second display panel. 2. The display device according to claim 1, wherein, in the first mode, the first output image is output to the first display panel and the first output image is output to the second display panel. The first determination unit obtains the number of fingers in contact with the second display panel based on the second input image, thereby satisfying the first condition based on the number of fingers. Whether or not
The second determination unit obtains the number of fingers in contact with the first display panel based on the first input image, thereby satisfying the second condition based on the number of fingers. The display device according to claim 1, wherein it is determined whether or not. A storage unit for storing a first pattern image indicating a thumb;
The second determining means determines whether or not a thumb is in contact with the first display panel based on the first input image and the first pattern image in the second mode. To determine whether the second condition is satisfied,
The first determining means determines whether or not a thumb is in contact with the second display panel based on the second input image and the first pattern image in the first mode. The display device according to claim 1, wherein it is determined whether or not the first condition is satisfied. With a display panel,
The display panel includes an optical sensor circuit that receives incident light, and a pixel circuit that outputs an output image,
Display control means for causing the display panel to output the output image in a vertical display mode in the vertical mode, and outputting the output image in a horizontal display mode to the display panel in the horizontal mode;
Generating means for generating an input image based on the incident light;
Vertical determination means for determining whether or not the input image satisfies a predetermined vertical condition;
Lateral determination means for determining whether or not the input image satisfies a preset lateral condition;
Switching to the vertical mode when the vertical determination unit determines that the input image satisfies the vertical condition, and switching to the vertical mode when the horizontal determination unit determines that the input image satisfies the horizontal condition And a display device. A display panel having four sides,
The display panel includes an optical sensor circuit that receives incident light, and a pixel circuit that outputs an output image,
In the first vertical mode, the output image is displayed on the display panel in a display mode with the upper side of the display panel on the upper side. In the second vertical mode, the lower side of the display panel is set on the display panel. The output image is displayed in the display mode, and in the first horizontal mode, the output image is displayed on the display panel in a display mode with the right side of the display panel facing up, and in the second horizontal mode, Display control means for displaying the output image in a display mode with the left side of the display panel on the display panel;
Generating means for generating an input image based on the incident light;
A display device further comprising selection means for selecting one of the first and second vertical modes and the first and second horizontal modes based on the input image. A display method for displaying an output image in a display device having a first display panel, a second display panel disposed on the back side of the first display panel, and an arithmetic processing unit,
The first display panel includes a first photosensor circuit that receives first incident light, and a first pixel circuit that outputs an output image;
The second display panel includes a second photosensor circuit that receives second incident light, and a second pixel circuit that outputs the output image,
The display method is:
The arithmetic processing unit outputting the output image to the first display panel in a first mode;
The arithmetic processing unit outputting the output image to the second display panel in a second mode;
The arithmetic processing unit generating a first input image based on the first incident light;
The arithmetic processing unit generating a second input image based on the second incident light;
The arithmetic processing unit determining whether at least one of the first and second input images satisfies a preset first condition; and
The arithmetic processing unit switching to the second mode when at least one of the first and second input images satisfies the first condition;
The arithmetic processing unit determining whether or not at least the other of the first and second input images satisfies a preset second condition;
The arithmetic processing unit includes a step of switching to the first mode when at least the other of the first and second input images satisfies the second condition. A display program for displaying an output image on a display device having a first display panel, a second display panel disposed on the back side of the first display panel, and an arithmetic processing unit,
The first display panel includes a first photosensor circuit that receives first incident light, and a first pixel circuit that outputs an output image;
The second display panel includes a second photosensor circuit that receives second incident light, and a second pixel circuit that outputs the output image,
The display program is stored in the arithmetic processing unit.
Outputting the output image on the first display panel in a first mode;
Outputting the output image to the second display panel in a second mode;
Generating a first input image based on the first incident light;
Generating a second input image based on the second incident light;
Determining whether at least one of the first and second input images satisfies a preset first condition;
Switching to the second mode when at least one of the first and second input images satisfies the first condition;
Determining whether at least the other of the first and second input images satisfies a preset second condition;
And a step of switching to the first mode when at least the other of the first and second input images satisfies the second condition.

Images