JP2010003325A - Display apparatus and display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a display apparatus capable of performing advanced operations by detecting a plurality of contact places on a display screen and continuously detecting the positions, sizes or the like of the contact places. <P>SOLUTION: By using the display apparatus which performs display with a plurality of display elements arranged in a matrix shape as a display/light receiving display panel unit 120, wherein light receiving elements arranged near the respective plurality of display elements can acquire an image of an object that comes into contact with or approaches a display surface, a light receiving image generation unit 114 generates a light receiving image on the basis of a light receiving signal acquired by a light receiving signal receiver 113, and an image processing/operating unit 115 detects the shape and position of a contact part. A command determination unit 116 determines a command corresponding to the shape and position of the contact part detected by the image processing/operating unit 115, and performs prescribed processing in accordance with the command. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a display device and a display method suitable for application to, for example, a liquid crystal display or an EL (electro-luminescence) display, and more particularly to a technique for efficiently and effectively performing an operation through the display.

  Conventionally, when a touch panel that can be operated by touching a display screen of a display device such as a television receiver is configured, a touch panel separate from the display device is stacked on the display screen. .

  As a configuration using a separate touch panel, for example, there is one in which a transparent and thin input detection device is pasted on a screen. As a means of detecting contact with the surface of the display, touch panels such as a resistive film type that senses changes in pressure and a capacitive type that senses electrical signals due to static electricity that changes due to contact with the human body have been put into practical use. ing. However, these touch panels can basically detect the position of only one point on the surface, and cannot simultaneously detect a plurality of points. Therefore, in an operation input device using a touch panel, a method of determining an instruction from a user based on the position of one point of contact or a method of determining an instruction based on a change in the position of one point has been common. .

  Patent Document 1 accepts a movement input of a designated position in a two-dimensional direction on a display screen, determines the instruction content from the user based on the coordinate change of the designated position from the movement start to the movement end, There is a disclosure of an operation input reception device that performs a process according to the above.

JP 2004-336597 A

  In order to perform operations through the display in an easy-to-understand and efficient manner, the method of directly touching and instructing the display surface is effective, but the conventional touch panel can recognize only one point on the display screen. The operation method is limited, and there is a limit to the processing that can be executed on the screen.

  In order to solve the disadvantages of these conventional touch panels, a touch panel has been devised in which a panel can be divided to detect a plurality of points, but each detects a single point within a limited range. Yes, it is difficult to detect multiple points at free positions on the screen. In addition, a contact position detection device that can detect the position of a plurality of points by providing a plurality of detection units such as an infrared type has been devised, but the size of the device is increased, such as the need to install a detection unit outside the screen, There was a problem of increasing complexity.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a display device that enables a high-level operation by detecting a plurality of contact points on a display screen and continuously detecting the positions and sizes thereof.

  One aspect of the present invention is based on a plurality of display elements for displaying an image on a display screen, a plurality of sensors arranged in the display screen, and outputs from the plurality of sensors. A calculation unit that performs calculation processing to obtain information on the position and size of an object that is in contact with or in proximity, and a plurality of contact or proximity objects that form a region in which the calculation unit is individually connected to each other. An instruction determination unit that determines an instruction of the operator according to the area of the region formed by each contact part and the mutual positional relationship of each contact part when the contact part is detected, and according to the determined instruction It is a display device that executes predetermined processing.

  The display control unit further controls an image displayed on the display screen, and the instruction determination unit further determines an instruction of the operator according to a position on a display surface of each contact unit, and the display The control unit can control the image in accordance with an instruction from the instruction determination unit.

  The instruction determination unit can further determine the instruction of the operator according to the change in the positional relationship between the proximity units and the direction thereof.

  A display control unit that controls an image displayed on the display screen is further provided, and the display control unit can control the image in accordance with an instruction from the instruction determination unit.

  The arithmetic unit binarizes the outputs from the plurality of sensors by comparing each of the outputs with a predetermined threshold, and the set of outputs having a predetermined value with respect to the binarized outputs from the plurality of sensors The noise removal process is performed before the isolated point removal process that excludes the connected components that are smaller than the predetermined threshold among the connected components formed by or the labeling process that adds a unique label to each of the connected components. Can do.

  The noise removal process is a process in which the output from each binarized sensor is set as a target pixel one by one, and the pixel value of the target pixel is specified by the state of several pixels around the target pixel. Can be.

  The arithmetic unit can perform the isolated point removal process before performing the labeling process.

  The instruction determination unit can determine an instruction of the operator based on object information that is information regarding each detected contact unit.

  The object information may include information related to a change in the coordinate position of the contact portion.

  The predetermined process is a process for changing the entire display image displayed on the display element or a process for changing an operation of an application for generating a display image displayed on the display element. The process of changing the entire image includes scrolling the display image or enlarging or reducing the display image, and the process of changing the operation of the application that generates the display image to be displayed on the display element is applied to the display image. Selection or deletion of the included character string can be included.

  One aspect of the present invention also performs a calculation process for obtaining information on the position and size of an object that is in contact with or close to the display screen based on outputs from a plurality of sensors arranged in the display screen. , Depending on the area of each region formed by each contact part and the mutual positional relationship of each contact part when detecting a plurality of contacts or contact parts of adjacent objects that form one individually connected region In this display method, the operator's instruction is determined and predetermined processing is executed in accordance with the determined instruction.

  In one aspect of the present invention, calculation processing for obtaining information on the position and size of an object that is in contact with or close to the display screen is performed based on outputs from a plurality of sensors arranged in the display screen. , When a plurality of contacts or contact parts of adjacent objects that form one individually connected region are detected, the area of the region formed by each contact part and the mutual positional relationship of each contact part In response, the operator's instruction is determined, and a predetermined process corresponding to the determined instruction is executed.

  According to the present invention, various processes can be performed according to changes in the position or proximity position of a plurality of locations on the display screen, or the size or position of a plurality of objects that are in contact with or close to each other. The number of types increases and advanced operations become possible.

It is a functional block diagram which shows the functional structural example of the information processing system to which this invention is applied. It is a functional block diagram which shows the detailed example of the command recognition / issue part 17 of FIG. It is a block diagram which shows the structural example of the display apparatus by one embodiment of this invention. It is a block diagram which shows the example of the display / light-receiving display panel by one embodiment of this invention. It is a connection diagram showing a pixel configuration example according to an embodiment of the present invention. It is a timing diagram which shows the example of a timing of the display and light reception by one embodiment of this invention. It is explanatory drawing which shows the operation example (1) of the display apparatus by one embodiment of this invention. It is a flowchart which shows the image processing example (1) by one embodiment of this invention. It is a flowchart which shows the detailed example of the image process (area calculation) of the processes of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is a flowchart which shows the detailed example of the image process (position calculation) of the processes of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is explanatory drawing which shows the operation example (2) of the display apparatus by one embodiment of this invention. It is a flowchart which shows the image processing example (2) by one embodiment of this invention. It is explanatory drawing which shows the example of operation (3) of the display apparatus by one embodiment of this invention. It is a flowchart which shows the image processing example (3) by one embodiment of this invention. It is a flowchart which shows the detailed example of step S108 among the processes of FIG. It is a figure which shows the specific example of the process result of FIG. FIG. 26 is a diagram illustrating a reason for adopting steps S108g to S108j in FIG. It is a figure which shows the specific example of the process result of step S114 among the processes of FIG. It is a flowchart which shows the detailed example of step S114 among the processes of FIG. It is a figure explaining the example of a judgment technique of the judgment process of step S120 among the processes of FIG. It is a flowchart which shows the detailed example of step S120 among the processes of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure explaining the example of a processing content of judgment processing of Step S122 among processings of FIG. It is a figure explaining the example of a processing content of judgment processing of Step S122 among processings of FIG. It is a figure explaining the example of a processing content of judgment processing of Step S123 among processings of FIG. It is explanatory drawing which shows the example after-operation screen (1) by one embodiment of this invention. It is explanatory drawing which shows the operation example (4) of the display apparatus by one embodiment of this invention. It is a flowchart which shows the image processing example (4) by one embodiment of this invention. It is explanatory drawing which shows the example (2) after operation screen by one embodiment of this invention. It is explanatory drawing which shows the operation example (5) of the display apparatus by one embodiment of this invention. It is a flowchart which shows the image processing example (5) by one embodiment of this invention. It is explanatory drawing which shows the example screen (3) after operation by one embodiment of this invention. It is a flowchart which shows the image processing example (6) by one embodiment of this invention. It is a block diagram which shows the structural example of the personal computer which performs the process with which this invention is applied with software.

  Hereinafter, an embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 is a functional block diagram showing a functional configuration example of an information processing system to which the present invention is applied.

  The information processing system in FIG. 1 includes an image input / output device 1 as an image processing device to which the present invention is applied, and an information processing device 2.

  The image input / output device 1 can display image data generated by itself or image data input from the information processing device 2 as an image recognizable to the operator. Further, the image input / output device 1 can appropriately output various information such as a command for obtaining predetermined image data to the information processing device 2.

  Specifically, the image input / output device 1 is configured to include a display image generation unit 11 to a command recognition / issue unit 17.

  The display image generation unit 11 generates image data for displaying a moving image or a still image on the display unit 13 and provides the generated image data to the display control unit 12. Hereinafter, when it is not necessary to distinguish between a moving image and a still image, they are collectively referred to as an image.

  The display image generation unit 11 exchanges various types of information with the information processing apparatus 2 as necessary. A specific example of information exchanged will be described later.

  The display control unit 12 converts the image data from the display image generation unit 11 into a format and timing format that can be displayed on the screen of the display unit 13 and provides the display unit 13 with the format. The display unit 13 displays the image data from the display control unit 12 on the screen as a recognizable image. As a result, the operator can visually recognize the image.

  The light receiving unit 14 is configured by arranging a plurality of light receiving elements in a matrix. From each of the plurality of light receiving elements, a signal based on the amount of received light (hereinafter referred to as a light receiving signal) is output and provided to the received light image generating unit 15.

  The light receiving unit 14 is disposed, for example, on the same plane as the display unit 13 or overlapped. Therefore, the light receiving unit 14 receives light incident on the display unit 13 while the display unit 13 displays an image.

  In other words, the operator can give a light instruction to the display unit 13 while observing the image displayed on the display unit 13. At this time, the light is incident on each light receiving element of the light receiving unit 14, and a light receiving signal having a level corresponding to the instruction is output from each light receiving element. Here, “instruction by light” refers to changing the amount of incident light of at least a part of the plurality of light receiving elements of the light receiving unit 14. In this case, the light source of the incident light is not particularly limited, and the change amount, change rate, change direction (bright → dark direction, or dark → light direction) of the incident light amount is not particularly limited. For example, an instruction by irradiating the light receiving unit 14 with light from an external light source (instruction to increase the amount of light), an instruction to block light from the external light source and give it as a shadow (instruction to decrease the amount of light), or a light source As described above, various instructions such as an instruction using the reflected light of the irradiation light from the display unit 13 instead of an external light source can be adopted as the “instruction by light”.

  The light reception image generation unit 15 is based on each light reception signal output from each of the plurality of light receiving elements of the light reception unit 14, and image data of one still image (hereinafter referred to as light reception image data, corresponding to the still image) Is referred to as a received light image) and provided to the object detection unit 16. More precisely, since the light reception signals from the respective light receiving elements of the light receiving unit 14 are sequentially output, the light reception image generation unit 15 continuously generates a plurality of light reception image data. The images are sequentially provided to the object detection unit 16 one by one. That is, the light reception image generation unit 15 generates stream data of a moving image composed of a plurality of fields or frames, for example, using one light reception image as a field or frame, and provides it to the object detection unit 16. In other words, hereinafter, when simply referred to as a received light image, it means one frame or field.

  The object detection unit 16 detects one or more objects from the received light image data from the received light image generation unit 15.

  Here, the object is an area in the received light image and is an area formed according to a predetermined rule. For example, assuming that the light-receiving image is composed of a plurality of pixels, an aggregate of one or more pixels (hereinafter referred to as a connection element) connected according to a predetermined rule is an example of the object.

  The object detection unit 16 further generates various types of information regarding each object (hereinafter, the various types of information are collectively referred to as object information). For example, various attribute information possessed by the object, more specifically, for example, the coordinates of the center of gravity, the coordinates of the center, the area, the shape thereof, and the like are generated by the object detection unit 16 as object information.

  Each object information generated by the object detection unit 16 is provided to the command recognition / issue unit 17.

  The object information generation function need not be included in the object detection unit 16 and may be delegated to a command recognition / issuance unit 17 described later. In this case, the object detection unit 16 detects the object and provides the detection result to the command recognition / issue unit 17. Then, the command recognition / issuance unit 17 generates object information based on the detection result.

  In any case, based on each object information, the command recognition / issuance unit 17 recognizes what the command “light instruction” given to the display unit 13 by the operator is, that is, The operator recognizes a command given to the image input / output device 1 and issues the command to the display image generation unit 11.

  A detailed example and command example of the command recognition / issue unit 17 will be described later with reference to FIG.

  When a predetermined command is issued from the command recognition / issuance unit 17 in this way, the display image generation unit 11 generates new image data to change the display image of the display unit 13 in accordance with the predetermined command. Provided to the display control unit 12. Then, the display image on the display unit 13 is updated according to the predetermined command.

  The display image generation unit 11 may provide the command issued from the command recognition / issuance unit 17 to the external information processing apparatus 2 without generating display image data by itself. In this case, the information processing apparatus 2 changes the image data to be displayed according to the obtained command, that is, generates new image data and provides it to the display image generation unit 11. The display image generation unit 11 provides the image data to the display control unit 12. Then, also in this case, the display image of the display unit 13 is updated according to the command. In other words, the image input / output device 1 is not particularly required to have the image data generation function, and can be delegated to the external information processing device 2.

  Depending on the screen display state of the display unit 13, it may be necessary to issue a command only at certain times. For example, a software button appears at the right end of the screen. In this case, the command recognition / issuance unit 17 provides the object information to the information processing apparatus 2 via the display image generation unit 11 without recognizing the command itself, and causes the information processing apparatus 2 to recognize the command. You may do it. Alternatively, the information processing apparatus 2 or the display image generation unit 11 recognizes the coordinate position of the software button in the display image and the presence / absence of the software button, and the command recognition / issue unit issues the recognition result as indicated by a dotted arrow. 17, a command definition may be newly added, and the command recognition / issue unit 17 may determine whether a command is issued.

  FIG. 2 shows a detailed functional configuration example of the command recognition / issuing unit 17 in the image input / output device 1 of FIG.

  The command recognition / issue unit 17 in the example of FIG. 2 is configured to include a detection result holding unit 21 to a command issue unit 27.

  Each object information output from the object detection unit 16 includes a detection result holding unit 21, a time position change type command recognition unit 22, a positional relationship type command recognition unit 23, a shape type command recognition unit 24, and a composite type command recognition unit 25. Provided to each of the.

  The command definition holding unit 26 holds one or more conditions for determining a predetermined command based on predetermined object information.

  In the present embodiment, the following four types of commands are defined according to the type of object information to be determined.

  That is, the first type of command is a command that is determined based on the shape of the object in the object information. Hereinafter, such a first type command is referred to as a shape type command.

  The second type of command is a command determined based on the coordinate position of the object in the object information. The second type of command includes a command that is determined based only on the coordinate position of one object and a command that is determined based on the relationship between the coordinate positions of a plurality of objects. Hereinafter, such a second type of command is referred to as a positional relationship type command.

  The third type of command is a command that is determined based on a change in the time position of the object in the object information. That is, as described above, the light reception image generation unit 15 sequentially generates a plurality of light reception image data sequentially in time. A command that is determined based on the relationship of changes in the coordinate position of the same object in two or more received light images having a time difference among a plurality of received light images corresponding to them is a third type command. Hereinafter, such a third type of command is referred to as a time position change type command.

  The fourth type of command is a command that is determined using a combination of two or more of the above-described object shape, object coordinate position, and object temporal position change in the object information. It is. Hereinafter, such a fourth type command is referred to as a composite type command.

  Further, details of each of the shape type command, the positional relationship type command, the time position change type command, and the composite type command will be described individually in that order.

  First, the shape type command will be described.

  One or more shapes of the object are associated with one shape type command, and this correspondence is held in the command definition holding unit 26 as a condition. Therefore, the shape-type command recognition unit 24 acquires the shape of the object in the object information provided from the object detection unit 16 and compares the shape with various conditions held in the command definition holding unit 26. . When the condition that matches the shape is held in the command definition holding unit 26, the shape type command recognition unit 24 associates the shape type command specified by the condition, that is, the shape. It recognizes that the shape type command is a command instructed by the operator, and notifies the command issuing unit 27 of the recognition result.

  Here, the shape of the object is a broad meaning including the shape of an object such as a circle or square, the length or width of a predetermined portion of the object, the size (area) of the object, or a combination of one or more thereof. It is a concept.

  A specific example in which the area of the object is employed as the shape of the object and a shape type command associated with the area of the object is employed will be described later with reference to FIG.

  Next, the positional relationship type command will be described.

  Predetermined coordinate positions of one object or predetermined relationships of coordinate positions of a plurality of objects are associated with one positional relationship type command, and the command definition is retained as a condition. Held in the portion 26. Therefore, the positional relationship type command recognition unit 23 acquires each coordinate position of each object in the object information provided from the object detection unit 16, and holds the command definition and the relationship between the coordinate position alone or two or more coordinate positions. The various conditions held in the unit 26 are respectively compared. If the command definition holding unit 26 holds a condition that matches a predetermined one of the coordinate positions or a predetermined relationship between two or more coordinate positions, the positional relationship type command recognition unit 23 is a positional relationship type command specified by the condition, that is, a positional relationship type command associated with a predetermined position of each coordinate position or a predetermined relationship between two or more coordinate positions, It recognizes that the command is instructed by the operator, and notifies the command issuing unit 27 of the recognition result.

  The coordinate position of the object is not particularly limited as long as the object can be uniquely specified in the received light image. For example, each coordinate position such as the center of gravity, center, and predetermined end point of the object can be adopted as the coordinate position of the object. In addition, when the light reception image produced | generated by the light reception image generation part 15 consists of a some pixel, the predetermined pixel position in a light reception image is employ | adopted as a coordinate position of an object.

  In addition, the predetermined relationship between the coordinate positions of a plurality of objects is the absolute positional relationship between the points indicated by the coordinate positions, the relative positional relationship between the points in the received light image, or a combination of these. It is a broad concept including

  As the former relationship, that is, the absolute positional relationship between the points, for example, a relationship based on the absolute distance between the points or a relationship based on the shape formed by connecting the points can be employed. A specific example in which a relationship based on the absolute distance of each point is employed and a positional relationship type command associated with the relationship is employed will be described later with reference to FIG. In addition, as a relationship based on the shape formed by connecting the points, for example, a relationship in which the points are arranged in a substantially straight line (lined up vertically or horizontally) is adopted, and is associated with the relationship. A specific example when the positional relationship type command is employed will be described later with reference to FIGS.

  On the other hand, the latter relationship, that is, the relative positional relationship between the points, for example, the positional relationship of the points with respect to the received light image, specifically, for example, all of the points are below the received light image or The relationship of being on the upper side can be adopted. A specific example when the positional relationship type command associated with such a relationship is employed will be described later with reference to FIG. More precisely, in the example of FIG. 22, the former relationship in which three points are arranged side by side (see step S114) and the three points that are arranged side by side are the screen (still image). A combination with the latter relationship (see step S116) of being on the upper side or the lower side is employed, and a positional relationship type command of “scrolling the display screen in a predetermined direction” associated with the relationship of the combination is employed. This is an example.

  Next, the time position change type command will be described.

  A predetermined time position change of the object is associated with one time position relation type command, and this correspondence relation is held in the command definition holding unit 26 as a condition. Therefore, the time position change type command recognition unit 22 acquires the coordinate position of the object in the object information provided from the object detection unit 16, and the past position of the same object held in the detection result holding unit 21 is acquired. One or more coordinate positions are acquired, time position changes of those coordinate positions are recognized, and the changes are compared with various conditions held in the command definition holding unit 26. If the command definition holding unit 26 holds a condition that matches the change, the time position change type command recognition unit 22 determines the time position change type command specified by the condition, that is, the time position. The time position change command associated with the change is recognized as a command instructed by the operator, and the recognition result is notified to the command issuing unit 27.

  The object coordinate position may be basically the same as the information used in the positional relationship type command recognition unit 23.

  The coordinate position of the object provided from the object detection unit 16 is a light reception image to be noted as a processing target among a plurality of light reception images (frames or fields) constituting a moving image (hereinafter referred to as a light reception image of interest). The coordinate position in the above). On the other hand, the past coordinate position of the object provided from the detection result holding unit 21 refers to the coordinate position in the received light image processed in the past from the noticed received light image.

  That is, in the detection result holding unit 21, one piece of received light image data is generated by the received light image generation unit 15, and then the object information is updated each time the object information is generated by the object detection unit 16. Thereby, in the time position change type command recognition unit 22, the coordinate position (latest data) of the object in the noticed light reception image and the coordinate position (one before) of the same object in the light reception image immediately before the noticeable light reception image, for example. Comparison with the data). Of course, the detection result holding unit 21 can not only keep the previous state but also hold object information for each of a plurality of past received light image data. In this case, Comparison with a certain time in the past is also possible.

  Further, the time position change of the object recognized by the time position change command recognition unit 22 is not particularly limited as long as it is information that can express the time change, and the movement direction, the movement amount, and the movement vector combining them are possible. The moving speed can be adopted.

  A specific example when the time position change command is employed will be described later with reference to FIG. More precisely, in the example of FIG. 24, the positional relationship (step S120) in which the three points are arranged in a substantially circular shape, and the time position at which the circumference moves in the enlargement or reduction direction. This is an example in which a combination with a change (see step S123) is employed, and a composite command “enlarge / reduce the display screen” associated with the combination is employed.

  Next, the composite command will be described.

  One compound type command is associated with a combination of two or more of the object shape, coordinate position, time position change, etc. as a condition, and the condition is stored in the command definition holding unit 26. Is retained. Therefore, the composite command recognition unit 25 acquires various object information provided from the object detection unit 16 and, if necessary, past object information (coordinate positions, etc.) held in the detection result holding unit 21. , And various combinations using them are compared with various conditions held in the command definition holding unit 26. When the condition that matches the predetermined combination is held in the command definition holding unit 26, the composite command recognition unit 25 determines that the composite command specified by the condition is a command instructed by the operator. And notifies the command issuing unit 27 of the recognition result.

  As described above, the command definition holding unit 26 holds definitions (conditions) for determining whether or not the object information provided from the object detection unit 16 is associated with a predetermined command. . The command definition holding unit 26 is not limited to the definition set in advance, but as indicated by the arrows drawn from the time position change type command recognition unit 22 to the composite type command recognition unit 25 to the command definition holding unit 26. It is also possible to hold a new definition based on object information generated by the operation of the operator. Further, as described above, when a comparison with the screen display state of the display unit 13 is necessary, for example, when a software button appears at the right end of the screen, the display definition generating unit 11 sends a command definition holding unit. As indicated by an arrow drawn at 26, the command definition holding unit 26 can also acquire and hold a new definition from the display image generation unit 11 or the information processing apparatus 2.

  The command issuing unit 27 issues a command recognized by any one of the time position change type command recognition unit 22 to the composite type command recognition unit 25 to the display image generation unit 11.

  It is possible to delegate the function of the composite command recognition unit 25 to the command issuing unit 27. In this case, the command issuing unit 27 comprehensively determines each recognition result of the time position change type command recognizing unit 22 to the shape type command recognizing unit 24, and finally determines what command is issued from the operator. The command is issued to the display image generation unit 11. In particular, when a state occurs in which two or more conditions among the shape, positional relationship, and temporal position change of the object overlap, the command issuing unit 27 determines the meaning and determines the final command, that is, the above-described state. It is responsible for issuing the combined command.

  One functional block shown in FIG. 1 or FIG. 2 described above may be composed of hardware alone, software alone, or a combination thereof. . The functional blocks shown in FIG. 1 or FIG. 2 are not particularly illustrated, but may be configured as functional blocks combined with other functional blocks, or may be divided and configured as a plurality of small functional blocks. .

  The following examples of FIGS. 3 to 43 (hereinafter referred to as this example) are examples in which the image input / output device 1 having the functional configuration of FIG. 2 is applied to a display device configured as a liquid crystal display. In this example, a light receiving element is arranged adjacent to each light emitting element constituting the liquid crystal display so that light emission (display) and light reception (reading) can be performed in parallel. Here, the display of this example capable of performing light emission (display) and light reception (reading) in parallel is referred to as a display / light reception display. That is, an example of a combination of the display unit 13 and the light receiving unit 14 of FIG. 1 is the display / light receiving display of this example. In addition, as will be described later, the display / light-receiving display of this example can detect not only a touch that touches the screen but also an object close to the screen. In some cases, proximity detection is also included, unless otherwise specified.

  FIG. 3 is a block diagram illustrating a configuration example of the display device of this example. In this example, a case will be described in which the input / output processing unit 101 that performs input / output processing of the display device and the document processing unit 102 that edits information such as a document displayed on the screen are described.

  For example, the display signal generation unit 110 generates display data for displaying information such as a document requested from the document processing unit 102 or an image requested from another application, and the display data is displayed on the signal control unit 111. Send to. The signal control unit 111 sends display data to the display signal driver 112, and the display signal driver 112 performs driving for displaying an image on the display / light receiving display panel unit 120.

  The display / light-receiving display panel unit 120 (referred to as a display / light-receiving panel) is configured as a liquid crystal display, and a transparent electrode or the like is disposed on a transparent substrate such as a glass substrate to display a display area (sensor area) 121 (FIG. 4) is a display in which a plurality of pixels (display elements) are formed in a matrix, and a backlight (not shown) is arranged on the back. The backlight of this example uses, for example, an array of a plurality of light emitting diodes, and can turn on / off the backlight at a relatively high speed. On / off control of lighting of the backlight is performed in conjunction with display driving by the display signal driver 112. The driving of the liquid crystal display by the display signal driver 112 is performed by applying a driving voltage signal to the pixel electrodes constituting the display.

  The display / light receiving panel unit 120 includes a plurality of light receiving elements separately from the display elements. That is, for example, light receiving elements are arranged in a matrix adjacent to each display pixel in the display area (sensor area) 121, and signal charges accumulated corresponding to the amount of light received by the light receiving elements are converted into light receiving signals. It is configured to read by driving from the receiver 113.

  The signal control unit 111 sends display data to the display signal driver 112 and instructs the display-side scanner 127 and the light-receiving-side scanner 128 to drive the display and light-receiving scanning lines.

  The light reception signal receiver 113 reads the light input to the display / light reception panel unit 120 as a light reception signal and sends it to the light reception image generation unit 114. The received light image generation unit 114 generates, as image data, the state of an object that touches or approaches the display screen based on the received light signal. The received light image generation unit 114 has a storage device (frame memory (not shown)) and stores image data for each frame. This image data is subjected to image processing by the image processing calculation unit 115, and determination is made of changes in the shape, size, position, etc. of an object that is in contact with or in proximity. When this determination is made, a difference from the determination result of the previous frame is determined to detect a change in the contact state. In this example, determination is also made for simultaneous contact at a plurality of locations. The determination result in the image processing calculation unit 115 is sent to the instruction determination unit 116, where it is determined how the instruction is given based on the determination result, and is sent to the display signal generation unit 110. The display signal generation unit 110 performs predetermined processing according to the application being executed. An example of a specific processing state will be described later.

  That is, the input / output processing unit 101 is an example of the image input / output device 1 of FIG. Specifically, the display signal generation unit 110 is an example of the display image generation unit 11, the signal control unit 111 and the display signal driver 112 are examples of the display control unit 12, and the display / light receiving panel unit 120 (display side) The scanner 127 and the light-receiving side scanner 128 may be included) are examples of the combination of the display unit 13 and the light-receiving unit 14, and the light-receiving signal receiver 113 and the light-receiving image generation unit 114 are examples of the light-receiving image generation unit 15. The image processing calculation unit 115 is mainly an example of the object detection unit 16, and the instruction determination unit 116 is mainly an example of the command recognition / issue unit 17. Note that “mainly” is described because the image processing calculation unit 115 may have some functions of the command recognition / issuance unit 17.

  Next, with reference to FIG. 4, an example of the arrangement of drivers in the display / light receiving panel unit 120 of this example will be described. As shown in FIG. 3, the display / light receiving panel unit 120 emits (displays) and receives (reads) image data through a display signal driver 112 for screen display and a received light signal receiver 113 for receiving light. As an actual configuration, both are configured by two sets of devices in the horizontal direction and the vertical direction in order to correspond to a two-dimensional image.

  As shown in FIG. 4, the display / light-receiving panel unit 120 has a transparent display area (sensor area) 121 disposed in the center, and a display horizontal driver 122 and a display vertical driver 123 on four end faces of the display area 121. The sensor horizontal receiver 125 and the sensor vertical receiver 124 are arranged. A display signal and a control clock are supplied as display data to the display horizontal driver 122 and the display vertical driver 123, and the display pixels arranged in a matrix in the display area 121 are driven. A reading clock is supplied to the sensor horizontal receiver 125 and the sensor vertical receiver 124, and a light reception signal read in synchronization with the clock is supplied to the light reception image generation unit 114 via the light reception signal line. To do.

  FIG. 5 is a diagram showing one configuration of the pixels arranged in the display area 121. As a configuration for display provided in one pixel 131, here, a gate electrode 131h is arranged in the horizontal direction, a drain electrode 131i is arranged in the vertical direction, and a switching element 131a is arranged at the intersection of both electrodes. The switching element 131a and the pixel electrode 131b are connected. The switching element 131a is turned on / off by a signal obtained through the gate electrode 131h, and a display state at the pixel electrode 131b is set by a signal supplied through the drain electrode 131i.

  A light receiving sensor (light receiving element) 131c is arranged at a position adjacent to the pixel electrode 131b, and the power supply voltage VDD is supplied. A reset switch 131d and a capacitor 131e are connected to the light receiving sensor (light receiving element) 131c, and after being reset by the reset switch 131d, charges corresponding to the amount of received light are accumulated by the capacitor 131e. A voltage proportional to the accumulated electric charge is supplied to the signal output electrode 131j via the buffer amplifier 131f at the timing when the readout switch 131g is turned on, and is output to the outside. On / off of the reset switch 131d is controlled by a signal obtained at the reset electrode 131k, and on / off of the readout switch 131g is controlled by a signal obtained at the readout control electrode 131m.

  FIG. 6 is a diagram illustrating a state in which image display (light emission) and light reception are performed in each frame period. In FIG. 6, the horizontal axis represents time, and the vertical axis represents the position of a scanning line (horizontal line) that performs display and light reception. Here, rewriting of the display signal and reading of the light reception signal are performed at the bottom of one screen. The scanning line is changed upward in order from the line, and finally the scanning of the uppermost line (first line) is performed. FIG. 6 shows the processing of the nth frame, which is an arbitrary frame position, and the processing of the (n + 1) th frame, which is the next frame of the nth frame, and the same processing is continuously performed.

  Here, one frame period is, for example, 1/60 seconds, and as shown in FIG. 6, the one frame period is divided into two parts, the first half and the second half, and the first half is a period during which the backlight is turned on. The second half is a period for turning off the backlight. The light reception signal is read out in each of the lighting period and the extinguishing period.

  Further, the backlight on period and the backlight off period are each divided into two equal parts. In the nth frame, the pixel electrode drive line G1 for display scans the lower half of the screen in the first half of the backlight on period and rewrites the display state of the line to an image of the frame period. In the second half of the backlight-on period, the scanning line is not changed and the rest period is set. In the first half of the backlight off period, the upper half of the screen is scanned to rewrite the display state of the line to the image of the frame period, and in the second half of the backlight off period, the scan line is changed. Without a rest period.

  As for the light receiving process, in the nth frame, the process RS1 for sequentially resetting the light receiving signals of all the lines is performed in the first half of the backlight on period, and the light receiving signals of all the lines are processed in the second half of the backlight on period. Processing RD1 is sequentially read, and the received light signals accumulated for a certain period by each sensor are read. Similarly, in the first half of the backlight off period, a process RS2 for sequentially resetting the light reception signals of all lines is performed, and in the second half of the backlight off period, a process RD2 for sequentially reading the light reception signals of all lines is performed, Read the received light signal accumulated for a certain period by each sensor.

  In this way, two readings are performed per frame, that is, reading the received light signal when the backlight is on and so-called self-emission, and reading the received light signal when the backlight is off and the light is turned off. The two readout signals of one frame are input to a frame memory (not shown) in the received light image generation unit 114, and a difference is detected for each signal at each pixel position. The difference received light signal is sent to the image processing calculation unit 115.

  FIG. 7 is a diagram illustrating an example in which the display device of this example is operated with a finger. Here, the display device is configured as a small and thin display device 130 that can be carried by a user (operator), for example, and the operator operates fingers f1 and f2 on the display surface of the display area 121 of the display / light receiving panel unit 120. , F3, etc., and can be operated by touching. As described above, when a reflective object such as a finger touches or approaches the surface of the display area 121 of the display / light-receiving panel unit 120, the reflected light is irradiated by the image displayed on the panel unit and reflected. Returns to the display / light-receiving panel unit 120 and receives the light to detect the shape and position of the reflector. When the operator touches the right edge of the display area 121 with three fingers f1, f2, and f3 as shown in the upper part of FIG. 7, a received light image as shown in the lower part of FIG. 7 is obtained. In this received light image, three contact portions 141, 142, and 143 are detected. More precisely, the received light image in the lower part of FIG. 7 is an image (see FIG. 12) after the image binarization process described later with reference to FIGS.

  The processing for detecting the shape and position of the contact portion from the received light image as shown in FIG. Then, the image processing calculation unit 115 sends information on the detected shape and position of the contact part to the instruction determination unit 116, and determines instructions corresponding to the shapes and positions of the plurality of contact parts detected by the instruction determination unit 116. The instruction is sent to the display signal generation unit 110.

  FIG. 8 is a flowchart illustrating an example of processing for determining an operation instruction from the received light image in the image processing calculation unit 115 and the instruction determination unit 116. With reference to this flowchart, a process for determining an instruction from the operator based on the detected positional relationship between the plurality of contact portions will be described. Here, a case will be described in which the correspondence between the contact portion and the operation instruction is defined as “when three fingers are simultaneously touched in the vertical direction, the display image is scrolled in that direction”.

  In other words, of the command recognition / issuance unit 17 of FIG. 2 configured as the instruction determination unit 116 in the example of FIG. 3, the positional relationship type command recognition unit 23, the command definition holding unit 26, and the command issuance unit 27. An example of the processing when the above functions mainly is shown in the flowchart of FIG. That is, the definition of “when three fingers are simultaneously touched in the vertical direction and the display image is scrolled in that direction” is held in the command definition holding unit 26, and the positional relationship type command recognition unit 23 Based on this definition, an example of processing when the command issuing unit 27 recognizes a command “scroll the display image leftward” or “scroll the display image rightward” and issues the command is shown in FIG. 8 is a flowchart.

  First, an area threshold A for detecting a contact portion is set (step S101). The threshold A is set so that it can be determined whether or not the operator's finger is in contact with the screen, and the area is set according to the operation target such as a finger or an operation instruction pen. . Next, the received light image data (hereinafter abbreviated as image data as appropriate) converted from the received light signal by the received light image generation unit 114 is acquired (step S102), and the area of the contact portion is calculated as the first image processing (step S102). Step S103). A detailed example of the process in step S103 will be described later with reference to FIGS. As a result of the area calculation, a point that is equal to or larger than the area threshold A in the image data (the point in FIG. 8 does not mean one pixel, but will be described later with reference to one object described above, that is, FIG. 9 and the like. It is determined whether there is one connected component (step S104). If there is a point, the process proceeds to the next process, and if not, the next image data is acquired. If there is a point of area A or more, it is determined whether or not there are 3 points (step S105).

  Next, as the second image processing, the positional relationship between the three contact portions is calculated (step S107). A detailed example of the process in step S107 will be described later with reference to FIGS. As a result of the position calculation, it is determined whether the three points are substantially aligned in the vertical direction (step S108). If the points are approximately aligned in the vertical direction, the process proceeds to the next process. Step S109). If the three points are almost aligned in the vertical direction, it is determined where the point is on the screen (step S110), and if it is on the left side of the screen, an instruction to scroll the display screen to the left is given ( If it is located on the right side of the screen (step S111), it is instructed to scroll the display screen in the right direction (step S112).

  For example, as shown in the upper part of FIG. 7, when the operator touches the right edge of the display area 121 with three fingers f1, f2, and f3, a received light image as shown in the lower part of FIG. 7 is obtained. Three contact portions 141, 142, and 143 are detected. In this example, the area of the contact portion is greater than or equal to the threshold value A, and the three contact portions are arranged substantially in the vertical direction and are positioned on the right side of the screen, so that the display screen can be scrolled to the right.

  Here, the details of the image processing (area calculation) in step S103 of FIG. 8 and the image processing (position calculation) in step S107 will be described individually in that order.

  In the following description, it is assumed that the image processing (area calculation) in step S103 in FIG. 8 and the image processing (position calculation) in step S107 are both executed by the image processing calculation unit 115 in FIG. However, at least a part of the image processing (area calculation) in step S103 and the image processing (position calculation) in step S107 in FIG. 8 may be executed by the instruction determination unit 116.

  FIG. 9 is a flowchart for explaining a detailed example of the image processing (area calculation) in step S103.

  In step S103a, the image processing calculation unit 115 executes an image binarization process.

  Specifically, for example, assume that image data corresponding to the received light image shown in FIG. 10 is acquired in step S102 of FIG. This image data is the result of recalibrating the output from each light receiving element arranged on the display surface (display / display surface of the light receiving panel section 120 in FIG. 3) in the state of the upper part in FIG. 7 described above. It is obtained. In the example of FIG. 10, the larger the output of the light receiving element is, the more white the light is output, and the output of the light receiving element from the place where the finger is in contact with or close to the light is large. The output of is designed to be a small value.

  The signal level (luminance) of each pixel in the horizontal line of the image data corresponding to the horizontal line L of the received light image in FIG. 10 is shown in the upper part of FIG. In this case, the image processing calculation unit 115 compares each signal level (each pixel value) of the horizontal line with the slice level (S) as an image binarization process, and a pixel that is equal to or higher than the slice level (S). Is assigned a pixel value of “1”, and if it is less than the slice level (S), a process of attaching a pixel value of “0” is executed. The result of such image binarization processing is shown in the lower part of FIG. In the example of FIG. 11, the slice level (S) is fixed, but it is not necessarily fixed. For example, the slice level (S) may be derived from the average value of the entire image data, or a filter value such as averaging created from the image data may be calculated as the slice level (S).

  FIG. 12 shows an image obtained as a result of performing such an image binarization process on all horizontal lines of image data corresponding to the received light image of FIG. 10 in the same manner. That is, as described above, the image of FIG. 12 is a diagram obtained when the operator touches the right edge of the display area 121 with three fingers f1, f2, and f3 as shown in the upper part of FIG. FIG. That is, each of the three contact portions 141, 142, and 143 described in FIG. 7 is more precisely the regions 141, 142, and 143 including many pixels (white pixels) having a pixel value “1”. Say.

  However, in each of the areas 141, 142, and 143 in FIG. 12, pixels (black pixels) having a pixel value of “0” exist, and the boundaries between the areas 141, 142, and 143 are ambiguous. . Here, if it is defined that a region composed of a collection of pixels having a pixel value “1”, that is, a connected component of pixels having a pixel value “1” is detected as an object, the regions 141, 142, and 143 in the state of FIG. Is not yet a detected object. This definition is adopted because the object information can be easily created. For example, the area calculation of the object (see step S103e described later) and the gravity center position of the object (see FIGS. 17 to 20). This is because it can be easily performed.

  Therefore, since the object (connected component) corresponding to each of the areas 141, 142, and 143 in FIG. 12 and the finger object candidate is detected, the processing in steps S103b to S103d in FIG. 9 is further executed. is there.

  That is, the image processing calculation unit 115 in FIG. 3 executes noise removal processing in step S103b, performs isolated point removal processing in step S103c, and further executes labeling processing in step S103d. As a result, several connected components that are finger object candidates are generated.

  The reason why the noise removal process is executed in the process of step S103b is as follows.

  That is, when the image data (see FIG. 10) output from the received light image generation unit 114 in FIG. 3 includes noise due to variations of the respective light receiving elements of the display / light receiving panel unit 120 and the surrounding environment. There are many. This noise may remain after the image binarization process (step S103a). If a lot of such noise remains, the isolated point removal process (step S103c) or the labeling process (step S103d) to be performed later will be performed. The throughput may increase significantly. In order to prevent this, the noise removal process is executed in the process of step S103b.

  Various methods have been proposed for the noise removal processing, and any method may be adopted. Here, it is assumed that the following method is adopted. That is, a method is employed in which the state of several pixels around a pixel to be processed as a processing target (hereinafter referred to as a pixel of interest) is investigated, and the pixel value of the pixel of interest is identified based on this state. To do.

  Here, for example, the following rule is defined. That is, when the number of pixels having a pixel value “0” (the number of black pixels) among the pixels in the vicinity of 8 of the pixel of interest is 1 or less, the pixel value of the pixel of interest is “1”. That is, the target pixel is a white pixel. On the other hand, when the number of pixels having the pixel value “0” (the number of black pixels) among the pixels in the vicinity of 8 of the pixel of interest is 5 or more, the pixel value of the pixel of interest is “0”. That is, the target pixel is a black pixel. In addition, when the number of pixels having a pixel value “0” (the number of black pixels) is 2 or more and 4 or less, the pixel value of the pixel of interest is the original value. It is assumed that a rule such as “0” or “1” because it is binarized is defined.

  In this case, the image processing calculation unit 115 in FIG. 3 determines the pixel value of the pixel of interest according to the above-described rule, with each pixel constituting the received light image after the image binarization processing being one pixel of interest. Note that the method of setting the target pixel is not particularly limited, but here, for example, when the target pixel is set sequentially from the upper left to the right side of the received light image, and the rightmost pixel is set as the target pixel, the next lower pixel is set. It is assumed that a setting method is adopted in which the setting of setting the pixel at the left end of this row (horizontal line) as the target pixel is repeated until the bottom row (final horizontal line).

  Note that the rules described above are merely examples, and other rules may be adopted. For example, the reference condition for setting the pixel value of the target pixel to “0” is set to one or less black pixels in the above-described example, but is not limited to the above-described example, and is appropriately changed depending on the situation. You can also Similarly, the reference condition for setting the pixel value of the target pixel to “1” is set to five or more black pixels in the above example, but is not limited to the above example, and is appropriately changed according to the situation. You can also

  Next, the isolated point removal process in step S103c will be described in detail.

  As described above, a finger object is detected from among several connected components obtained as a result of the labeling process in step S103d. That is, as a result of the labeling process in step S103d, several connected components that are finger object candidates are obtained. Here, a pixel data (pixel value) group corresponding to an object (here, a finger) in contact with or close to the object is detected as an object of the finger. Therefore, for the purpose of easily performing such detection processing, that is, for the purpose of reducing the number of finger object candidates, of the several connected components included in the image data after the noise removal processing, it is clearly in contact or proximity. It is preferable to perform processing such as removing a connected component that is not an object, that is, a connected component formed by noise that cannot be removed. As an example of such processing, in the example of FIG. 9, the isolated point removal processing in step S103c is executed.

  Specifically, for example, it is assumed that a light reception image (a part) as shown in FIG. 13 is obtained as a result of the noise removal processing in step S103b.

  In this case, the connected components I and II, which are aggregates of white pixels surrounded by black, are finger object candidates.

  As described above, the area threshold A is set in the process of step S101 in FIG. 8, and the point of area A or larger in the data in the process of step S104 (as described above, this point is not a single pixel. It is determined whether or not there is a connected component. That is, in the example of FIG. 8, a connected component (the point in step S104) having an area A or more is detected as a finger object.

  In this case, if the area is the total number of pixels belonging to the connected component and the area threshold A = 100 is set, the connected component (pixel group) having an area much smaller than the area threshold A, for example, “5 It can be determined that the connected component of the following area is not a finger object but noise.

  In the example of FIG. 13, the connected component I is an aggregate of three white pixels. That is, the area of the connected component I is “3”. Therefore, the process of determining that the connected component I is not significant data, that is, determining that the connected component I is not a finger object candidate and removing the connected component I is an isolated point removal process. The removal of the connected component I means that the three white pixels constituting the connected component I are converted to black pixels, that is, the pixel values of the three pixels constituting the connected component I are changed from “1” to “0”. It means to convert to. The connected component II is an aggregate of 15 white pixels. That is, the area of the connected component II is “15”. Accordingly, the connected component II remains without being removed by the isolated point removal process. That is, when isolated point removal processing is performed on the received light image (part) of FIG. 13, an image as shown in FIG. 14 is obtained.

  Next, the labeling process in step S103d will be described in detail.

  That is, the labeling process is a process in which a unique label is attached to each connected component made up of a collection of pixels (white pixels) having a pixel value “1” from the image data after the isolated point removal process. is there.

  Specifically, for example, it is assumed that a light reception image (partial) as shown in FIG. 15 is obtained as a result of the isolated point removal process in step S103c.

  In this case, when the received light image of FIG. 15 is subjected to labeling processing, as shown in FIG. 16, each of the four connected components is labeled 01, Label 02, Label 03, and Label 04, respectively. Will be attached.

  In this way, when the labeling process of step S103d in FIG. 9 is executed, the process proceeds to step S103e. In step S103e, the image processing calculation unit 115 in FIG. 3 counts the number of points (pixels) of each connected component with a label.

  The processing result of step S103e is the area of each connected component. Specifically, for example, when the process of step S103e is performed on the received light image of FIG. 16, the area of the connected component of Label 01 is “189”, and the area of the connected component of Label 02 is “6”. The area of the connected component of Label 03 is calculated as “236”, and the area of the connected component of Label 04 is calculated as “18”.

  In this case, for example, if the area threshold A = 100 is set, the connected component of Label 01 and the connected component of Label 03 are detected as finger objects. That is, here, it is a connected component corresponding to a finger that is close to or touching, and is detected as “a point greater than area A” in step S104 of FIG.

  The details of the image processing (area calculation) in step S103 of FIG. 8 have been described above.

  Next, details of the image processing (position calculation) in step S107 of FIG. 8 will be described.

  FIG. 17 is a flowchart illustrating a detailed example of the image processing (position calculation) in step S107.

  In step S107a, the image processing calculation unit 115 in FIG. 3 calculates the barycentric position coordinates of each connected component.

  The connected component for which the center-of-gravity position coordinates are calculated in the process of step S107a is detected as the connected component detected as the finger object, that is, the “point having an area of A or more” in step S104 of FIG. Connected components. Specifically, for example, in the example of FIG. 16 described above, the connected component of Label 01 and the connected component of Label 03 are processing targets of Step S107a.

  However, more precisely, in order for the connected component of Label 01 and the connected component of Label 03 to be processed in step S107a, as shown in step S105 of FIG. 8, it is not shown in FIG. One more point (connected component) of area A or more needs to exist. Here, in addition to the connected component of Label 01 in FIG. 16 and the connected component of Label 03, the connected component of Label 02 (not shown in FIG. 16) (the connected component different from the connected component of Label 02 shown in FIG. 16) is used. Component) is area A or more. The following description will be made assuming that the image processing (position calculation) of FIG. 17 is performed on the connected component of Label 01, the connected component of Label 02, and the connected component of Label 03.

  In this case, for example, if each pixel constituting the connected component of Label 01 in FIG. 16 is Pn (n is any integer value from 1 to 189) and the coordinates are (xn, yn), Label 01 The barycentric position coordinates of the connected component are calculated as follows.

  That is, since the connected component of Label 01 is an aggregate of 189 pixels, the average of the position coordinates of 189 pixels is the barycentric position coordinate. That is, the center-of-gravity position coordinates G01 = ((x1 + x2 + x3 +... X189) / 189, (y1 + y2 + y3 +... Y189) / 189).

Alternatively, the equation for obtaining the center-of-gravity position coordinates of Labelk (here, k is any value of 01, 02, 03) is as shown in the following equation (1) when all pixels are calculated in order. expressed.
Gk = ((Σxi * wi) / Σwi, (Σyi * wi) / Σwi) (1)
However, in Expression (1), wi is 1 for a pixel having a Label and 0 otherwise.

  In this way, the gravity center position coordinate G01 of the connected component of Label 01, the gravity center position coordinate G02 of the connected component of Label 02, and the gravity center position coordinate G03 of the connected component of Label 03 are calculated by the processing of Step S107a. FIG. 18 shows the result of calculating the gravity center position coordinates G01 of the connected component of Label 01 and the gravity center position coordinates G03 of the connected component of Label 03, corresponding to the example of FIG.

  Hereinafter, the point of the center-of-gravity position coordinate GK of the connected component of Label K (here, K is one of 01, 02, and 03) is simply referred to as point GK. Further, the following description will be made assuming that the result shown in FIG. 19 is obtained in step S107a.

  Next, in step S107b, the image processing calculation unit 115 calculates the distance between the gravity center positions. Here, as shown in FIG. 19, the distance between point G01 and point G02, the distance between point G02 and point G03, and the distance between point G03 and point G01 are calculated.

  In step S107c, the image processing calculation unit 115 calculates the angle formed by the highest point and the lowest point. Here, the highest point refers to a point having the largest y coordinate value among the barycentric position coordinates (points G01, G02, G03) of each connected component calculated in step S107a. On the other hand, the lowest point refers to the smallest point. Here, as shown in FIG. 19, since the highest point is the point G01 and the lowest point is G03, the angle θs formed by the line segment connecting these two points and the horizontal line is calculated.

  In step S107d, the image processing calculation unit 115 calculates the coordinates of the center of each barycentric position. Specifically, for example, as shown in FIG. 19, the center coordinate Gtotal of each gravity center position is calculated by the following equation (2).

  Gtotal = ((x1 + x2 + x3) / 3, (y1 + y2 + y3) / 3) (2)

  By using the processing result of the image processing (position calculation) described above, the determination processing in steps S108 and S110 in FIG. 8 can be executed.

  For example, the angle θs calculated in the process of step S107c can be used to determine whether or not “three points are arranged substantially vertically” in step S108.

Specifically, for example, if the angle θv is set as a determination condition and the following inequality (3) is satisfied, it is possible to determine that the three points are substantially aligned vertically.
90-θv <θs <90 + θv (3)

  For example, when the angle θv = 10 ° is set, when the angle θs is 80 ° to 100 °, YES is determined in the process of step S108, that is, “three points are almost aligned vertically” is determined. Will be.

  Further, for example, in determining whether or not “the three points are substantially vertically aligned” in step S108, in addition to the angle θs described above, the distance between the gravity center positions calculated in the process in step S107b can also be used. . As a result, more detailed determination is possible.

Specifically, for example, the following inequalities (4) and (5) are satisfied in addition to the above-described condition using the angle θs as a condition for determining whether or not the three points are arranged substantially vertically. This condition can be added.
(Distance between point G01 and point G02) <(Distance between point G03 and point G01) (4)
(Distance between point G02 and point G03) <(Distance between point G03 and point G01) (5)

Alternatively, for example, a condition that the following inequality (6) is satisfied may be adopted as the condition to be added.
(Distance between point G02 and point Gmid) <(Half of distance between point G03 and point G01) (6)
Note that the point Gmid in the equation (6) is a point of coordinates ((x1-x3) / 2, (y1-y3) / 2) as shown in FIG. The details of the determination process in this case will be described later with reference to FIG.

  Further, for example, for the determination of “in which position on the screen” in step S110 of FIG. 8, the center coordinate Gtotal of the center of gravity calculated in the process of step S107d can be used. Specifically, for example, when the center coordinate Gtotal of the center of gravity position is located on the left side of the center of the received light image, it can be determined that it is on the left side of the screen. On the other hand, when the center coordinate Gtotal of the center of gravity position is located on the right side of the center of the received light image, it can be determined that it is on the right side of the screen.

  Next, another example of processing for determining an instruction from an operator based on the positional relationship between a plurality of contact portions will be described with reference to FIGS. FIG. 21 shows a case where the operator touches the lower end of the display area 121 with three fingers f1, f2, and f3. At this time, the received light image as shown in the lower part of FIG. 21 is obtained, and the three contact portions 151, 152, 153 are detected.

  FIG. 22 is a flowchart illustrating an example of processing for determining an operation instruction from a received light image when a contact portion as illustrated in FIG. 21 is detected. With reference to this flowchart, a process for determining an instruction from the operator based on the detected positional relationship between the plurality of contact portions will be described.

  Here, as a definition of the correspondence between the contact portion and the operation instruction in advance, in addition to the definition that “when three fingers are simultaneously touched in the vertical direction, the display image is scrolled in that direction”, “ A case will also be described in which the definition of “when the three fingers are simultaneously touched in the horizontal direction and the display image is scrolled in that direction” is also described.

  In other words, of the command recognition / issuance unit 17 of FIG. 2 configured as the instruction determination unit 116 in the example of FIG. 3, the positional relationship type command recognition unit 23, the command definition holding unit 26, and the command issuance unit 27. An example of the process when is mainly functioning is shown in the flowchart of FIG. That is, “when three fingers are touched side by side in the vertical direction at the same time, the display image is scrolled in that direction” and “when three fingers are touched side by side in the horizontal direction at the same time, The definition of “scroll in the direction” is held in the command definition holding unit 26, and the positional relationship type command recognition unit 23 performs “scroll the display image leftward”, “ Processing example when recognizing commands such as “scroll right”, “scroll display image upward”, and “scroll display image downward” and command issue unit 27 issues such a command Is shown in the flowchart of FIG.

  Note that the point in FIG. 22 does not mean one pixel, like the point in FIG. 8, but means one object described above, that is, one connected component in FIG.

  In this process, from the process of setting the threshold value A of the area for detecting the contact part (step S101) to the process of determining whether the detected three contact parts are substantially aligned in the vertical direction (step S108). This is the same as that described in the flowchart of FIG. If the three points are substantially aligned in the vertical direction, the process proceeds to another process (step S113), and if not, the process proceeds to the next process. Note that the separate processing in step S113 here refers to, for example, the processing in steps S110 to S112 in FIG. Next, it is determined whether the three points are substantially aligned in the horizontal direction (step S114). If the three points are not aligned in the horizontal direction, the process proceeds to another process (step S115). If the three detected points are arranged substantially horizontally, it is determined where the points are on the screen (step S116), and if it is located on the lower side of the screen, the display screen is scrolled downward. If it is located at the upper side of the screen (step S117), the display screen is scrolled upward (step S118).

  In addition, what is necessary is just to perform the determination process of whether three points of step S114 are located in a line substantially similarly to the determination process of step S108 mentioned above. Moreover, what is necessary is just to perform the determination process of which position on the screen of step S116 is the same as the determination process of step S110 of FIG. 8 mentioned above.

  For example, as shown in the upper part of FIG. 21, when the operator touches the lower end of the display area 121 with three fingers f1, f2, and f3, a received light image as shown in the lower part of FIG. 21 is obtained. Since the detected three contact portions 151, 152, 153 are arranged in the horizontal direction and are located on the lower side of the screen, the display screen can be scrolled down.

  In the above example, the operator's instruction is determined based on the detected positional relationship of the contact portion.Next, for a processing example in which an instruction from the operator is determined based on the movement position change of the plurality of contact portions, This will be described with reference to FIGS. FIG. 23 shows a state where the operator touches the display surface of the display area 121 with three fingers f1, f2, and f3. Here, it is assumed that the positions touched by the fingers f1, f2, and f3 are moved in the directions indicated by the arrows a1, a2, and a3 from this state. The received light image in this case is shown in the lower part of FIG. The light reception image P1 is an example of a light reception image before the finger is moved, and three contact portions 161a, 162a, and 163a are detected. The received light image P2 is an example of a received light image while moving a finger, and the received light image P3 is an example of a received light image after moving the finger. Thus, in the received light image data, the position of the contact portion changes with the passage of time, and in the received light image P3, it can be seen that the intervals between the three contact portions 161c, 162c, 163c are separated from the first received light image P1.

  FIG. 24 is a flowchart illustrating an example of processing for determining an operation instruction from a received light image when a contact portion as illustrated in FIG. 23 is detected. With reference to this flowchart, a process for determining an instruction from the operator based on the detected change in the movement position of the plurality of contact portions will be described. Here, the correspondence between the contact portion and the operation instruction is preliminarily defined as “the display image is enlarged when the three fingers are in a substantially circumferential contact and moved on the screen so that the circle becomes large” and “ It is defined that “the display image is reduced when three fingers come into contact with each other in a substantially circular shape and moved on the screen so that the circle becomes smaller”.

  In other words, among the command recognition / issue unit 17 of FIG. 2 configured as the instruction determination unit 116 in the example of FIG. 3, the detection result holding unit 21, the composite command recognition unit 25, the command definition holding unit 26, and An example of processing when the command issuing unit 27 mainly functions is shown in the flowchart of FIG. That is, “When three fingers touch almost a circle and move on the screen so that the circle is larger, the display image is enlarged.” “Three fingers touch almost a circle. The definition that “the display image is reduced when moving on the screen so that the circle becomes smaller” is held in the command definition holding unit 26, and the composite command recognition unit 25 is based on these two definitions. FIG. 24 is a flowchart showing an example of processing when the command issuing unit 27 recognizes the command “display the display image in an enlarged manner” or “display the display image in a reduced size” and issues the command. .

  That is, the condition based on the positional relationship of the three objects (finger) when “three fingers are in contact with a substantially circle” and the object “when moved on the screen so that the circle becomes larger” The combined condition with the condition based on the time position change is a condition of the combined command of “enlarged display of display image”. Similarly, the condition based on the positional relationship of the three objects “when three fingers are almost in contact with a circle” and the time of the object “when moving on the screen so that the circle becomes smaller” The combined condition with the condition based on the position change is the condition of the combined command “display the display image in a reduced size”.

  Therefore, the composite command recognition unit 25 functions in the example of FIG. In other words, as described above, when the function of the composite command recognition unit 25 is delegated to the command issuing unit 27, the positional relationship type command recognition unit 23 and the time position change type command recognition unit 22 function. It will be.

  Note that the point in FIG. 24 does not mean one pixel, like the point in FIG. 8, but means one object described above, that is, one connected component in FIG.

  In this process, from the process of setting the threshold value A of the area for detecting the contact part (step S101) to the process of determining whether the three contact parts detected are aligned substantially in the horizontal direction (step S114) This is the same as that described in the flowchart of FIG. If the three points are arranged almost horizontally, the process proceeds to another process (step S119).

  Note that the separate processing in step S113 here refers to, for example, the processing in steps S110 to S112 in FIG. 8 described above. However, in order to execute such a processing example, the command definition holding unit 26 holds a definition that “when three fingers are simultaneously touched in the vertical direction, the display image is scrolled in that direction”. Need to be.

  Further, the separate processing in step S119 here refers to, for example, the processing in steps S116 to S118 in FIG. 22 described above. However, in order to execute such a processing example, the command definition holding unit 26 holds a definition that “when three fingers are simultaneously touched in the horizontal direction, the display image is scrolled in that direction”. Need to be.

  If it is determined NO in the process of step S114, that is, if the three points are not arranged in the horizontal direction, it is further determined whether the three points are arranged in a substantially circular shape (step S120). If not, the process proceeds to another process (step S121). If they are arranged in a circle, it is determined whether each point has moved (step S122), and if not, the process ends. If it is moving, it is determined how it is moving (step S123). If it is moving in the direction in which the three points are separated, an instruction to enlarge the display screen is given (step S124). Is moving in the direction of approaching, it is instructed to reduce the display screen (step S125).

  Further, with reference to FIGS. 25 to 35, details of main processes in the process of the example of FIG. 24 will be described below.

  In the following description, it is assumed that the processing up to step S120 in FIG. 24 is executed by the image processing calculation unit 115 in FIG. However, you may make it make the instruction | indication determination part 116 perform at least one part of the processes to FIG.24 S120.

  FIG. 25 is a flowchart for explaining a detailed example of the determination process of “whether the three points are substantially vertically aligned” in step S108.

  The following description will be made assuming that the result shown in FIG. 26 is obtained by the image processing (position calculation) in step S107 of FIG.

  Steps S108a to S108f are determination processing using the inequality of equation (3) described above.

  That is, in step S108a, the image processing calculation unit 115 in FIG. 3 sets the angle θv. As described above, this angle θv is an angle as a condition for determining whether or not three connected components (three points in step S108 in FIG. 24) are arranged substantially vertically. It is held in advance in the holding unit 26 (FIG. 2) or the like.

  In step S108b, the image processing calculation unit 115 selects the center of gravity having the minimum y coordinate among the three connected components, and sets it as the point Gmin. For example, in the example of FIG. 26, the point G03 becomes the point Gmin.

  In step S108c, the image processing calculation unit 115 selects the center of gravity having the maximum y coordinate among the three connected components, and sets it as a point Gmax. For example, in the example of FIG. 26, the point G01 becomes the point Gmax.

  In step S108d, the image processing calculation unit 115 sets the remaining centroid of the three connected components as the point Gaux. For example, in the example of FIG. 26, the point G02 becomes the point Gaux.

  In step S108e, the image processing calculation unit 115 obtains an angle θs formed by the points Gmax and Gmin. For example, in the example of FIG. 26, the angle θs formed by the points G01 and G03 is obtained.

  In step S108f, the image processing calculation unit 115 determines whether 90−θv <θs and θs <90 + θv.

  That is, the determination process in step S108f is a process for determining whether or not the above inequality (3) is satisfied.

  If it is determined as NO in the process of step S108f, it is determined that the three points (three connected components) are not arranged substantially vertically, that is, it is finally determined as NO as the determination process in step S108, and the process Advances to step S114 in FIG.

  On the other hand, if it is determined YES in step S108f, steps S108g to S108j are further executed. That is, steps S108g to S108j are determination processing using the inequality of equation (6) described above. In other words, the determination processing from steps S108g to S108j is determination processing when attention is paid to the arrangement of three points.

  In step S108g, the image processing calculation unit 115 obtains a midpoint Gmid between the point Gmax and the point Gmin. For example, in the example of FIG. 26, the midpoint Gmid0103 between the points G01 and G03 is obtained as the midpoint Gmid.

  In step S108h, the image processing calculation unit 115 obtains a distance “Gmid-Gaux” between the point Gmid and the point Gaux (G02 in the example of FIG. 26). In step S108i, the image processing calculation unit 115 obtains a distance “Gmax−Gmin” between the point Gmax and the point Gmin.

  In step S108j, the image processing calculation unit 115 determines whether or not “Gmid-Gaux” <“Gmax-Gmin” / 2. In the case of the example of FIG. 26, that is, when Gmax is G01, Gmin is G03, and Gaux is G02, “Gmid-Gaux” <“Gmax-Gmin” / 2 is described above. The inequality equation (6) is obtained. That is, in this case, the determination process in step S108j is a determination process as to whether or not inequality (6) is satisfied.

  In other words, the determination process of step S108j is a determination process of whether or not the distance “Gmid-Gaux” is shorter than half of the distance “Gmax-Gmin”. The distance “Gmid-Gaux” is shorter than half of the distance “Gmax-Gmin”. The distance “Gmid-Gaux” (“Gmid-G02” in the example of FIG. 26) is the distance “Gmax-Gmid”. ("G01-Gmid0103" in the example of FIG. 26) or shorter than the distance "Gmin-Gmid" ("G03-Gmid0103" in the example of FIG. 26).

  Accordingly, the distance “Gmid-Gaux” (“Gmid-G02” in the example of FIG. 26) is shorter than the distance “Gmax-Gmid” (“G01-Gmid0103” in the example of FIG. 26) or the distance “Gmin-Gaux”. If it is shorter than “Gmid” (“G03-Gmid0103” in the example of FIG. 26), it is determined YES in the process of step S108j, and as a result, the three points (three connected components) are arranged substantially vertically. That is, it is finally determined that the determination process in step S108 is YES, and the process proceeds to step S113 in FIG.

  On the other hand, if it is determined as NO in the process of step S108j, it is determined that the three points (three connected components) are not aligned substantially vertically, that is, the determination process of step S108 is NO. The final determination is made, and the process proceeds to step S114 in FIG.

  In this way, in the example of FIG. 25, not only the determination processing from steps S108a to S108f, that is, not only the determination processing using the inequality of equation (3) described above, but also the determination processing from steps S108g to S108j, The determination processing using the inequality of the equation (6) is executed. The reason for this is that the determination process using the inequality of expression (3) described above may not lead to the determination process of “whether the three points are substantially circumferential” in step S120 of FIG. Because.

  Specifically, for example, it is assumed that the result shown in FIG. 27 instead of FIG. 26 is obtained by the image processing (position calculation) in step S107 of FIG. In this case, as shown in FIG. 27, since the angle θs is an angle formed by the centroid G2 ′ and the centroid G3 ′, only the determination processing from steps S108a to S108f, that is, the above-described equation (3). Only with the determination process using the inequality, it is determined that the three points (three connected components) are arranged substantially vertically. However, in the case of the positional relationship shown in FIG. 27, it is not determined that “the three points are arranged substantially vertically”, but “the three points are arranged substantially circumferentially”. Is often preferred. Therefore, in the example of FIG. 25, the condition that “the three points are almost vertically aligned” is made stricter by further adding a determination process using the inequality of the above-described equation (6). However, there is no problem in consciously determining a rule that determines that “the three points are almost vertically aligned” with the positional relationship shown in FIG.

  The details of the determination process of whether or not “the three points are substantially vertically aligned” in step S108 of FIG. 24 have been described above.

  In the following, details of the determination process of “whether the three points are substantially side by side” in step S114 of FIG. 24 will be described using the case shown in FIG. 28 as an example. That is, the result shown in FIG. 28 is obtained by the image processing (position calculation) in step S107 in FIG. 24. As a result, it is determined NO in step S108 in FIG. 24, and the process proceeds to step S114. The case will be described below as an example.

  Note that the determination process for determining whether or not “three points are substantially aligned” is basically the same process as the determination process for determining whether or not “three points are approximately aligned”. Therefore, description of processes corresponding to the processes described above will be omitted as appropriate.

  FIG. 29 is a flowchart for explaining a detailed example of the determination processing of “whether the three points are substantially side by side” in step S114.

  The processing from step S114a to S114f corresponds to the processing from step S108a to S108f in FIG.

  That is, in step S114a, the image processing calculation unit 115 in FIG. 3 sets the angle θh. Like the angle θv, this angle θh is an angle as a condition for determining whether or not three connected components (three points in step S114 in FIG. 24) are arranged substantially horizontally. It is held in advance in the definition holding unit 26 (FIG. 2).

  In step S114b, the image processing calculation unit 115 selects the center of gravity having the minimum x coordinate among the three connected components, and sets it as the point Gmin. That is, in the example of FIG. 28, the point G13 becomes the point Gmin.

  In step S114c, the image processing calculation unit 115 selects the center of gravity having the maximum x coordinate among the three connected components, and sets it as the point Gmax. That is, in the example of FIG. 28, the point G11 becomes the point Gmax.

  In step S114d, the image processing calculation unit 115 sets the remaining centroid of the three connected components as a point Gaux. That is, in the example of FIG. 28, the point G12 becomes the point Gaux.

  In step S114e, the image processing calculation unit 115 obtains an angle θs (see FIG. 28) formed by the points Gmax and Gmin.

  In step S114f, the image processing calculation unit 115 determines whether -θh <θs and θs <θh.

  If it is determined NO in the process of step S114f, it is determined that the three points (three connected components) are not substantially aligned side by side, that is, the determination is NO as the determination process of step S114, and the process Advances to step S120 in FIG.

  On the other hand, if it is determined YES in the process of step S114f, steps S114g to S114j are further executed. That is, the processing from step S114g to S114j corresponds to the processing from step S108g to S108j in FIG. In other words, the determination process from step S114g to S114j is a determination process when attention is paid to the arrangement of three points.

  In step S114g, the image processing calculation unit 115 obtains a midpoint Gmid between the point Gmax and the point Gmin. For example, in the example of FIG. 28, the midpoint Gmid0103 between the points G11 and G13 is obtained as the midpoint Gmid.

  In step S114h, the image processing calculation unit 115 obtains a distance “Gmid-Gaux” between the point Gmid and the point Gaux (G12 in the example of FIG. 28). In step S114i, the image processing calculation unit 115 obtains a distance “Gmax−Gmin” between the point Gmax and the point Gmin.

  In step S114j, the image processing calculation unit 115 determines whether or not “Gmid-Gaux” <“Gmax-Gmin” / 2.

  If it is determined as YES in the process of step S114j, it is determined that the three points (three connected components) are substantially aligned side by side, that is, the determination is YES as the determination process in step S114, and the process Advances to step S119 in FIG.

  On the other hand, when it is determined NO in the process of step S114j, it is determined that the three points (three connected components) are not substantially aligned side by side, that is, the determination process of step S114 is NO. The final determination is made, and the process proceeds to step S120 in FIG.

  As described above, in the example of FIG. 29, not only the determination processing from steps S114a to S114f but also the determination processing from steps S114g to S114j are executed. The reason is that only the determination process from steps S114a to S114f, that is, only the determination process using the angle θs, determines whether or not “three points are arranged substantially in a circle” in step S120 of FIG. This is because there is a case where it does not reach the end. This is as described above with reference to FIG.

  As above, the details of the determination process of whether or not “three points are arranged substantially horizontally” in step S114 of FIG. 24 have been described.

  Hereinafter, the details of the determination process of “whether the three points are arranged substantially in a circle” in step S120 of FIG. 24 will be described.

  As described above, in the example of FIG. 24, it is determined whether or not “3 points are substantially vertically aligned” in the process of step S108, and further, “3 points are approximately horizontally aligned” in the process of step S114. Is determined. That is, in the example of FIG. 24, the conditions “vertical or horizontal” and “lined up” are set. Therefore, all the other three points (three connected components) may be determined as “almost circumferentially arranged”. In this case, it is determined in the process of step S108 that the three points are not substantially aligned vertically, and further, in the process of step S114, it is determined that the three points are not substantially aligned in the horizontal direction. Since the determination process of whether or not “they are arranged in a substantially circumferential shape” is executed, it is always determined as YES in the process of step S120.

  However, in this case, when the result shown in FIG. 30 is obtained by the image processing (position calculation) in step S107 in FIG. 24, that is, the centroids of the three connected components (three points in step S120). As shown in FIG. 30, even when “not vertical or horizontal” is a relatively “aligned” arrangement, it is determined that “almost circumferentially aligned”. Become.

  Therefore, in order to exclude the arrangement in which “not vertical or horizontal” is relatively “aligned” from the “almost circumferentially aligned” arrangement, “3 points in step S120 of FIG. For example, a process as shown in FIG. 31 can be adopted as a process for determining whether or not they are arranged in a circle.

  That is, FIG. 31 is a flowchart for explaining a detailed example of the determination process of “whether the three points are arranged substantially in a circle” in step S120.

  The following description will be given assuming that the result shown in FIG. 32 is obtained by the image processing (position calculation) in step S107 of FIG.

  In step S120a, the image processing calculation unit 115 in FIG. 3 obtains the distance between the centroids of the three connected components (three points in step S120).

  Here, assuming that the centers of gravity of the three connected components are referred to as points G31, G32, and G33, as shown in FIG. 32, the distance between point G31 and point G32, and the distance between point G32 and point G33. , And the distance between point G33 and point G31.

  In step S120b, the image processing calculation unit 115 sets two centroids that make the longest distance as points Ga and Gb, and sets the middle point as a point Gmid. In step S120c, the image processing calculation unit 115 sets the remaining center of gravity as the point Gaux. For example, in the example of FIG. 32, the point G31 becomes the point Ga, the point G32 becomes the point Gb, the point Gmid3132 becomes the middle point Gmid, and the point G33 becomes the point Gaux.

  In step S120d, the image processing calculation unit 115 obtains a distance “Gmid-Gaux” between the point Gmid and the point Gaux. In step S120e, the image processing calculation unit 115 obtains a distance “Ga−Gb” between the point Ga and the point Gb.

  In step S120f, the image processing calculation unit 115 determines whether or not “Gmid-Gaux”> “Ga-Gb” / 2.

  When the distance “Gmid-Gaux” is longer than half of the distance “Ga-Gb”, it is determined as YES in the process of step S120f, and as a result, the three points (three connected components) are arranged approximately in a circle. That is, it is finally determined that the determination process in step S120 is YES, and the process proceeds to step S122 in FIG.

  On the other hand, when the distance “Gmid-Gaux” is less than or equal to half of the distance “Ga-Gb”, it is determined as NO in the process of step S120f, and as a result, three points (three connected components) are almost circles. It is determined that they are not arranged in a circle, that is, it is finally determined as NO as the determination process in step S120, and the process proceeds to step S121 in FIG.

  The details of the determination process of “whether the three points are arranged substantially circumferentially” in step S120 of FIG. 24 have been described above.

  Hereinafter, the details of the determination process of “whether each point is moving” in step S122 of FIG. 24 will be described.

  A plurality of received light image data (field data or frame data) are sequentially and sequentially output from the received light image generation unit 114 in FIG. 3, that is, from the received light image generation unit 15 in FIG. In this case, the process in the example of FIG. 24 is executed for each received light image data in units of received light image data. Therefore, hereinafter, the received light image data to be noticed as an object of the processing in the example of FIG. 24 is referred to as noticed received light image data.

  In this case, coordinate information (one of the above-described object information) obtained by performing the process of the example of FIG. 24 (for example, the process of step S107) on the received light image data immediately before the focused received light image data. Is held in the detection result holding unit 21 of FIG. 2 as described above.

  On the other hand, coordinate information (one of the above-described object information) obtained by performing the process of the example of FIG. 24 (for example, the process of step S107) on the received light image data is the object detection unit 16 of FIG. Is provided to the command recognition / issuer 17.

  In this case, the time position change type command recognition unit 22 or the composite type command recognition unit 25 of FIG. 2, that is, the instruction determination unit 116 of FIG. 3 compares the information of these coordinates, thereby performing step S122 of FIG. Whether each point is moving or not is determined.

  In the following description, for the sake of brevity, it is assumed that the operation subject of the processing after step S122 is the time position change command recognition unit 22.

  Further, in the following description, the information on the gravity centers G31, G32, and G33 in the positional relationship shown in FIG. 32 described above is the object information (hereinafter referred to as the immediately preceding object information) of the received light image data immediately before the focused received light image data. ) Is held in the detection result holding unit 21 of FIG. Then, information on the center of gravity G41, G42, and G43 in the positional relationship shown in FIG. 33 is obtained from the object detection unit 16 in FIG. It is assumed that the command recognition unit 22 has been provided.

  In this case, the positional relationship between the target object information and the immediately preceding object information is as shown in FIG. Hereinafter, the centroids G41, G42, and G43 are referred to as attention centroids G41, G42, and G43, and the centroids G31, G32, and G33 are referred to as immediately preceding centroids G31, G32, and G33.

  That is, the time position change command recognition unit 22 in FIG. 2 can determine whether or not “each point is moving” in step S122 in FIG. 24 by referring to the information in FIG.

  Specifically, for example, the time position change type command recognition unit 22 first determines which of the respective center of gravity G41, G42, and G43 corresponds to one of the immediately preceding centers of gravity G31, G32, and G33. It is determined whether or not the immediately preceding center of gravity is a moving change, and based on the determination result, the moving direction and the moving distance (movement vector) for each of the attention centers of gravity G41, G42, and G43 are obtained.

  Although the determination method itself is not particularly limited, it is assumed here that a method for determining based on the distance from the coordinates is adopted. That is, it is assumed that a technique is adopted in which the immediately preceding centroid of the coordinate closest to the coordinate of the focused centroid is the corresponding centroid.

  In this case, in the example of FIG. 34, it is determined that the target gravity center G43 and the nearest previous gravity center G33 correspond to each other, that is, it has been moved and changed from the coordinate of the previous gravity center G33 to the coordinate of the attention gravity center G43. Thereby, the movement direction and movement distance (movement vector) about the center of gravity G43 are obtained. Similarly, it is determined that the center of gravity G42 of interest corresponds to the nearest center of gravity G32 closest thereto, that is, the coordinate of the center of gravity G32 immediately before is moved and changed to the coordinate of the center of gravity G42 of interest. Thereby, the movement direction and movement distance (movement vector) about the center of gravity G42 are obtained. Further, it is determined that the center of gravity G41 and the nearest previous center of gravity G31 correspond to each other, that is, the coordinate of the immediately preceding center of gravity G31 has moved and changed to the coordinate of the center of gravity G41. Thereby, the movement direction and movement distance (movement vector) about the center of gravity G41 are obtained.

  Then, when the movement distance for each of the attention centers of gravity G41, G42, and G43 exceeds a predetermined threshold value, the time position change type command recognition unit 22 determines each point in the process of step S122 of FIG. If it is determined that the point has moved, the process proceeds to step S123. Otherwise, it is determined in step S122 that each point has not moved, and the process in FIG. 24 is terminated. In this case, the determination condition that each point is moving may be a condition that at least one of the moving distances for each of the focused gravity centers G41, G42, and G43 exceeds a threshold value. However, two or more of the moving distances for each of the focused centroids G41, G42, and G43 may be set as a condition, or alternatively, for each of the focused centroids G41, G42, and G43, It is good also as a condition that all the movement distances have exceeded the threshold value.

  When the process proceeds to step S123, the time position change type command recognition unit 22 further determines how it is moving.

  The determination method itself is not particularly limited. For example, as described above, the movement vector (movement direction and movement distance) for each of the attention gravity centers G41, G42, and G43 is known. It can also be adopted.

  Also, for example, as shown in FIG. 35, the area of a triangle (hereinafter referred to as the immediately preceding triangle) having the respective centroids G31, G32, and G33 as vertices, and the triangle ( In the following, it is possible to adopt a method of determining how the object moves by comparing the area with the triangle of interest. When such a method is employed, when the area of the target triangle is larger than the area of the immediately preceding triangle by a certain amount (see FIG. 35), it is determined that the target triangle is moving in the “enlargement” direction in step S123 of FIG. Then, the process proceeds to step S124. On the other hand, when the area of the target triangle is smaller than the area of the previous triangle by a certain amount or more, it is determined in the process of step S123 in FIG. 24 that the area is moving in the “reduction” direction, and the process proceeds to step S125. It will be.

  The details of the main processing in the processing of the example of FIG. 24 have been described above with reference to FIGS. 25 to 35.

  24 is executed, for example, as shown in the upper part of FIG. 23 described above, the operator touches the display screen with three fingers f1, f2, and f3, and the finger When the distance between the finger and the finger is moved away from each other, a light reception image as shown in the lower part of FIG. 23 is obtained, and the three detected contact portions are moved away from each other. As shown in the example of the post-operation screen as shown in FIG.

  Up to this point, the example of changing the display state of the display image has been described. However, various application processes can be performed by touching the display area of the display device with a finger, a pen, or the like. Next, an example of processing for editing the text displayed in the display area will be described with reference to FIGS. 37, 38, and 39. FIG. In this example, as shown in the upper part of FIG. 37, document data such as a character string is displayed in the display area 121 and an operation is performed on the document data. Here, the operator touches a part of the sentence displayed on the screen so as to be sandwiched between two fingers f1 and f2. At this time, a received light image as shown in the lower part of FIG. 37 is obtained, and the two contact portions 171 and 172 are detected.

  FIG. 38 is a flowchart illustrating an example of processing for determining an operation instruction from a received light image when a contact portion as illustrated in FIG. 37 is detected. With reference to this flowchart, a process for determining an instruction from the operator based on the detected sizes of the plurality of contact portions will be described. Here, the correspondence between the contact portion and the operation instruction is defined in advance as “if the display data is document data, the character string is selected when both ends of the character string are touched simultaneously with two fingers”. .

  In other words, of the command recognition / issuance unit 17 of FIG. 2 configured as the instruction determination unit 116 in the example of FIG. 3, the positional relationship type command recognition unit 23, the command definition holding unit 26, and the command issuance unit 27. An example of the processing when the above functions mainly is shown in the flowchart of FIG. That is, the definition that “when the display data is document data, the character string is selected when both ends of the character string are simultaneously touched with two fingers” is held in the command definition holding unit 26, and the positional relationship An example of processing when the type command recognition unit 23 recognizes a command “select a character string in a range between two fingers” based on such a definition, and the command issue unit 27 issues such a command. This is shown in the flowchart of FIG.

  Note that the point in FIG. 38 does not mean one pixel as in the point in FIG. 8, but means one object described above, that is, one connected component in FIG.

  First, an area threshold A for detecting the case where the contact portion is the fingertip and a threshold B for detecting the case where the contact portion is the belly of the finger are set (step S131). Next, the image data converted from the received light signal by the received light image generation unit 114 is acquired (step S132), and the area of the contact portion is calculated as the first image processing (step S133). Note that a detailed example of the processing in step S133 is as described above with reference to FIGS. As a result of the area calculation, it is determined whether there is a point equal to or larger than the area threshold B in the image data (step S134). If there is a point, the process proceeds to another process (step S135). A: It is determined whether or not there are more points (step S136). If there is no point greater than or equal to the threshold A, the next image data is acquired. If there is a point of area A or more, it is determined whether or not there are two points (step S137). If there are two points, the process proceeds to the next process. If not, the next image data is acquired. Next, as the second image processing, the positional relationship between the two contact portions is calculated (step S138). A detailed example of the process in step S138 is as described above with reference to FIGS. As a result of the position calculation, if the two points are separated from each other by a certain distance, an instruction to select a character string displayed between the two points is made (step S139).

  In this example, since an instruction from the operator is an instruction for editing a sentence, after the processing by the instruction determination unit 116 is performed as described in the flowchart, the determined instruction content is input to the input / output processing unit. 101 notifies the document processing unit 102 which is an external application. Then, the document processing unit 102 performs document data editing processing according to the instruction content, sends the result to the input / output processing unit 1 again, and generates and displays the data as display data.

  For example, as shown in the upper part of FIG. 37, when an operator touches a part of a sentence displayed in the display area 121 with two fingers f1 and f2, the operation is performed as shown in FIG. As in the screen example, the character string H1 between the fingers f1 and f2 is selected and displayed.

  Next, another example of processing for editing the text displayed in the display area will be described with reference to FIGS. 40, 41, and 42. FIG. 40 shows a case where document data such as a character string is displayed in the display area 121 and the operator places the finger f1 on the character string of the sentence displayed on the screen. At this time, a light reception image as shown in the lower part of FIG. 40 is obtained, and one elongated contact portion 181 is detected.

  FIG. 41 is a flowchart illustrating an example of processing for determining an operation instruction from a received light image when a contact portion as illustrated in FIG. 40 is detected. Here, the correspondence between the contact portion and the operation instruction is defined in advance as “if the display data is document data, the character string is erased when the finger is placed on the character string”.

  In other words, of the command recognition / issuance unit 17 of FIG. 2 configured as the instruction determination unit 116 in the example of FIG. 3, the shape-type command recognition unit 24, the command definition holding unit 26, and the command issuance unit 27 are An example of processing when mainly functioning is shown in the flowchart of FIG. That is, the definition of “when the display data is document data, the character string is erased when the finger is placed on the character string” is held in the command definition holding unit 26, and the shape type command recognition unit 24. FIG. 41 is a flowchart showing an example of processing when the command issuing unit 27 recognizes a command “deleting a character string included in a range where a finger is placed” based on such a definition and issues the command. It is shown.

  Note that the point in FIG. 41 does not mean one pixel as in the point in FIG. 8, but means one object described above, that is, one connected component in FIG.

  In this process, from the process of setting the area threshold A and threshold B for detecting the contact portion (step S131), it is determined whether or not there is a point equal to or larger than the area threshold B in the image data (step S134). The processes up to this point are the same as those described in the flowchart of FIG. The processing in the case where there is no point equal to or larger than the area B has been described with reference to the flowchart of FIG. If there is a point equal to or larger than the area threshold B in step S134, the position of the contact portion is calculated as the second image processing (step S141). Note that the position calculation here is different from the calculation of FIG. 17 described above, and the position occupied by the contact portion (the entire one connected component) in the received light image is calculated. As a result of the position calculation, when the contact portion is placed so as to overlap the character string displayed on the screen, an instruction to delete the character string of the portion where the contact portion is placed is set (step S142). ).

  FIG. 42 shows an example of the post-operation screen when the above processing is performed. As shown in the upper part of FIG. 40, when the operator places the finger f1 on the character string of the text displayed in the display area 121, as shown in FIG. 42, the character string of the portion where the finger f1 is placed. H2 is displayed in an erased state.

  Note that the processes shown in FIGS. 38 and 41 are not exclusive, and both can be realized simultaneously. A processing example in this case is shown in the flowchart of FIG. In this example, two examples of “selection” and “deletion” in which text is edited are shown, but the correspondence between the contact portion and the operation instruction may be different. It is also possible to realize an instruction for performing another editing process.

  43 does not mean one pixel as in the case shown in FIG. 8, but means one object described above, that is, one connected component shown in FIG.

In the present embodiment, an example has been described in which an object in contact with or close to the display area of the display device is detected and the state of the contact portion is determined as follows.
-The operation instruction is determined based on the positional relationship between a plurality of contact parts (objects).
-Judge the operation instruction based on the time position change of multiple contact parts (objects).
-The operation instruction is determined based on the shape (size, etc.) of a plurality of contact parts (objects).
By defining the correspondence between these conditions and other conditions such as the number of contact parts to be operated, positional relationship and movement method, and operation instructions, other screen operations and other editing of document data It is also possible to perform processing. In addition to screen operations, specific application processing can also be executed. In the case of executing another application, the processing can be realized by configuring the application processing unit to be incorporated outside the input / output processing unit 101.

  In the above description, the case where a reflective object such as a finger is in contact with or close to the display surface has been mainly described. However, since the display / light-receiving panel unit 120 in the present invention has the light-receiving elements arranged two-dimensionally on the display surface, it is possible to instruct operation even in other situations. For example, the operation may be instructed by using a plurality of light sources such as a laser pointer, changing the size of the beam, changing the shape of the beam, or moving these.

  Alternatively, any image can be used as long as a received light image can be formed and can be changed, such as forming an image of another object on the display or detecting a shadow. For example, if a clip is placed on the display, execute a specific application process or place a key on the display and rotate the key to the right to “lock”, rotate to the left to “lock” It is also possible to execute processing such as “release”.

  By the way, the series of processes described above can be executed by hardware, but can also be executed by software.

  In this case, for example, a personal computer shown in FIG. 44 may be adopted as at least a part of the image input / output device 1 of FIG.

  44, a CPU (Central Processing Unit) 201 executes various processes according to a program recorded in a ROM (Read Only Memory) 202 or a program loaded from a storage unit 208 to a RAM (Random Access Memory) 203. To do. The RAM 203 also appropriately stores data necessary for the CPU 201 to execute various processes.

  The CPU 201, the ROM 202, and the RAM 203 are connected to each other via the bus 204. An input / output interface 205 is also connected to the bus 204.

  The input / output interface 205 includes an input unit 206 such as a keyboard and a mouse, an output unit 207 including a display, a storage unit 208 including a hard disk, and a communication unit 209 including a modem and a terminal adapter. It is connected. The communication unit 209 controls communication performed with other devices (not shown) via a network including the Internet.

  A drive 210 is also connected to the input / output interface 205 as necessary, and a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately installed, and a computer program read from them is loaded. Installed in the storage unit 208 as necessary.

  When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, a general-purpose personal computer is installed from a network or a recording medium.

  As shown in FIG. 44, the recording medium including such a program includes a magnetic disk (including a floppy disk) on which the program is recorded, which is distributed to provide the program to the operator separately from the apparatus main body. ), Optical disk (including compact disk-read only memory (CD-ROM), DVD (digital versatile disk)), magneto-optical disk (including MD (mini-disk)), or semiconductor memory, etc. (package) Medium) 211, and a ROM 202 in which a program is recorded and a hard disk included in the storage unit 208 provided to the operator in a state of being incorporated in the apparatus main body in advance.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the order, but is not necessarily performed in chronological order, either in parallel or individually. The process to be executed is also included.

  Further, in the present specification, the system represents the entire apparatus including a plurality of apparatuses and processing units.

  DESCRIPTION OF SYMBOLS 1 Image input / output device, 2 Information processing apparatus, 11 Display image generation part, 12 Display control part, 13 Display part, 14 Light reception part, 15 Light reception image generation part, 16 Object detection part, 17 Command recognition / issue part, 21 Detection Result holding unit, 22 Time position change type command recognizing unit, 23 Position relation type command recognizing unit, 24 Shape type command recognizing unit, 25 Compound type command recognizing unit, 26 Command definition holding unit, 27 Command issuing unit, 101 I / O processing Unit, 102 document processing unit, 110 display signal generation unit, 111 signal control unit, 112 display signal driver, 113 light reception signal receiver, 114 light reception image generation unit, 115 image processing calculation unit, 116 instruction determination unit, 120 display / light reception panel , 121 Display area (sensor area ), 122 display horizontal driver, 123 display vertical driver, 124 sensor vertical receiver, 125 sensor horizontal receiver, 127 display side scanner, 128 light receiving side scanner, 130 display device, 131 pixels, 131a switching element, 131b pixel electrode , 131c sensor element, 131d reset switch, 131e capacitor, 131f buffer amplifier, 131g readout switch, 131h gate electrode, 131i drain electrode, 131j signal output electrode, 131k reset electrode, 131m readout control electrode, f1, f2, f3 finger , 141, 142, 143, 151, 152, 153, 161, 162, 163, 171, 172, 181 contact part, 201 CPU, 2 02 ROM, 203 RAM, 204 bus, 205 input / output interface, 206 input unit, 207 output unit, 208 storage unit, 209 communication unit, 210 drive, 211 removable media

Claims (11)

A plurality of display elements for displaying an image on a display screen;
A plurality of sensors arranged in the display screen;
An arithmetic unit that performs arithmetic processing to obtain information on the position and size of an object that is in contact with or close to the display screen, based on outputs from the plurality of sensors;
When the calculation unit detects a plurality of contacts that form one region that is individually connected to each other or a contact portion of an adjacent object, the area of the region formed by each contact portion and each other of each contact portion An instruction determination unit that determines an instruction of the operator according to the positional relationship,
A display device that executes predetermined processing according to a determined instruction. The display device according to claim 1,
A display control unit for controlling an image displayed on the display screen;
The instruction determination unit further determines the instruction of the operator according to the position on the display surface of each contact unit,
The display control unit controls the image according to an instruction from the instruction determination unit. The display device according to claim 1,
The instruction determination unit further determines an instruction of the operator according to a change in the positional relationship between the proximity units and a direction thereof. The display device according to claim 1,
A display control unit for controlling an image displayed on the display screen;
The display control unit controls the image according to an instruction from the instruction determination unit. The display device according to claim 1,
The arithmetic unit binarizes the outputs from the plurality of sensors by comparing each of the outputs with a predetermined threshold, and the set of outputs having a predetermined value with respect to the binarized outputs from the plurality of sensors A noise removal process is performed before performing an isolated point removal process that excludes a connected component that is smaller than a predetermined threshold among connected components consisting of or a labeling process that adds a unique label to each of the connected components. apparatus. The display device according to claim 5, wherein
The noise removal process is a process in which the output from each binarized sensor is set as a target pixel one by one, and the pixel value of the target pixel is specified according to the state of several pixels around the target pixel. apparatus. The display device according to claim 5, wherein
The calculation unit performs the isolated point removal process before performing the labeling process. The display device according to claim 1,
The said instruction | indication determination part determines the said operator's instruction | indication based on the object information which is the information regarding each detected contact part. Display apparatus. The display device according to claim 8, wherein
The object information includes information related to a change in the coordinate position of the contact portion. The display device according to any one of claims 1 to 9,
The predetermined process is a process of changing the entire display image to be displayed on the display element, or a process of changing an operation of an application for generating a display image to be displayed on the display element.
The process of changing the entire display image displayed on the display element includes scrolling the display image, or enlarging or reducing the display image,
The process of changing the operation of an application that generates a display image to be displayed on the display element includes selection or deletion of a character string included in the display image. Based on the output from a plurality of sensors arranged in the display screen, performing an arithmetic process to obtain information on the position and size of an object in contact with or close to the display screen,
Depending on the area of each region formed by each contact part and the mutual positional relationship between the contact parts when detecting a plurality of contacts or contact parts of adjacent objects forming a region where each is individually connected , Determine the operator's instructions,
A display method that performs predetermined processing according to the determined instruction.

Images