JP2016122410A - Information processing apparatus, control method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To distinguish, when a plurality of people can operate an operation screen, a multi-touch operation by a single user from a single-touch operation by each of a plurality of users.SOLUTION: An information processing apparatus comprises detection means that detects objects approaching on a projection surface to which an image is projected from a projection part and determination means that determines the number and the direction of approach of the detected objects, and controls movements of gesture operations by the objects according to the determined number and direction of approach of the objects.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an information processing technique for detecting a gesture operation on an image projected from a projection unit and controlling an operation based on the gesture operation.

  When making a loan contract at a bank, a bank employee (hereinafter referred to as “Teller”) is obliged to explain the items necessary for creating a contract or a product explanation in a face-to-face manner with a customer. Then, in order to indicate that the customer has received an explanation from the teller, it is common to put a name, a seal, etc. in a predetermined column. In addition, a system that performs such work semi-electronically has also been proposed (Patent Document 1).

  There are also devices that use a terminal such as a tablet to facilitate operations related to procedures and simplify the entry of necessary documents. In addition, there is a technique for projecting information related to a procedure and projecting information by projecting an operation screen related to the procedure on a desk using a projector and detecting a gesture with a finger or the like on the operation screen with a camera (Patent Document 2).

JP 2009-122796 A JP 2014-203176 A

  In a system such as that disclosed in Patent Document 2, a touch operation on a device having a touch panel such as a tablet or a smartphone with respect to an operation screen projected on a desk can be realized by using a camera that detects a gesture. . This touch operation includes a single touch by one point touch and a multi-touch operation by two point touch. As an example of the multi-touch operation, as shown in FIG. 11, a pinch out that expands / contracts the operation screen 500 with the index fingers of the right hand and the left hand of one user (Mr. A) (or the thumb and index finger in one hand). / There is a pinch-in operation.

  However, when such an operation screen that can accept multi-touch operations is used in operations where the teller and the customer face each other and perform various procedures, the teller and the customer operate the operation screen at the same time. It is in an environment that can. That is, there is a possibility that a plurality of people perform touch operations on the operation screen. Then, as shown in FIG. 12, in a situation where a plurality of people can operate the operation screen 500, a multi-touch operation by one user (Mr. A) and a single by each of a plurality of users (Mr. A and Mr. B). Although a touch operation may occur, in a conventional system, both operations are recognized as multi-touch operations, and the two cannot be distinguished. For this reason, the operation intended by the user is not realized depending on the contents of the operation, and there are some cases where convenience and operability are lacking.

  The present application has been made in view of the above-described problems, and an object thereof is to provide a technique capable of improving operability with respect to an operation screen.

In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is,
An information processing apparatus that detects a gesture operation on an image projected from a projection unit and controls an operation based on the gesture operation,
Detecting means for detecting an object entering the projection surface on which an image is projected from the projection unit;
Determination means for determining the number of objects detected by the detection means and the approach direction thereof;
Control means for controlling the operation of the gesture operation by the object according to the number of objects determined by the determining means and the approach direction thereof.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which can improve the operativity with respect to an operation screen can be provided.

It is an external appearance block diagram of a system. It is a block block diagram of a system. It is a figure which shows the utilization form of a facing system. It is a flowchart which shows the basic process of a facing system. It is a figure which shows the data structure of sheet DB. It is a figure which shows an example of touch operation. It is a figure which shows an example of touch operation. It is a figure which shows an example of touch operation. It is a flowchart which shows the detail of a display control process. It is a figure which shows an example of a gesture table. It is a figure which shows an example of touch operation. It is a figure which shows an example of touch operation. 5 is a diagram illustrating an example of a hand information management table 1300. FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  An example in which the system in the present embodiment is used in a bank will be described. Moreover, in the following description, a banker (banker) is assumed to be a teller and is described separately from a customer. The bank was used as an example in order to facilitate understanding by embodying the business, and it can be used if it is a life insurance company, etc. that sells or contracts products in person. However, the present invention is not limited to this. Note that this is an example only.

  FIG. 1 is an external configuration diagram of a system in the present embodiment, and FIG. 2 is a block configuration diagram thereof. This system includes a control device 100 that controls the entire system as an information processing device. The control apparatus 100 is connected to a projection imaging apparatus 110, a display 120 that functions as a user interface with the user, and input devices (such as a keyboard and a mouse) 130. The control device 100 is also connected to a backbone network (not shown) related to business.

  The control device 100 has a hardware configuration similar to that of a normal PC (personal computer) or the like. The control device 100 includes a CPU 101 that performs processing according to a program, a ROM 102 that stores a boot program, BIOS, and the like, a RAM 103 that stores an OS and applications executed by the CPU 101, and that the CPU 101 uses as a work area. The control device 100 also includes an I / F unit 105 for communicating with the projection imaging device 110, and a video controller 106 that performs drawing processing, video signal output processing, and the like in accordance with instructions from the CPU 101. Further, the control device 100 includes a storage controller 107 that reads / writes from / to an external storage device 108 typified by a hard disk (HDD), and a network I / F 104 for communicating with a backbone network (in particular, a network printer). The external storage device 108 stores an operating system (OS) 108a, an application program 108b for functioning as a face-to-face system, a sheet database (sheet DB) 108c, environment information 108d, and a gesture table (gesture TL) 108e. Yes. Details of the sheet DB 108c and the gesture TL 108e will be described later.

  The projection imaging apparatus 110 accommodates an I / F unit 118, a projection unit 115, an imaging unit 116, and a distance measurement unit 117 for communicating with the control device 100 in the casing. The projection imaging device 110 is provided with an opening 111 for allowing the projection light emitted from the projection unit 115 to pass therethrough. Furthermore, an opening 112 is also provided which is shared by the imaging unit 116 and the distance measurement unit 117 and for receiving light (including infrared rays) from the outside.

  The projection unit 115 emits projection light representing an image requested from the control device 100 toward the top plate 113 through the opening 112 and slightly obliquely. A mirror is provided on the lower surface side of the top plate 113 and reflects the projection light from the projection unit 115. As a result, the image on the front side of the installation position of the projection imaging apparatus 110 in FIG. 1 (actually, the upper surface of the table) is displayed as a projection screen. Note that the top plate 113 may be inclined instead of obliquely emitting the light.

  In addition, the imaging unit 116 and the distance measuring unit 117 are set as an imaging range or a range-finding range through the mirror of the top plate 113, which is wider than the projection area on the table by the projection unit 115. The imaging unit 116 has the same configuration as an imaging device typified by a normal digital camera, and optically reads an image within the imaging range of the table. The distance measuring unit 117 detects the depth, and includes an irradiating unit that irradiates infrared rays indicating a preset pattern and a sensor that detects reflected infrared rays from the object. Then, based on the image data obtained by the imaging unit 116 and the change of the distance data detected by the distance measuring unit 117 with respect to the time axis, the size and movement of the user's finger (for example, by the CPU 101 of the control device 100) (for example, The detection process of the operation (gesture) for tapping the table with the fingers is realized.

  Since a technique for detecting a user's gesture by pattern irradiation / detection using an image and infrared rays is publicly known (for example, KINTECT (trademark) of Microsoft Corporation in the United States), details thereof are omitted here. However, the configuration and the principle are not particularly limited as long as the user's gesture can be detected. Further, if there is a sufficient space in the housing of the projection imaging device 110, these may be integrated by incorporating the circuit constituting the control device 100 in the projection imaging device 110.

  In the above configuration, when the power of the apparatus is turned on, the CPU 101 executes the boot program stored in the ROM 102, loads the OS 108a stored in the external storage device 108 into the RAM 103, and executes it. Then, under the control of the OS, the application program 108b of the facing system is loaded from the external storage device 108 to the RAM 103 and executed. As a result, this system functions as a face-to-face system for banking business.

  FIG. 3 shows a usage pattern of the system shown in FIG. In this embodiment, as shown in the figure, a face-to-face system is assumed in which a teller (customer) 401 and a customer 402 sandwich a table 400 between them and sit on a chair and face each other. In the table 400, the display 120 and the input device 130 are arranged at positions where the teller 401 can be easily operated. Further, since the projection imaging device 110 irradiates the projection light from the oblique direction toward the table 400 by the top plate 113, the projection imaging device 110 is placed at a position that does not get in the way when the teller 401 and the customer 402 face each other. It is possible. As a result, a sufficient space is secured between the teller 401 and the customer 402 in the table 400, and the detection range 410 of the imaging unit 116 and the distance measurement unit 117 and the projection range 411 by the projection unit 115 are secured.

  Note that the rectangles indicated by the detection range 410 and the projection range 411 are for indicating that the respective ranges are defined on the table 400 that is a projection surface of an image such as a sheet image, and the boundary line is displayed. It is not intended to be. In addition, the control device 100 is arranged below the table 400 on the teller side, for example.

Next, basic processing of the facing system will be described with reference to FIG. This basic processing is realized by the CPU 101 reading the application program 108 b stored in the external storage device 108 and executing it on the RAM 103.
The processing contents of the face-to-face system will be described as a processing procedure of the CPU 101 in particular.

  When this system is operated, various sheet images are projected and displayed from the projection imaging apparatus 110 on the table 400 (the projection range 411) between the teller 401 and the customer 402. Then, the teller 401 gives explanations and explanations each time, and the customer 402 shows the intention of understanding the projection contents and the explanation of the teller 401, and proceeds with the procedure. If this procedure is shown by the processing of the CPU 101, it is as follows. The CPU 101 selects a sheet image to be projected from the external storage device 108, displays the selected sheet image on the projection unit 115, and operates the teller 401 and the customer 402 as gestures, and the imaging unit 116 and the distance measurement unit 117. Will be recognized. Then, the selection of the sheet image to be displayed and the like are repeated, and when the gesture indicating that the teller 401 and the customer 402 have agreed is detected in the display of the final sheet image, the processing is ended.

  This will be described in more detail below. FIG. 4 is a flowchart showing the processing procedure of the CPU 101 when the application program 108b of the face-to-face system is executed.

  When this application is executed, first, in step S101, the CPU 101 displays a menu screen on the display 120 and causes the teller 401 to select one of the items. The items to be displayed include items that are face-to-face procedures at banks such as “mortgage borrowing procedure” and “account opening procedure”. However, one of the items includes “environment setting”.

  In step S <b> 102, the CPU 101 determines whether an item in the selection menu is selected by operating the input device 130 by the teller 401. If there is no selection (NO in step S102), the process waits until it is selected. On the other hand, if there is a selection (YES in step S102), in step S103, the CPU 101 determines whether or not the selected item is “environment setting”.

  If the result of determination is environment setting (YES in step S103), in step S104, the CPU 101 performs processing related to environment setting. This environment setting is for setting whether the projection imaging apparatus 110 is installed on the right side or the left side when viewed from the teller 401. This is because the sheet image projected by the projection imaging apparatus 110 on the table 400 has orientations of the teller side and the customer side, and the orientation and the positional relationship between the teller 401 and the customer 402 attached to the seat are taken. . When the setting is completed, the CPU 101 saves the setting contents as environment information 108 d of the external storage device 108.

  Note that this environment setting need not be performed unless the arrangement position of the projection imaging apparatus 110 is changed. That is, when the sheet image is projected on the projection unit 115, the CPU 101 displays the sheet image rotated by 180 ° as necessary according to the information stored in the environment information 108d. Also, at the initial stage of startup, the CPU 101 displays a test message indicating in which direction the currently set teller or customer is set according to the information stored in the environment information 108d, and sets the environment setting. May be notified to the teller 401. For example, a message “We accept consultation” which appears upright when viewed from the customer 402 is displayed. If the message looks upside down as seen from the teller 401, it can be determined that the correct environment setting has been made.

  As a result of the determination, if the environment is not set (NO in step S103), that is, the teller 401 selects the corresponding menu item by operating the input device 130 for the purpose of consultation of the customer 402. When the CPU 101 detects that this selection has been made, in step S105, the CPU 101 refers to the sheet DB 108c of the external storage device 108 and reads the GUI configuration information of the selected procedure. In step S106, the CPU 101 projects a sheet image on the table 400 based on the read GUI configuration information, and controls the display of the sheet image based on an operation (gesture) on the projected sheet image. Execute the process. Further, along with this display control process, the gesture operation recognition process of both the teller 401 and the customer 402 is performed, and the procedure support process is executed while supporting the exchange between the teller 401 and the customer 402. Note that, in the display control of the sheet image accompanying this gesture, the CPU 101 performs, for example, based on the contents registered in the gesture table 108e. Details of the gesture table 108e will be described later.

  Here, the configuration of the sheet DB 108c will be described with reference to FIG.

  As shown in FIG. 5, the sheet DB 108c is managed in units of menu items displayed in step S101. In each item, GUI configuration information describing the transition of the procedure and the sheet image to be used is stored. In the present embodiment, the “application for mortgage” item 300 will be described as an example, and the GUI configuration information 301 included therein will be described. The GUI configuration information 301 describes information for specifying the GUI sheet information 302 that describes a procedure related to a mortgage borrowing application and a display format of a sheet image used at that time.

  The procedures between the teller 401 and the customer 402 include, for example, (1) customer registration, (2) product description, (3) loan simulation, (4) document reading, and (5) description of future procedures. There are 5 stages. In the GUI sheet information 302 in FIG. 5, form information (“customer registration form”, “product description form”, “form description form”, which describes the display form of the sheet image (GUI) to be projected in each procedure (1) to (5). ...) is managed. Further, the form information includes a description regarding the direction of the teller side and the customer side, and a plurality of forms can be described in a nested manner.

  FIG. 5 shows a specific example of information managed for the “application for mortgage” item 300. However, for each necessary item (for example, “account opening” item 350), necessary information is provided. Needless to say, the information is managed on the sheet DB 108c as appropriate.

  In the present embodiment, in the face-to-face system as shown in FIG. 3, whether the pinch-in / pinch-out operation by the multi-touch operation on the operation screen is performed by two hands of one user from the same direction or from different directions. It is determined whether each of the two users is using the hand, and if it is the latter, execution of the pinch-in / pinch-out operation is prohibited (pinch-in / pinch-out operation is invalidated).

  Specifically, as shown in FIG. 6 (a), it is a pinch-in / pinch-out operation on the operation screen 500 with two hands that have entered from the same direction (Teller 401 is Mr. A's own left hand and right hand). Is detected, the enlargement / reduction operation of the operation screen 500 according to the operation is enabled (allowed). On the other hand, as shown in FIG. 6B, it is detected that the operation is a pinch-in / pinch-out operation on the operation screen 500 with two hands entering from different directions (the left hand of Mr. A who is the teller 401 and the right hand of the customer 402). In such a case, the enlargement / reduction operation of the operation screen 500 corresponding to the operation is disabled (prohibited). However, in such a case, one of the single touch operation of the teller 401 and the single touch operation of the customer 402 constituting the multi-touch operation is validated, and an operation (for example, a drag operation) by the single touch operation is allowed. Anyway.

  In addition, as shown in FIG. 7, when a multi-touch operation with two hands entering from the same direction is detected, the shape of the two hands is further determined, and the hands with the same orientation ( For example, in the case of a combination of a right hand and a right hand), it may be determined that the operation is a single touch operation by two users.

  Furthermore, as shown in FIG. 8, when a multi-touch operation with two hands entering from the same direction is detected, the shape of the two hands is further determined and they are in a predetermined positional relationship. If not, it may be determined that the operation is a single touch operation by two users. Here, the predetermined positional relationship means, for example, a normal correct positional relationship in which each hand is physically present, with the right hand on the right side and the left hand on the left side when a human is facing the front. In other words, when each hand is brought to the front of the face so that the back of the hand can be seen, the thumbs of each hand are in a positional relationship relative to each other. When operating the touch panel, it is generally operated with the back of the hand up, and even when using both hands, it is usually not possible to perform multi-touch operations by crossing hands. . Therefore, when the positional relationship between the two hands does not satisfy the predetermined positional relationship, it is determined that the multi-touch operation is not performed with one operator's hands (two hands).

  In performing such control, it may be determined whether the operator of the pinch-in / pinch-out operation is the teller 401 or the customer 402. As shown in FIG. 3, in this embodiment, the teller 401 and the customer 402 are seated so as to face each other with the table 400 interposed therebetween. The detection range 410 includes the projection range 411. Therefore, the direction in which the operator's hand enters the projection range 411 can be detected at a timing before the timing at which the GUI component on the operation screen is designated. Since it is known from the environmental information 108d in which direction the teller side or the customer side is, whether the operator's hand performing the pinch-in / pinch-out operation is for the teller 401 or the customer 402 is determined. It can be judged.

  Here, the determination of whether the operator of the pinch-in / pinch-out operation is the teller 401 or the customer 402 is determined from the approach direction of the hand, but is not limited to this. For example, a touch operation including a pinch-in / pinch-out operation is generally performed with an index finger, and therefore, it may be determined from the outer shape pattern and orientation of the hand protruding the index finger.

  Next, an operation screen display control process based on a user operation as described in FIGS. 6 to 8 of the present embodiment will be described with reference to FIG. This display control processing is realized by the CPU 101 reading the application program 108 b stored in the external storage device 108 and executing it on the RAM 103. This display control process has the detailed configuration of the display control process in step S106 of FIG.

  First, in step S <b> 201, the CPU 101 detects insertion (entrance) of a hand into the detection range 410 by the distance measurement unit 117 in a state where an operation screen based on the GUI configuration information 301 is projected. In step S202, CPU 101 detects the presence or absence of a touch operation and determines whether or not a touch operation has been accepted. In this determination, touch operations at a plurality of locations within the detection range 410 are detected. By detecting a touch operation at a plurality of locations, the touch operation at the plurality of locations is the single multi-touch operation of the teller 401 or the customer 402 or the operation source of the touch operation detected earlier in the processing after step S207 described later. It can be determined which of the single touch operations.

  If the result of determination is that a touch operation has not been accepted (NO in step S202), the process waits until a touch operation is accepted. On the other hand, when a touch operation is received (YES in step S202), in step S203, the CPU 101 specifies an insertion source (the teller 401 side or the customer 402 side, or both) of the hand performing the received touch operation. The insertion source information (teller direction or customer direction or both directions) indicating the specified insertion source is stored as the hand insertion source 1301 in the hand information management table 1300 (FIG. 13).

  In step S204, the CPU 101 recognizes the shape of the hand specifying the insertion source. In step S205, the CPU 101 specifies the type (right hand or left hand) from the recognized hand shape of the recognized hand. The type information (right or left) indicating the specified type is stored in association with the previously specified hand insertion source 1301 as a hand type 1302 in the hand information management table 1300 (FIG. 13). The recognition of the hand shape and the specification of the orientation thereof can be realized by using, for example, a pattern matching technique that uses a reference pattern for determining the shape and orientation of the hand.

  In step S206, the CPU 101 specifies coordinates indicating the touch position of the touch operation with the fingertip of the specified hand. For the coordinates indicating the touch position of the touch operation (contact point with respect to the operation screen), for example, the detection range 410 is defined as an XY coordinate plane, and coordinate information defined on the XY coordinate plane is acquired. For example, when two touch positions are detected, two pieces of coordinate information (X1, Y1) and (X2, Y2) are acquired and associated with the previously specified hand insertion source 1301 and hand type 1302. Then, it is stored as the contact point 1303 in the hand information management table 1300 (FIG. 13) secured on the RAM 103.

  First, in step S <b> 207, the CPU 101 determines that one hand has been inserted based on the measurement result from the distance measurement unit 117, the captured image of the imaging unit 116, and information managed in the hand information management table 1300. Determine whether there are two.

  Note that the determination in step S207 is executed during a predetermined time range after the operation screen is displayed, for example. In other words, the distance measuring unit 117 and the imaging unit 116 continuously perform measurement and imaging on the entire projection surface on which the operation screen is projected, so that an object entering the projection surface (for example, an operator's hand). ) Movement trajectory and behavior can be grasped. Here, since the principle of the advancing (approaching) direction of an object such as a hand and the detection of motion itself is known as described above, the details thereof are omitted here.

  If the number of inserted hands is one as a result of the determination, in step S212, the CPU 101 identifies the GUI component that is the target of the touch operation with the inserted hand. For this specification, the coordinate information of the touch position of the touch operation by the inserted hand specified by the information managed in the hand information management table 1300 is the GUI component arranged in the operation screen based on the GUI configuration information 301. The GUI component to be touched is specified based on whether or not the coordinate range information corresponds to the occupied area.

  In step S213, it is determined whether or not the specified GUI component is already an operation target. As a result of the determination, if it is already an operation target (YES in step S213), for example, if an operation on the GUI component has already been completed by another touch operation, the operation by the touch operation to be processed is not accepted. Then, the process proceeds to step S215. On the other hand, if it is not yet an operation target (NO in step S213), in step S214, the CPU 101 determines that the touch operation performed by the inserted hand is a single touch operation, and the single touch operation is performed. Processing (single touch operation processing) is executed. However, even when the inserted hand is one, it is possible to perform a multi-touch operation using a plurality of fingers (for example, thumb and index finger). Therefore, when it is detected that the gesture operation performed after determining the number of inserted hands in step S207 is a multi-touch operation, the multi-touch operation is accepted as it is, and a corresponding operation (for example, pinch-in) is performed. / Pinch out) may be performed.

  On the other hand, if the number of inserted hands is two as a result of the determination, in step S208, the CPU 101 manages the measurement result from the distance measurement unit 117, the captured image of the imaging unit 116, and the hand information management table 1300. Further, it is determined whether or not the directions of approach of the two inserted hands are the same direction based on the received information. In particular, this determination can be made by referring to the hand insertion source 1301 managed in the hand information management table 1300 to determine whether or not the inserted directions of the two hands are the same direction. it can.

  As a result of the determination, if they are not in the same direction (NO in step S208), the processes after step S212 are executed. On the other hand, if the result of determination is that the directions are the same (YES in step S208), in step S209, the CPU 101 determines whether or not the combinations of orientations of the two inserted hands are different. Note that this orientation determination particularly determines whether or not the combinations of orientations of the two inserted hands are different by referring to the hand type 1302 managed in the hand information management table 1300. Can do.

  As a result of the determination, if the combinations of orientations of the two inserted hands are the same (NO in step S209), for example, as shown in FIG. 7, the inserted two hands are in the same orientation. In the case of (right hands), the processing after step S212 is executed. On the other hand, if the combinations of orientations of the two inserted hands are different (YES in step S209), that is, if the combination is a right hand and a left hand, in step S210, the CPU 101 determines that the two inserted hands It is determined whether or not the hand positional relationship is correct. The determination as to whether the positional relationship between the two hands is correct refers to the coordinate information indicated by the contact point 1303 managed by the hand information management table 1300 for each of the two inserted hands. Thus, it is performed by determining whether or not the positional relationship between the two inserted hands is the predetermined positional relationship as described above with reference to FIG.

  More specifically, it is determined whether the positional relationship of the hand is correct based on the relationship of the X coordinate of the contact point 1303. In the present embodiment, for example, when two hands are inserted from the customer direction and the combination of the hands is different, the “hand type 1302” corresponding to one hand of the two hands is “ If the X coordinate of the “right” contact point 1303 is smaller than the X coordinate of the “left” contact point 1303 of the “hand type 1302” corresponding to the other hand, it is determined that the positional relationship of the hand is not correct. Also, for example, the X coordinate of the contact point 1303 where the “hand insertion source 1301” of two hands is “Teller direction” and the “hand type 1302” corresponding to one hand is “right” is If the “hand type 1302” corresponding to the other hand is smaller than the X coordinate of the “left” contact point 1303, it is determined that the positional relationship of the hand is correct.

  As a result of the determination, if the positional relationship between the inserted two hands is not correct (NO in step S201), the processing from step S212 is executed. On the other hand, when the positional relationship between the two inserted hands is correct (YES in step S210), in step S211, the CPU 101 determines that the operation with the two inserted hands is a multi-touch operation by the same user. Determination is made, and processing by multi-touch operation (multi-touch operation processing) is executed.

  Then, after executing the single touch operation process in step S214 or the multi-touch operation process in step S211, in step S215, the CPU 101 determines whether an end instruction has been accepted. This determination is performed based on, for example, the presence or absence of a button control operation (touch operation) such as an end button prepared on the operation screen.

  If the result of determination is that an end instruction has not been received (NO in step S215), processing returns to step S201. On the other hand, if an end instruction is accepted (YES in step S215), the process ends.

  Through the above processing, based on the determined operation source (operation content), the CPU 101 refers to the gesture table 108e and executes display control of the corresponding operation screen. FIG. 10 shows an example of the gesture table 108e, which realizes the operation examples and display control examples shown in FIGS.

  That is, the gesture table 108e of FIG. 10 is for controlling operations on the following operation screen.

(1) When the operation source determined to be a single touch operation is by one hand, the drag operation is performed on the operation screen according to the touch operation. (2) The operation source determined to be a single touch operation is 2 In the case of each hand of the book, execution of the drag operation on the operation screen corresponding to the user's touch operation of the previously detected hand (3) The operation source determined to be multi-touch operation is two of the same user If it is by hand, execute a pinch-in / pinch-out operation on the operation screen corresponding to the touch operation. (4) When the operation source determined to be a multi-touch operation is by another user's two hands Prohibits operations on the operation screen according to the touch operation.

  Note that the gesture table 108e illustrated in FIG. 10 is an example, and settings for operation and display control on the operation screen associated with the gesture operation (single / multi-touch operation) can be appropriately defined according to the use and purpose. . For example, in FIG. 10, when the operation source of the single touch operation is based on the hands of two operators, a drag operation is performed on the operation screen according to the touch operation of the user's previously detected hand. In this case, however, the execution of the single touch operation may be prohibited. As described above, the control device 100 also functions as a display control device that detects an operation on an image projected on the projection plane, accepts the operation, and controls display of the image.

  Moreover, in the said embodiment, although the operator's hand who performs gesture operation is made into the operation origin which performs operation of an operation screen, the approach direction to the operation screen is determined, but it is not limited to this. For example, the approach direction to the operation screen of the operation source may be determined based on the direction of the arm connected to the hand that is the operation source. The operation source is not limited to the hand or arm, and may be an indicator such as a stylus or a pen held in the hand. Or you may make it determine the approach direction to the operation screen of the operation origin by giving a marker to a hand etc. and tracking the behavior of the marker.

  In the process of FIG. 9, the number of operation sources (number of hands) is determined, and then the operation corresponding to the multi-touch operation performed thereafter is controlled. It is not limited. After detecting the multi-touch operation, the operation source and the number of the multi-touch operations are determined and then the operation corresponding to the multi-touch operation is controlled. However, the processing order is not limited to this. . For example, the number of operation sources (number of hands) may be determined, and when a multi-touch operation is detected thereafter, an operation corresponding to the multi-touch operation may be controlled.

  In the above embodiment, the case where the detection target object inserted into the detection range 410 is two hands has been described as an example. However, the target object is not limited to two, and 3 The present invention can be similarly applied to a plurality of target objects. That is, in the information management table 1300, various information about each of the plurality of objects detected in the detection range 410 is managed, and the approach direction of the operation source of the operation by the object assumed according to the number of the objects, What is necessary is just to determine a combination and positional relationship suitably.

  As described above, according to the present embodiment, even in an environment where an operation screen (operation target image) can be operated by a plurality of users, more appropriate operation and display control of the operation screen are performed according to the operation state. Can be executed.

  Note that the present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  100: control device, 101: CPU, 102: ROM, 103: RAM, 104: network I / F, 105: I / F, 106: video controller, 107: storage controller, 108: external storage device, 110: projection imaging Apparatus: 115: projection unit, 116: imaging unit, 117: distance measurement unit

In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is,
An information processing apparatus that detects a gesture operation on an image displayed on a display surface and controls an operation based on the gesture operation,
Detecting means for detecting an object entering before Symbol display plane,
Determination means for determining the number of objects detected by the detection means and the approach direction thereof;
Control means for controlling the operation of the gesture operation by the object according to the number of objects determined by the determining means and the approach direction thereof.

Claims (14)

An information processing apparatus that detects a gesture operation on an image projected from a projection unit and controls an operation based on the gesture operation,
Detecting means for detecting an object entering the projection surface on which an image is projected from the projection unit;
Determination means for determining the number of objects detected by the detection means and the approach direction thereof;
An information processing apparatus comprising: control means for controlling the operation of a gesture operation by the object according to the number of objects determined by the determining means and the approach direction thereof. When the number of the objects is plural and the approach directions of the plural objects are not the same direction as a result of the determination by the determining means, the control means performs a touch operation performed by the plural objects. The information processing apparatus according to claim 1, wherein the operation is controlled as a single touch operation by at least one of the plurality of objects. The determination means determines the combination of the directions of the plurality of objects when the number of the objects is plural and the approach directions of the plural objects are the same direction,
As a result of the determination by the determination means, the control means has a plurality of the objects, the entry directions of the objects are the same, and the combinations of the directions of the objects are the same. In some cases, the touch operation performed by the plurality of objects is controlled as a single touch operation by at least one of the plurality of objects. Information processing device. In the case where the number of the objects is plural, the approach directions of the plural objects are the same direction, and the combinations of the directions of the plural objects are different, the determination unit further includes the plural objects. Determine the positional relationship of the object,
As a result of the determination by the determination unit, the control unit has a plurality of the objects, and the entry directions of the plurality of objects are the same direction, and the combinations of the directions of the plurality of objects are different. If the positional relationship between the plurality of objects is not a predetermined positional relationship, the touch operation performed by the plurality of objects is performed as a single touch operation by at least one of the plurality of objects. The information processing apparatus according to claim 1, wherein the information processing apparatus is controlled. In the case where the number of the objects is plural, the approach directions of the plural objects are the same direction, and the combinations of the directions of the plural objects are different, the determination unit further includes the plural objects. Determine the positional relationship of the object,
As a result of the determination by the determination unit, the control unit has a plurality of the objects, and the entry directions of the plurality of objects are the same direction, and the combinations of the directions of the plurality of objects are different. When the positional relationship between the plurality of objects is a predetermined positional relationship, a touch operation performed by the plurality of objects is controlled as a multi-touch operation by the plurality of objects. The information processing apparatus according to any one of claims 1 to 3, wherein the information processing apparatus is characterized in that: An information processing apparatus that detects a gesture operation on an image projected from a projection unit and controls an operation based on the gesture operation,
When detecting a multi-touch operation by an object on a projection surface on which an image is projected from the projection unit, a determination unit that determines the number of objects and an approach direction thereof,
An information processing apparatus comprising: control means for controlling the operation of a multi-touch operation by the object according to the number of objects determined by the determining means and the approach direction thereof. If the multi-touch operation is performed by a plurality of objects and the approach directions of the plurality of objects are not the same direction as a result of the determination by the determination unit, the control unit performs the multi-touch operation. The information processing apparatus according to claim 6, wherein the operation is controlled as a single touch operation by at least one of the plurality of objects. The determination means further determines a combination of directions of the plurality of objects when the multi-touch operation is performed by a plurality of objects and the approach directions of the plurality of objects are the same direction. And
As a result of the determination by the determination means, the control means has a plurality of the objects, the entry directions of the objects are the same, and the combinations of the directions of the objects are the same. 8. The information processing apparatus according to claim 6, wherein in some cases, the multi-touch operation is controlled as a single touch operation by at least one of the plurality of objects. In the case where the multi-touch operation is performed by a plurality of objects, the approach directions of the plurality of objects are the same direction, and the combinations of the directions of the plurality of objects are different. And determining a positional relationship between the plurality of objects,
As a result of the determination by the determination unit, the control unit has a plurality of the objects, and the entry directions of the plurality of objects are the same direction, and the combinations of the directions of the plurality of objects are different. When the positional relationship between the plurality of objects is not a predetermined positional relationship, the multi-touch operation is controlled as a single touch operation by at least one of the plurality of objects. The information processing apparatus according to any one of claims 6 to 8. In the case where the number of the objects is plural, the approach directions of the plural objects are the same direction, and the combinations of the directions of the plural objects are different, the determination unit further includes the plural objects. Determine the positional relationship of the object,
As a result of the determination by the determination unit, the control unit has a plurality of the objects, and the entry directions of the plurality of objects are the same direction, and the combinations of the directions of the plurality of objects are different. When the positional relationship between the plurality of objects is a predetermined positional relationship, the multi-touch operation is controlled as the multi-touch operation by the plurality of objects. The information processing apparatus according to any one of 1 to 8. The information processing apparatus according to claim 1, wherein the object is a hand of an operator who performs the gesture operation. A method for controlling an information processing apparatus that detects a gesture operation on an image projected from a projection unit and controls an operation based on the gesture operation,
A detection step of detecting an object entering the projection surface on which an image is projected from the projection unit;
A determination step of determining the number of objects detected in the detection step and its approach direction;
A control method for controlling an information processing apparatus, comprising: a control step of controlling a gesture operation performed by the target object according to the number of target objects determined in the determination step and an approach direction thereof. A method for controlling an information processing apparatus that detects a gesture operation on an image projected from a projection unit and controls an operation based on the gesture operation,
When a multi-touch operation by an object on the projection surface on which an image is projected from the projection unit is detected, a determination step of determining the number of the objects and an approach direction thereof,
A control method for controlling an information processing apparatus, comprising: a control step of controlling an operation of a multi-touch operation by the object according to the number of objects determined in the determination step and an approach direction thereof.   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 11, or making a computer perform the control method of Claim 11 or 12.

Images