JP5244084B2 - Display control system and display control method - Google Patents

Description

  The present invention relates to a display control system and a display control method for receiving an operation input from a user on a display screen of a display device.

  Conventionally, when an image is displayed by a display device such as a projector, switching of the displayed image is generally performed using a button provided on the display device or a remote control transmitter attached to the display device. Is.

  On the other hand, using a projector and a camera, from a difference image between a captured image obtained by imaging a display area (screen, desk, etc.) of the projector with a camera and a display image displayed on the projector, A display control system that detects the position of a user's fingertip or the like has been proposed (see, for example, Patent Document 1).

  In this display control system, an image including a button is displayed on the projector, so that the user points to one of the buttons on the display screen based on the positional relationship between the position of the button and the position of the user's fingertip or the like. Thus, an operation input corresponding to the button can be performed. That is, the image on the display screen can be used as a user interface for operating the display device.

JP 2009-64110 A

  However, in the display control system described in Patent Document 1, in order to detect the position of the user's fingertip or the like from the captured image, it is necessary to use a relatively high-resolution camera, and the image processing for the captured image is complicated. Therefore, there is a problem that the introduction cost of the system becomes high. In addition, it is conceivable that a user can input an operation by touching the screen by using a touch panel display as a display device. However, as the screen size increases, the touch panel display becomes expensive, resulting in a system introduction cost. Becomes higher.

  The present invention has been made in view of the above-described reasons, and is capable of accepting an operation input from a user with respect to a display screen by a relatively simple process, and can suppress a system introduction cost and a display. An object is to provide a control method.

According to the first aspect of the present invention, a display device that displays an image, an imaging device that acquires an image including at least a part of a display screen of the display device in an imaging range, display control of the display device, and a captured image obtained by the imaging device A processing control device that performs image processing of the image, and the processing control device displays at least an image indicating a range in which an operation input from a user can be received at a position included in the imaging range of the imaging device on the display screen. One is displayed, area setting means for setting a detection area in a range corresponding to the image on the captured image, and monitoring whether there is a change in the image in the detection area according to a predetermined determination criterion from the captured image acquired at any time When a change in the image is detected, a determination unit that determines that the operation input from the user has been made to the detection area, and a control signal corresponding to the determination result of the determination unit are externally provided. And output means for outputting, in the setting mode for determining the scope of the detection region in the captured image, to distinguish between at least one of the said iconography ranges and other ranges of brightness and color on the display device The image processing apparatus includes an automatic setting unit that displays a setting image and extracts a range to be a graphic image from the captured image obtained by capturing the setting image and automatically determines a detection area range from the extraction result .

According to this configuration, the determination unit monitors the presence or absence of a change in the image in the detection area according to a predetermined determination criterion from the captured image acquired at any time, and when a change in the image is detected, the determination unit Since it is determined that there has been an operation input from the user, processing for detecting the position of the user's fingertip or the like from the captured image as in the conventional case is not necessary. That is, since it is only necessary to detect a change in the image within the detection region set on the captured image, it is not necessary to use a high-resolution image capturing apparatus, image processing for the captured image is simplified, and system introduction costs are reduced. Can be kept low. In addition, since the automatic setting means automatically determines the detection area range from the captured image in the setting mode, it is possible to save the user from setting the detection area range.

In the invention of claim 2 , a display device for displaying an image, an imaging device for acquiring an image including at least a part of a display screen of the display device in an imaging range, display control of the display device, and imaging obtained by the imaging device A processing control device that performs image processing of an image, and the processing control device displays an image indicating a range in which an operation input from a user can be received at a position included in the imaging range of the imaging device on the display screen. At least one is displayed, and area setting means for setting a detection area in a range corresponding to the image on the captured image, and monitoring whether there is a change in the image in the detection area according to a predetermined determination criterion from the captured image acquired at any time When a change in the image is detected, a determination unit that determines that an operation input from the user has been made to the detection area, and a control signal corresponding to the determination result of the determination unit are externally provided. Output means for outputting, and display switching means for switching the display screen of the display device, wherein the determination means is the detection region set at the same position in the captured image before and after switching of the display screen. When the change in the image is detected continuously, a confirmation operation is requested to the user before confirming the operation input from the user for the detection area, and the operation input is performed only when there is a confirmation operation from the user. It has an operation confirmation function to confirm.
According to this configuration, the determination unit monitors the presence or absence of a change in the image in the detection area according to a predetermined determination criterion from the captured image acquired at any time, and when a change in the image is detected, the determination unit Since it is determined that there has been an operation input from the user, processing for detecting the position of the user's fingertip or the like from the captured image as in the conventional case is not necessary. That is, since it is only necessary to detect a change in the image within the detection region set on the captured image, it is not necessary to use a high-resolution image capturing apparatus, image processing for the captured image is simplified, and system introduction costs are reduced. Can be kept low. Further, according to this configuration, when a change in the image is detected continuously in the detection area set at the same position in the captured image before and after the display screen is switched, it is immediately determined that there has been an operation input. Therefore, when an object is left in the detection area, it is possible to prevent unintentional operation input from being performed by the object.
According to a third aspect of the present invention, in the first or second aspect of the present invention, the processing control device includes setting storage means for storing a predetermined response process in a one-to-one correspondence for each of the detection areas, The output means switches the display screen of the display device in accordance with a response process associated with the detection area where the operation input has been made and the setting storage means.

  According to this configuration, the display screen is switched as a response to the operation input from the user to the display screen of the display device, so that the user can operate as if directly interacting with the display screen and understand the operation method. It becomes easy.

  According to a fourth aspect of the present invention, in the invention according to any one of the first to third aspects, the determination means is based on the captured image and is within the detection region when compared with an initial state in which there is no operation input from the user. The determination criterion is determined such that a total value of pixel value change amounts for each of the pixels is obtained, and it is determined that there is an operation input when the total value exceeds a predetermined threshold value.

  According to this configuration, since a change in the image is detected from the amount of change in the pixel value in the detection area, it is possible to determine whether or not there is an operation input with a relatively simple calculation process.

  According to a fifth aspect of the present invention, in the invention according to any one of the first to third aspects, the determination unit obtains the number of edges from the edge image in the detection area in the captured image, and there is no operation input from the user. The determination criterion is determined so that it is determined that an operation input has been made when the amount of change in the number of edges when compared with the initial state exceeds a predetermined threshold value.

  According to this configuration, since a change in the image is detected from the amount of change in the number of edges in the detection area, an operation input does not occur to the extent that the position of the object in the detection area is slightly deviated. Generation of detection can be prevented.

The invention of claim 6 uses a display device that displays an image and an imaging device that acquires an image including at least a part of the display screen of the display device in the imaging range, and an operation input from the user to the display screen of the display device. Display control method for displaying at least one image indicating a range in which an operation input from a user can be received at a position included in the imaging range of the imaging device on the display screen. The region setting process for setting the detection region in the range corresponding to the image above, and the presence or absence of a change in the image in the detection region is monitored from the captured image acquired at any time according to a predetermined criterion, and the change in the image is detected. A determination process for determining that there is an operation input from the user in the detection area, and an output for switching the display screen of the display device according to the determination result in the determination process When determining the range of the detection region in the captured image, a setting image in which at least one of luminance and color is distinguished from the range to be the image and the other range is displayed on the display device. A display area is extracted, a range to be a graphic image is extracted from the captured image obtained by capturing the setting image, and a detection area range is automatically determined from the extraction result .

According to the present invention, in the determination process, the presence or absence of a change in the image in the detection area is monitored from a captured image acquired at any time according to a predetermined determination criterion, and when a change in the image is detected, the detection area is detected. Thus, it is determined that there has been an operation input from the user, so that the process of detecting the position of the user's fingertip or the like from the captured image is not required as in the prior art. That is, since it is only necessary to detect a change in the image within the detection region set on the captured image, it is not necessary to use a high-resolution image capturing apparatus, image processing for the captured image is simplified, and system introduction costs are reduced. Can be kept low. Further, when the range of the detection area in the captured image is determined, the range of the detection area is automatically determined from the captured image, so that it is possible to save the user from setting the range of the detection area.

The invention of claim 7 uses a display device that displays an image and an imaging device that acquires an image including at least a part of the display screen of the display device in an imaging range, and an operation input from the user to the display screen of the display device Display control method for displaying at least one image indicating a range in which an operation input from a user can be received at a position included in the imaging range of the imaging device on the display screen. The region setting process for setting the detection region in the range corresponding to the image above, and the presence or absence of a change in the image in the detection region is monitored from the captured image acquired at any time according to a predetermined criterion, and the change in the image is detected. A determination process for determining that there is an operation input from the user in the detection area, and an output for switching the display screen of the display device according to the determination result in the determination process Possess a degree, the determination in the process, when continuously in the detection area set at the same position in said captured image before and after the switching of the display screen to detect a change in the image, with respect to the detection region Before confirming the operation input from the user, a confirmation operation is requested from the user, and the operation input is confirmed only when there is a confirmation operation from the user .

According to the present invention, in the determination process, the presence or absence of a change in the image in the detection area is monitored from a captured image acquired at any time according to a predetermined determination criterion, and when a change in the image is detected, the detection area is detected. Thus, it is determined that there has been an operation input from the user, so that the process of detecting the position of the user's fingertip or the like from the captured image is not required as in the prior art. That is, since it is only necessary to detect a change in the image within the detection region set on the captured image, it is not necessary to use a high-resolution image capturing apparatus, image processing for the captured image is simplified, and system introduction costs are reduced. Can be kept low. Further, according to this configuration, when a change in the image is detected continuously in the detection area set at the same position in the captured image before and after the display screen is switched, it is immediately determined that there has been an operation input. Therefore, when an object is left in the detection area, it is possible to prevent unintentional operation input from being performed by the object.

  The present invention has an advantage that it is possible to accept an operation input from a user on the display screen by a relatively simple process, and to reduce the introduction cost of the system.

1 is a schematic system configuration diagram showing the configuration of Embodiment 1 of the present invention. It is a top view which shows the example of a display screen same as the above. It is explanatory drawing which shows the structure of the screen data table used in the same as the above. It is explanatory drawing which shows the structure of the scenario table used above. It is explanatory drawing which shows operation | movement same as the above. It is a flowchart which shows operation | movement same as the above. It is a flowchart which shows operation | movement same as the above. It is a flowchart which shows operation | movement same as the above. It is a flowchart which shows operation | movement of the setting mode same as the above. It is explanatory drawing which shows operation | movement of Embodiment 2 of this invention. It is a flowchart which shows operation | movement same as the above. It is a flowchart which shows operation | movement of Embodiment 3 of this invention.

(Embodiment 1)
As shown in FIG. 1, the display control system according to the present embodiment is installed so that a projector (display device) 1 that projects a content image onto a projection area on a desk 10 and the projection area are included in an imaging range. And a processing control apparatus 3 connected to the projector 1 and the camera 2. Here, the projector 1 and the camera 2 have a positional relationship in which the fields of view are substantially the same, and both are disposed above the desk 10 serving as a projection area.

  The processing control device 3 includes an arithmetic unit 30 that performs various arithmetic processes, an image output unit 31 connected to the projector 1, and an image input unit 32 connected to the camera 2. Further, the processing control device 3 includes an output image memory 33 for storing an image to be output to the projector 1, an input image memory 34 for storing a captured image obtained by the camera 2, a built-in internal storage device 35, and a removal A possible external storage device 36 and an I / O control unit 37 serving as an interface of an external device are provided. The I / O control unit 37 is given stimulus (for example, smell, wind, vibration, etc.) to the sense other than the visual and auditory senses of the user H and the mouse 5 and keyboard 6 as input means and the speaker 7 as sound output means. A sensory stimulation device 8 is connected. Here, the captured image obtained by the camera 2 is a grayscale image (an image having a grayscale value as a pixel value).

  The calculation unit 30 causes the projector 1 to display at least one icon indicating a range in which an operation input from the user H can be received at a position included in the imaging range of the camera 2 on the display screen 11. As a function. That is, the display screen 11 of the projector 1 has video display areas Fc1 and Fc2 (hereinafter simply referred to as “video display areas Fc” unless otherwise distinguished) for displaying appropriate content images as shown in FIG. And an image display area indicating a range of buttons Bt1 to Bt4 (hereinafter, simply referred to as “button Bt” when not particularly distinguished from each other) for receiving an operation input from the user H.

  Here, based on the display screen 11 shown in FIG. 2 designed by the system designer when creating the scenario, graphic data (representative points) indicating the display range (display range of the image) of the video display area Fc and the button Bt on the display screen 11. Coordinates, shape, size) are preset. As shown in FIG. 3, the graphic data includes “screen No.” for specifying the display screen 11, “video display area No.” for specifying the video display area Fc, and “button No.” for specifying the button Bt. Corresponding and stored in the internal storage device 35 as a screen data table Ta1.

  Further, the area setting means 41 has a function of setting a detection area in a range corresponding to each button Bt on the captured image 21 (see FIG. 5) obtained by the camera 2. In short, the range of the button Bt is presented as a graphic on the display screen 11, and the detection area is set in a range corresponding to the graphic on the captured image 21 obtained by imaging the display screen 11. Here, the data (representative point coordinates, shape, size) indicating the range of the detection area on the captured image 21 is associated with each “button No.” in the scenario table Ta2 (see FIG. 4) described later. It is stored in advance in the “detection area data on image” column.

  In addition, the calculation unit 30 acquires a captured image 21 from the camera 2 at any time, and has a function as a determination unit 42 that monitors whether there is a change in the image in the detection area from the captured image 21 according to a predetermined determination criterion. doing. When the change in the image in the detection area is detected, the determination unit 42 determines that an operation input from the user H has been made to the button Bt corresponding to the detection area. Note that how to determine whether the image has changed will be described later.

  Furthermore, when the determination unit 42 determines that there has been an operation input, the calculation unit 30 outputs a control signal according to the determination result to the outside, and displays it as an output unit that executes response processing according to the operation input A switching means 43 is provided. The display switching unit 43 has a function of switching the display screen 11 of the projector 1 in accordance with an operation input from the user H.

  With the configuration described above, the display control system performs a predetermined response process (here, switching of the display screen 11) in response to an operation input from the user H to the display screen 11 of the projector 1.

  That is, taking the case where the display control system is used for explanation of a certain product as an example, the processing control device 3 displays the explanation about the product in the video display area Fc on the display screen 11 of the projector 1 and goes to the next explanation. The image of the button Bt showing the options is displayed. In this state, since the camera 2 captures the display screen 11 as illustrated in FIG. 2, when the processing control device 3 detects the operation of the user H on the detection area corresponding to the button Bt from the captured image 21, Assuming that the button Bt is selected, the response process set in association with the button Bt is performed.

  In the following description, it is assumed that the response processing here is presented to the user H by the projector 1 and the speaker 7 using a method such as video and sound. However, the sensory stimulating device 8 applies a sense other than the sight and hearing of the user H. Stimulus (for example, smell, wind, vibration, etc.) may be returned to the user H as a response process. The image displayed as the response process may be either a still image or a moving image, and the output sound is arbitrarily set such as voice, music, environmental sound (natural sound such as water flowing sound or wind sound). The

  Here, the correspondence between the operated button Bt and the response process is determined on the scenario table Ta2 stored in the external storage device 36 as the setting storage means. That is, in the scenario table Ta2, as shown in FIG. 4, the content of “operation” (here, “screen 1, screen 2,...” Specifying the display screen 11 and “sound 1” specifying the output sound. , Sound 2,...)) For each “operation No.” specifying “button No.” for specifying each button Bt displayed on the display screen 11, and specifying response processing “button selection time” “Next operation” is stored in association with each other. As a result, the calculation unit 30 reads and executes the response process corresponding to the “button No.” of the button Bt for which the operation input from the user H has been input from the “next operation at the time of button selection” column of the scenario table Ta2.

  In the scenario table Ta2, as described above, “detection area data on the captured image” representing the range of the detection area on the captured image 21 is also stored in association with each “button No.”. Further, in the scenario table Ta2, a “button No. determination priority” indicating the priority when a plurality of buttons Bt are simultaneously operated in each display screen 11, “measurement data storage address at initial measurement” described later, “Detection threshold for detection area” is also stored in association with each “button No.”.

  FIG. 5 shows an operation example of the processing control apparatus 3 when the scenario table Ta2 of FIG. 4 is used.

  That is, in the “operation 1” that is the initial operation, the processing control device 3 displays the “screen 1” on the projector 1 and monitors the detection region corresponding to the “button 1” on the captured image 21 (FIG. 5). (A)). When the operation input from the user H to the detection area is detected, the processing control device 3 shifts to “operation 2” which is the next operation. In “Operation 2”, the processing control apparatus 3 displays “Screen 2” on the projector 1 and monitors each detection area corresponding to “Button 1” to “Button 4” on the captured image 21 (FIG. 5). (B)). When detecting an operation input from the user H in any of the detection areas, the processing control device 3 shifts to the next operation associated with each button Bt (FIGS. 5C to 5E). Thereafter, the process control device 3 operates in accordance with the correspondence between the button Bt and the response process in the scenario table Ta2.

  Hereinafter, the operation of the display control system having the above-described configuration will be described with reference to the flowchart of FIG.

  First, the processing control device 3 reads a scenario table Ta2 created in advance externally from the external storage device 36 into the internal storage device 35 (S1). Then, the calculation unit 30 sequentially executes each operation in a state where there is no operation input from the user H (hereinafter, referred to as “initial state”), so that a detection region corresponding to each button Bt is displayed on each captured image 21. Set sequentially. At this time, the arithmetic unit 30 stores the feature amount of the image of each detection area in the captured image 21, and stores the address of the stored data in the “measurement data storage address at initial measurement” column of the scenario table Ta2. The button No. "is stored in association with it (S2).

  Thereafter, the calculation unit 30 sets the variable An representing “operation No.” to an initial value “1” (S3), displays “screen 1” that is processing of “operation 1”, and outputs “sound 1”. Are executed (S4). At this time, as shown in FIG. 7, the arithmetic unit 30 reads the image data and sound data set in the “operation” column of the scenario table Ta2 corresponding to “operation No.” (S21), and performs display control ( S22) and sound output (S23) are executed. In this state, the calculation unit 30 sets a variable Bn representing “button No.” to an initial value “1” (S5), sets a detection region corresponding to “button 1” (region setting process) It is determined whether or not there is an operation input to the detection area (S6: determination process).

  Here, if it is determined that there is no input (S7: No), the calculation unit 30 increments the variable Bn (S10), and proceeds to a process for determining an operation input to the detection region corresponding to “button 2”. On the other hand, when it is determined that there is an input (S7: Yes), the calculation unit 30 stores “button No.” of the button Bt that has been input (S9), increments the variable Bn (S10), The process proceeds to a process for determining an operation input to the detection area corresponding to the button 2 ”.

  The calculation unit 30 repeats the above-described operation input determination process for all the buttons Bt corresponding to the display screen 11 being displayed (that is, until the button data ends) (S11). When the button Bt that has received the operation input is associated with the termination process on the scenario table Ta2, the operation of the display control system is terminated when the operation input to the button Bt is received (S8: Yes).

  When the operation input determination process is completed for all the displayed buttons Bt (S11: Yes), if there is no operation input to any button Bt (S12: No), the initial value “1” is set to the variable Bn. Returning to the process (S5), the operation input determination process (S6 to S11) is repeated.

  On the other hand, when there is an operation input (S12: Yes), if the operation input is only once (S13: No), the calculation unit 30 refers to the scenario table Ta2 and performs the next operation associated with the button Bt. “Operation No.” is set in the variable An (S15), the process returns to the operation execution process and the next operation is executed (S4: output process). If there are a plurality of operation inputs (S13: Yes), the calculation unit 30 identifies the button Bt according to the priority order set in advance in the scenario table Ta2 (S14), and the next action associated with the button Bt. “Operation No.” is set to the operation number An (S15), and the execution process of the operation proceeds (S4: output process).

  Here, the processing of the determination means 42 for determining the presence / absence of an image change performed when the presence / absence of an operation input from the user H to the button (detection region) Bt is determined (S6) is referred to the flowchart of FIG. To explain.

  In the present embodiment, the arithmetic unit 30 calculates the difference value between the image (feature amount) in the detection region in the captured image 21 obtained by the camera 2 and the image (feature amount) in the detection region obtained in the initial state. Is used to determine the operation input. Here, a grayscale image (image with a grayscale value as a pixel value) captured by the camera 2 is used as it is for determination of an operation input.

  That is, the process control device 3 first captures the display screen 11 with the camera 2 and stores the obtained captured image 21 in the input image memory 34 (S101). Next, the determination means 42 is a pixel in the detection area between the grayscale image Ii in the initial state (the state where there is no operation input) stored in the scenario table Ta2 and the grayscale image I in the input image memory 34. The difference between the pixel values is taken for each time, and the total Si of the difference values of each pixel in the detection area is calculated (S102).

  The determination unit 42 compares the total Si of the difference values thus obtained with a predetermined threshold value αi (S103), and determines that there is an operation input when the total Si is larger than the threshold value αi (S104). In other cases, it is determined that there is no operation input (S105). The threshold value αi is individually set for each button Bt in advance, and stored in the “detection area determination threshold value” column of the scenario table Ta2.

  According to the display control system of the present embodiment described above, the processing control device 3 determines the presence / absence of an operation input to the button Bt based on the presence / absence of an image change in the detection area on the captured image 21. In addition, the process of detecting the position of the fingertip of the user H from the captured image 21 becomes unnecessary. Therefore, since it is only necessary to detect a change in the image within the detection area set on the captured image 21, it is not necessary to use a high-resolution camera 2, and the image processing for the captured image 21 is also simplified and simplified. The system configuration can be achieved, and as a result, the introduction cost of the system is reduced.

  Further, when the grayscale image captured by the camera 2 is used as it is for the determination of the operation input as described above, the preprocessing for the captured image 21 becomes unnecessary, and the calculation amount can be reduced and the processing can be simplified. . However, the determination of the operation input is likely to be affected by the change in the ambient light, and if the entire captured image 21 becomes bright (or dark) due to the change in the ambient light, there is a possibility that the operation input is erroneously detected.

  Therefore, instead of using the gray image as it is, the operation is performed using the differential intensity image of the gray image or the ternary code image obtained by differentiating the gray image in the horizontal and vertical directions and coding the differential values into positive, negative, and zero values. It is also conceivable to perform input determination.

  When the differential intensity image is used, the determination unit 42 differentiates the image in the detection region from the grayscale image obtained by the camera 2 as a pre-process of the process S102 in FIG. A differential intensity image is created by “image processing”, CG-ARTS Association, p. 114-117]. Thereafter, the determination unit 42 calculates a pixel value difference for each pixel in the detection area between the created differential intensity image and the initial differential intensity image stored in the scenario table Ta2. The sum Sd of the difference values of each pixel is calculated.

  The determination means 42 compares the sum Sd of the difference values obtained in this way with a predetermined threshold value αd, and determines that there is an operation input when the total Sd is larger than the threshold value αd, otherwise. It is determined that there is no operation input. Thus, when using a differential intensity image, it is necessary to perform a differential process as a pre-process, which increases the amount of calculation, but has the advantage of being less susceptible to changes in ambient light than a grayscale image.

  On the other hand, when the ternary code image is used, the determination unit 42 differentiates the image in the detection region in the grayscale image obtained by the camera 2 in the horizontal direction as the preprocessing of the process S102 in FIG. A ternary code image in the horizontal direction is generated from the gradient of the above, coded into ternary values of positive, negative, and zero. Further, the determination means 42 differentiates the image in the detection region in the vertical direction, and creates a vertical ternary code image encoded into a ternary value of positive, negative, and zero from the gradient of the differential value.

  Thereafter, the determination unit 42 detects pixels in the detection area between the created ternary code image and the initial ternary code image stored in the scenario table Ta2 for each of the horizontal direction and the vertical direction. The difference between the pixel values is taken for each, and the sum Sh, Sv of the difference values of each pixel in the detection area is calculated.

  The determination means 42 determines that there is an operation input when the horizontal direction total Sh thus obtained is larger than a predetermined threshold value αh or when the vertical direction total Sv is larger than a predetermined threshold value αv. In other cases, it is determined that there is no operation input. When a ternary code image is used in this way, coding processing corresponding to the differential and the gradient of the differential value is required as preprocessing, and the amount of calculation increases, but the influence of changes in ambient light is greater than that of the differential intensity image. It becomes difficult to receive.

  By the way, the processing control apparatus 3 of the present embodiment has two operation modes: a normal mode for accepting an operation input from the user H to the button Bt in the display screen 11 and a setting mode for designating the range of the detection area. In the setting mode, automatic setting means (not shown) operates according to the flowchart of FIG.

  That is, in the setting mode, the processing control device 3 reads the scenario table Ta2 and the screen data table Ta1 from the external storage device 36 to the internal storage device 35 (S31). Then, the processing control device 3 sets an initial value “1” for the variable Gn representing “screen No.” on the display screen 11 (S32), and also sets the initial value “1” for the variable Bn representing “button No.”. Is set (S33). At this time, the projector 1 sets “screen 1” with the pixel value (luminance) of the image corresponding to “button 1” being the maximum (here 255) and the pixel value of the other region being the minimum (here zero). It displays as a business image (S34). As a result, only the area corresponding to “button 1” in “screen 1” is brightly highlighted.

  In this state, the processing control device 3 reads a captured image 21 obtained by capturing the display screen 11 with the camera 2 (S35), binarizes the captured image 21 with a preset threshold value, and exceeds the threshold value. This area is set as a detection area. The processing control device 3 associates the data (representative point coordinates, shape, size) indicating the range of the detection area thus determined with the “operation No.” and “button No.” in the scenario table Ta2. In the “detection area data on the captured image” field (S36).

  The processing control device 3 repeats the above-described processing while incrementing the variable Bn and the variable Gn (S37, S39) until the setting of the detection area is completed for all the buttons Bt on all the display screens 11 (S38). , S40).

  In this way, by displaying the setting image in which the region (image) corresponding to the button Bt for setting the detection region is distinguished from other regions, the detection region on the captured image 21 obtained by capturing the setting image is displayed. Extraction can be performed stably, setting can be performed automatically, and labor can be saved. In the setting image, the area corresponding to the button Bt and the other areas are not limited to the above-described brightness, and if the brightness and the color are distinguished from each other, the setting The detection area can be extracted from the captured image 21 obtained by capturing the image.

  In the present embodiment, the projector 1 that projects an image is shown as an example of the display device. However, the present invention is not limited to this example, and the display device 11 is used as the display device and the display screen 11 is captured by the camera 2. Also good. Thereby, even a normal display device that is not a touch panel display can be combined with the camera 2 and the processing control device 3 to constitute a display control system that performs various operations in response to an operation input by the user H. it can. Note that the display device here means all display devices such as a cathode ray tube, a liquid crystal display, a plasma display, and a rear projection display.

(Embodiment 2)
The display control system according to the present embodiment is different from the display control system according to the first embodiment in that an object that actually exists on the display screen 11 displayed by the projector 1 is used as an operation input target.

  In other words, in order to use an object as the button Bt for operation input, a uniform image is displayed in a certain illumination area including the object instead of illumination. As a result, the image displayed in the illumination range constitutes an image showing the range of the button Bt that can accept an operation input from the user H. The determination unit 42 sets a detection area in a range corresponding to the illumination area (image) in the captured image 21 and monitors operation input. That is, when the user H holds his hand over an object in the illumination area or picks up an object, the determination unit 42 detects a change in the image and determines that an operation input from the user H has been received.

  An operation example of the display control system of this embodiment will be described with reference to FIG. In the example of FIG. 10, there are three objects Ob <b> 1 to Ob <b> 3 (hereinafter simply referred to as “object Ob” when not distinguished from each other) on the display screen 11.

  The processing control device 3 firstly “screen 1” configured to illuminate illumination areas La1 to La3 corresponding to the objects Ob1 to Ob3 (hereinafter simply referred to as “illumination area La” unless otherwise distinguished). Is displayed on the projector 1 and the illumination areas La1 to La3 are monitored as detection areas on the captured image 21 (FIG. 10A). Here, when an operation input from the user H is detected for the illumination area (detection area) La corresponding to any of the objects Ob, the processing control device 3 proceeds to the next operation.

  The display screen 11 at this time illuminates only the illumination area La selected on the previous screen, and displays buttons Bt1 to Bt3 for switching to an explanation screen for the function and the like of the object Ob corresponding to the illumination area La. It is configured (FIGS. 10B to 10D). In this state, the process control device 3 monitors each detection area corresponding to the button Bt1 to the button Bt3 on the captured image 21, and when an operation input from the user H is detected, it is associated with each button Bt. The operation proceeds to the next operation (FIGS. 10E to 10J). In FIG. 10, illumination areas La1 to La3 indicated by broken lines are not used for operation input determination (that is, only illumination is performed), and thus indicate illumination areas in which no corresponding detection area is set. .

  Here, the illumination area used for the determination of the operation input is set in the “graphic data” field corresponding to the “button No.” in the screen data table Ta1 of FIG. In the illumination area, data is set in the “graphic data” column corresponding to “video display area No.”.

  As described above, by using an object that exists in the space for the operation input from the user H, the operation input target is set to the user H compared to the case where only the icon image on the display screen 11 is the operation input target. There is an advantage that it is easily recognized.

  By the way, in this embodiment, the operation input determination process is also different from that of the first embodiment, and the determination process will be described below with reference to the flowchart of FIG.

  In the present embodiment, the determination unit 42 obtains the number of edges from the edge image in the detection region in the captured image 21 obtained by the camera 2, and the amount of change in the number of edges when compared with the initial state exceeds a predetermined threshold value. In this case, it is determined that there has been an operation input.

  That is, the processing control device 3 first captures the display screen 11 with the camera 2 (S41), saves the captured image 21 obtained in the input image memory 34, and then creates an edge image in the detection region (S42). ). Here, the well-known Canny edge detection (see ["Digital Image Processing", CG-ARTS Association, p.209-210]) is used for the creation of the edge image.

  Next, the determination unit 42 calculates the total number E of edge pixels in the detection region (S43), and calculates the total number E and the total number Ei of edge pixels in the initial state stored in the scenario table Ta2. The absolute value D of the difference is calculated (S44). The determination means 42 compares the absolute value D thus obtained with a predetermined threshold value β (S45), and determines that there is an operation input when the absolute value D is larger than the threshold value β (S46). Otherwise, it is determined that there is no operation input (S47).

  According to this configuration, the presence / absence of an operation input is determined by the total number of edge pixels in the detection region, not by the difference for each pixel, so that the user H once picks up the object but immediately returns it to the original position. In such a case, even if there is a slight misalignment in the object in the detection area, it is allowed. That is, it is possible to prevent erroneous detection of an operation input by the user H when the position of the object is slightly changed in the detection area.

  Other configurations and functions are the same as those of the first embodiment.

(Embodiment 3)
The display control system according to the present embodiment is different from the display control system according to the first embodiment in that a confirmation function for confirming input contents is added.

  Here, when the determination unit 42 continuously detects an operation input in the detection area set at the same position in the captured image 21 before and after the switching of the display screen, the determination unit 42 receives the operation input from the user H for the detection area. An operation confirmation function for requesting the user H to perform confirmation operation before confirmation is provided. The determination unit 42 determines the operation input only when a confirmation operation from the user H is performed.

  In the present embodiment, the process after the operation input detection process (S1 to S14 in FIG. 6) is different from that in the first embodiment. Therefore, the process after the operation input is detected will be described below with reference to the flowchart in FIG. Note that the processing (S41 to S44) shown in FIG. 12 is inserted between the processing S14 and the processing S15 in FIG.

  That is, when the operation input detection process (S1 to S14 in FIG. 6) is completed, the calculation unit 30 is operated before the response process corresponding to the button Bt is executed. It is determined whether or not the button Bt is the same as the previously operated button Bt (that is, the same button Bt has been operated continuously) (S41).

  If it is determined that they are the same, the mode shifts to a confirmation mode in which the user H confirms the operation input of the button Bt. In the confirmation mode, the arithmetic unit 30 displays “button No.” of the button Bt to be confirmed, and a message requesting the user H for confirmation operation for determining whether or not to confirm the operation of the button Bt (for example, “If this No. is acceptable, please change your hand from the input button”) is displayed in the display area (S42). At this time, the determination means 42 determines whether or not the operation input is continuously detected with only the detection region corresponding to the button Bt being confirmed as the operation input detection target (S43). Here, when the operation input is continuously detected (S44: Yes), the message is continuously displayed. When the operation input is not detected (S44: No), the process proceeds to the next operation (S15 in FIG. 6). To do.

  According to the configuration described above, the user B keeps operating the button Bt against the intention of the user H by leaving his / her hand on the button Bt or leaving an object on the button Bt. Can avoid misoperation.

  Other configurations and functions are the same as those of the first embodiment.

1 Projector (display device)
2 Camera (imaging device)
3 Processing Control Device 11 Display Screen 41 Area Setting Unit 42 Determination Unit 43 Display Switching Unit (Output Unit)
H user

Claims (7)

Display device for displaying image, imaging device for obtaining image including at least part of display screen of display device in imaging range, display control of display device and processing control for image processing of captured image obtained by imaging device With the device,
The processing control device
The display device displays at least one icon indicating a range in which an operation input from the user can be received at a position included in the imaging range of the imaging device on the display screen, and is detected in the range corresponding to the icon on the captured image. Area setting means for setting an area;
The presence or absence of a change in the image in the detection area is monitored from a captured image acquired at any time according to a predetermined criterion, and when a change in the image is detected, there is an operation input from the user to the detection area Determination means for determining;
Output means for outputting a control signal according to the determination result of the determination means to the outside ;
In the setting mode for determining the range of the detection area in the captured image, the display device displays a setting image in which at least one of luminance and color is distinguished from the range to be the image and the other range. A display control system comprising: automatic setting means for extracting a range to be a graphic from the captured image obtained by capturing the setting image and automatically determining the range of the detection region from the extraction result . Display device for displaying image, imaging device for obtaining image including at least part of display screen of display device in imaging range, display control of display device and processing control for image processing of captured image obtained by imaging device With the device,
The processing control device
The display device displays at least one icon indicating a range in which an operation input from the user can be received at a position included in the imaging range of the imaging device on the display screen, and is detected in the range corresponding to the icon on the captured image. Area setting means for setting an area;
The presence or absence of a change in the image in the detection area is monitored from a captured image acquired at any time according to a predetermined criterion, and when a change in the image is detected, there is an operation input from the user to the detection area Determination means for determining;
Output means for outputting a control signal according to the determination result of the determination means to the outside;
Display switching means for switching the display screen of the display device,
When the determination unit detects a change in the image continuously in the detection region set at the same position in the captured image before and after switching the display screen, the determination unit receives an operation input from the user to the detection region. Table示制control system that is characterized by having an operation confirmation function for determining the operation input only when requesting confirmation operation to the user, there is a confirmation operation from the user before committing. The processing control device includes setting storage means for storing a predetermined response process in a one-to-one correspondence for each detection area, and the output means corresponds to the detection area in which the operation input has been made in the setting storage means. The display control system according to claim 1, wherein the display screen of the display device is switched in accordance with attached response processing .   The determination unit obtains a total value of changes in pixel values for each pixel in the detection region when compared with an initial state where there is no operation input from a user based on the captured image, and the total value is a predetermined value. The display control system according to any one of claims 1 to 3, wherein the determination criterion is determined so as to determine that an operation input has occurred when the threshold value is exceeded.   The determination unit obtains the number of edges from the edge image in the detection area in the captured image, and the amount of change in the number of edges when compared with an initial state in which there is no operation input from the user exceeds a predetermined threshold. The display control system according to claim 1, wherein the determination criterion is determined so that it is determined that an operation input has been made. A display control method that uses a display device that displays an image and an imaging device that acquires an image including at least a part of the display screen of the display device in an imaging range, and that accepts an operation input from a user on the display screen of the display device. And
The display device displays at least one icon indicating a range in which an operation input from the user can be received at a position included in the imaging range of the imaging device on the display screen, and is detected in the range corresponding to the icon on the captured image. An area setting process for setting an area;
The presence or absence of a change in the image in the detection area is monitored from a captured image acquired at any time according to a predetermined criterion, and when a change in the image is detected, there is an operation input from the user to the detection area A judging process for judging,
An output process for switching the display screen of the display device according to the determination result in the determination process,
When determining the range of the detection area in the captured image, the display device displays a setting image in which at least one of luminance and color is distinguished from the range to be the graphic and the other range, and for setting Viewing control how to characterized by automatically determining the range of the detection area from the extracted extraction result a range of the iconography from the captured image of the captured image. A display control method that uses a display device that displays an image and an imaging device that acquires an image including at least a part of the display screen of the display device in an imaging range, and that accepts an operation input from a user on the display screen of the display device. And
The display device displays at least one icon indicating a range in which an operation input from the user can be received at a position included in the imaging range of the imaging device on the display screen, and is detected in the range corresponding to the icon on the captured image. An area setting process for setting an area;
The presence or absence of a change in the image in the detection area is monitored from a captured image acquired at any time according to a predetermined criterion, and when a change in the image is detected, there is an operation input from the user to the detection area A judging process for judging,
Possess an output process for switching the display screen of the display device according to the determination result of the determination process,
In the determination process, when a change in the image is continuously detected in the detection area set at the same position in the captured image before and after switching of the display screen, an operation input from the user to the detection area is performed. A display control method, wherein a confirmation operation is requested to the user before confirmation, and the operation input is confirmed only when the confirmation operation is performed by the user .

Images