JP2019016141A - Input device and input method - Google Patents

Abstract

To provide an input device and an input method capable of suppressing erroneous detection of an instruction input.SOLUTION: An input device comprises a foreground mask generation part 12 generating a foreground mask indicating whether a pixel value of pixels of a current image which is one frame of a captured image and an image value of pixels having same coordinates of a background image are different, a foreground image generation part 13 masking the current image with a foreground mask to generate a foreground image, an instruction area setting part 14 setting an instruction area on coordinates of pixels not included in a pixel area corresponding to the foreground image of pixels of an imaging frame of an imaging part, an image for operation display part 17 displaying an image for operation being formed by overwriting an instruction area icon image indicating the instruction area on an operation object image and a second foreground image which is a foreground image generated after setting of the instruction area, and an instruction input determination part 18 determining that an instruction input of the predetermined operation is performed if it is determined that there is an overlapping portion of the instruction area of the pixels of the imaging frame and a pixel area corresponding to the second foreground image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an input device and an input method.

  2. Description of the Related Art Conventionally, there has been proposed an input device that captures an image of a user and determines that an instruction input for a predetermined operation has been made when the user's hand on the image obtained by the image touches the instruction area (see Patent Document 1). . In the input device described in Patent Document 1, the position of the user's eyes and the position of the hand on the image are detected, an instruction area is set at an intermediate position between the detected eye position and the hand position, and the instruction is performed. When the pixel value of the pixel in the area changes, it is determined that the user's hand on the image has touched the instruction area.

Japanese Patent Application No. 2009-259117

However, when the input device described in Patent Document 1 is used at home or the like, a moving body such as a pet may appear in addition to the user. In this case, for example, if a moving body such as a pet is reflected between the position of the user's eyes and the position of the hand on the image, an erroneous detection of an instruction input unintended by the user occurs.
The present invention focuses on the above-described problems, and an object thereof is to provide an input device and an input method capable of suppressing erroneous detection of an instruction input.

  In order to solve the above-described problem, an aspect of the present invention is as follows. (A) A current image acquisition unit that acquires one frame of a video output by an imaging unit that continuously images a predetermined space as a current image; and (b) A background image storage unit that stores a background image generated from a video imaged in advance with the same angle of view as the current image; and (c) a pixel value of a pixel of the current image and an image value of a pixel having the same coordinates as the background image A foreground mask generating unit that generates a foreground mask indicating whether or not are different, (d) a foreground image generating unit that generates a foreground image by masking the current image with a foreground mask, and (e) an imaging frame of the imaging unit An instruction area setting unit that sets an instruction area at the coordinates of a pixel that is not included in the pixel area corresponding to the foreground image among the pixels; (f) an instruction area icon image that indicates the instruction area in a predetermined operation target image; An operation image for displaying an operation image overwritten with a second foreground image, which is a foreground image generated based on the current image acquired after setting the display area, on an image display unit directed to a predetermined space A display unit; (g) a pixel region overlap determination unit that determines whether or not there is an overlapping portion between the pointing region and the pixel region corresponding to the second foreground image; and (h) the pointing region and the second foreground image. The gist of the present invention is that the input device includes an instruction input determination unit that determines that there has been an instruction input for a predetermined operation when it is determined that there is an overlapping portion with the pixel region corresponding to.

  Another aspect of the present invention is an input device including: (a) an imaging unit that continuously images a predetermined space; and (b) an image display unit that displays an operation image. The image for use is an image in which an instruction area is set, and a predetermined operation target image is overwritten with a current foreground image and an instruction area icon image indicating the instruction area, and (d) a current foreground image. Is formed from pixels of the current image acquired from one frame of the video output by the imaging unit, and pixels having a pixel value different from those of the background image captured in advance at the same angle of view as the current image. (E) the indication area is set to an area not included in the pixel area corresponding to the past foreground image generated at the time when the indication area is set, and the indication area is set Current foreground image generated later And summarized in that at least a portion of which is an input device used to determine whether or not to present at least a portion of the instruction region.

  In another aspect of the present invention, (a) a step of acquiring one frame of a video output from an imaging unit that continuously captures a predetermined space as a current image, (b) a pixel value of a pixel of the current image, and Generating a foreground mask indicating whether or not the image value of a pixel having the same coordinates of a background image captured at the same angle of view as the current image is different; and (c) masking the current image with the foreground mask. A step of generating an image; and (d) selecting one or more pixels from pixels not included in the pixel region corresponding to the foreground image among the pixels of the imaging frame of the imaging unit and setting an indication region at the coordinates thereof; (E) An operation image formed by overwriting an instruction area icon image indicating an instruction area on a predetermined operation target image and a second foreground image that is a foreground image generated after setting the instruction area. A step of displaying toward the constant space; and (g) determining whether or not there is an overlapping portion between the designated area and the pixel area corresponding to the second foreground image among the pixels of the imaging frame of the imaging unit. And (h) determining that there is an instruction input for a predetermined operation when it is determined that there is an overlapping portion between the instruction area and the pixel area corresponding to the second foreground image. This is the gist.

  According to the present invention, for example, a portion in which a moving body other than the user, such as a pet, is included in the foreground image, and no indication area is set in the pixel area corresponding to the foreground image. An input device and an input method that can be suppressed can be provided.

It is a conceptual diagram showing the structure of the input device which concerns on embodiment of this invention. It is a block diagram showing the internal structure of a processor. It is a block diagram showing the internal structure of a memory | storage device. It is a flowchart which shows a background image storage process. It is a block diagram showing the internal structure of the processor which concerns on a modification. It is explanatory drawing for demonstrating a background image. It is a flowchart which shows execution permission determination processing. It is a block diagram showing the internal structure of the processor which concerns on a modification. It is a flowchart which shows an instruction | indication detection loop process. It is explanatory drawing for demonstrating a current image. It is explanatory drawing for demonstrating the production | generation method of a foreground mask. It is explanatory drawing for demonstrating the production | generation method of a foreground image. It is explanatory drawing for demonstrating the setting method of an instruction | indication area | region. It is explanatory drawing for demonstrating the setting method of an instruction | indication area | region, (a) is a figure which shows a current image, (b) is a figure which shows a foreground image. It is explanatory drawing for demonstrating the operation target image. It is explanatory drawing for demonstrating the image for operation. It is explanatory drawing for demonstrating the mirror image of a foreground image. It is explanatory drawing for demonstrating resizing of a foreground image. It is explanatory drawing for demonstrating the determination method of instruction input.

Hereinafter, an input device according to an embodiment of the present invention will be described with reference to the drawings. The input device according to the embodiment of the present invention is for changing a pitch in karaoke, for example.
The following embodiments exemplify apparatuses and methods for embodying the technical idea of the present invention, and the technical idea of the present invention includes the shape, structure, arrangement, etc. of components. It is not specified to the following. The technical idea of the present invention can be variously modified within the technical scope defined by the claims described in the claims.

(Constitution)
As shown in FIG. 1, the input device 1 according to the embodiment of the present invention includes an imaging unit 2, an image display unit 3, and a computer 4 connected to the imaging unit 2 and the image display unit 3.
The imaging unit 2 continuously images the space in front of the image display unit 3 (hereinafter also referred to as “predetermined space”). Then, the video data of the captured video is output to the computer 4. As the imaging unit 2, for example, a WEB camera, a video camera, or a camera attached to a personal computer can be used.

The image display unit 3 displays a karaoke movie or the like in a predetermined space in accordance with a signal from the computer 4. For example, a liquid crystal display can be used as the image display unit 3.
The computer 4 includes hardware resources such as a processor 5 and a storage device 6.

  As shown in FIG. 2, the processor 5 includes a background image storage determination unit 7, a face image detection unit 8, a background image writing unit 9, a current image acquisition unit 10, an execution permission determination unit 11, a foreground mask generation unit 12, a foreground. Hardware resources such as an image generation unit 13, an instruction region setting unit 14, a pixel region overlap determination unit 15, an operation target image acquisition unit 16, an operation image display unit 17, and an instruction input determination unit 18 are logically provided. In FIG. 2, the background image storage determination unit 7 and the like formally express hardware resources focusing on logical functions. That is, the representation of the internal structure of the processor 5 shown in FIG. 2 does not necessarily mean a dedicated integrated circuit or functional block that exists independently as a physical area on the semiconductor chip, but is general-purpose in terms of software. It is also possible to realize a function equivalent to the background image storage determination unit 7 and the like by controlling the circuit of the computer system.

  As illustrated in FIG. 3, the storage device 6 logically includes hardware resources such as a background image storage unit 19 and a foreground information storage unit 20. In FIG. 3, the background image storage unit 19, foreground information storage unit 20, and the like formally express hardware resources focusing on logical functions. Therefore, the expression of the storage device 6 shown in FIG. 3 does not necessarily mean a storage device that exists independently as a physical area on the substrate. As the background image storage unit 19 and the foreground information storage unit 20, for example, a format such as a file stored in different areas in one main storage device can be adopted, and a data structure can be constructed.

(Background image storage processing)
Next, background image storage processing executed by the processor 5 will be described. The background image storage process is executed when a predetermined start operation is performed on the computer 4 by the user, and then executed every time a predetermined time (for example, 1 second) elapses. If an instruction area to be described later is set, the execution of the process is interrupted, and the update of the background image is interrupted.
As shown in FIG. 4, in step S101, the background image storage determination unit 7 performs one frame (hereinafter also referred to as “candidate image”) of the video indicated by the video data output from the imaging unit 2, that is, a current described later. An image generated from video captured at the same angle of view as the image is acquired.

  Subsequently, the process proceeds to step S102, where the background image storage determination unit 7 and the face image detection unit 8 determine whether to update and store the background image based on the candidate image acquired in step S101. As a method for determining whether or not the background image is updated and stored, for example, the following method can be used. First, the face image detection unit 8 detects a face image from candidate images. Then, when the face image detection unit 8 does not detect a face image of a predetermined size or larger, the background image storage determination unit 7 determines to update and store the background image (Yes), and proceeds to step S103. On the other hand, when a face image of a predetermined size or more is detected by the face image detection unit 8, it is determined not to store the candidate image as a background image (No), and this calculation process is terminated.

  In the input device 1 according to the embodiment of the present invention, the background image storage determination unit 7 updates and stores the background image when the face image detection unit 8 does not detect a face image of a predetermined size or larger. Although an example of determination is shown, another configuration that does not include the face image detection unit 8 may be employed. For example, when it is determined that the user has performed an operation for storing the background image in the computer 4, the background image may be determined to be updated and stored. Further, for example, as shown in FIG. 5, a human detection unit 21 such as a human sensor that detects a specific heat source from a range within a predetermined distance (for example, 50 [cm]) from the imaging unit 2 is provided. It is good also as a structure which determines with acquiring a background image, when a specific heat source is not detected by this.

  In step S103, as shown in FIG. 6, the background image writing unit 9 uses the candidate image acquired in step S101, that is, the image generated from the video imaged at the same angle of view as the current image as the background image. After storing the image in the image storage unit 19, the calculation process is terminated. As a result, the background image storage unit 19 stores one frame of a video in which the user's face is not reflected in a predetermined size or more as a background image, and updates and stores the background image one after another for a predetermined time.

  The input device 1 according to the embodiment of the present invention is an example in which a candidate image is stored as a background image instead of the background image stored in the background image storage unit 19, that is, the background image is quickly updated to the candidate image. Although the example which does is shown, you may employ | adopt another structure. For example, the update may be performed after waiting for a predetermined time (for example, 1 second), that is, the background image stored at the past time for the predetermined time may be used. Alternatively, for each pixel of the candidate image, the difference between the pixel value of each pixel and the pixel value of the pixel of the stored background image corresponding to that pixel is calculated, and the calculated difference is divided by a predetermined integer value N. It is also possible to use a method of calculating the integer value by returning the sign after rounding up the first decimal place of the absolute value of the quotient obtained and adding the calculated integer value to the pixel value of the pixel of the background image. As the pixel value, for example, Y value of YCbCr (hereinafter also referred to as “luminance value”), RGB can be used.

  Therefore, for example, when the pixel value is represented by 8 bits (that is, an integer from 0 to 255) and the integer value N is “255”, the pixel value of the pixel of the background image is “1” for each candidate image. It approaches the pixel value of this pixel. Therefore, for example, when a black wall image is included in the background image, but the black wall is changed to a wall that shines in white by direct sunlight, the black wall included in the background image is 255 frames later. Changes to an image of a wall that glows white. After that, when the direct sunlight is no longer applied and the white wall returns to the black wall, the image of the white wall included in the background image returns to the black wall image again after 255 frames.

  Further, the input device 1 according to the embodiment of the present invention shows an example in which one frame (candidate image) of a video indicated by video data output from the imaging unit 2 is used as a background image to be stored in the background image storage unit 19. However, other configurations can be employed. For example, an image generated from a video imaged with the same angle of view as a current image, which will be described later, by a camera separate from the imaging unit 2 can be used. In this case, the background image stored in the background image storage unit 19 is not updated.

(Execution permission judgment process)
Next, an execution permission determination process executed by the processor 5 will be described. The execution permission determination process is executed when the user performs a predetermined start operation on the computer 4. As the start operation, for example, the same operation as the start operation of the background image storage process can be used.
As illustrated in FIG. 7, in step S <b> 201, the current image acquisition unit 10 acquires one frame (hereinafter also referred to as “current image”) of the video indicated by the video data output from the imaging unit 2.

  Subsequently, the process proceeds to step S202, where the execution permission determination unit 11 determines whether to permit execution of an instruction detection loop process (described later) based on the current image acquired in step S201. As a method for determining whether or not to allow execution of the instruction detection loop process, for example, a face image having a size equal to or larger than a predetermined size (for example, defined by horizontal × vertical pixel count) that is predetermined from the current image is used. A method of determining that execution of the instruction detection loop process is permitted when it is detected can be used. The predetermined size representing the size of the face image may be configured to be set for each user, for example. And when it determines with permitting execution of an instruction | indication detection loop process (Yes), it transfers to step S203. On the other hand, when it determines with not permitting (No), it returns to step S201.

  In the input device 1 according to the embodiment of the present invention, the example in which execution of the instruction detection loop process is permitted when a face image of a predetermined size or more is detected from the current image is shown, but another configuration is adopted. You can also For example, as shown in FIG. 5, a human detection unit 21 such as a human sensor that detects a specific heat source from a range within a predetermined distance (50 cm) from the imaging unit 2 is provided. It may be configured to permit the execution of the instruction detection loop process when the heat source is detected.

Further, for example, as shown in FIG. 8, a personal authentication unit 22 that authenticates an individual based on the current image is provided, and only when a specific individual is authenticated by the personal authentication unit 22, execution of the instruction detection loop process is permitted. It is good also as composition to do. As an authentication method, for example, face image authentication can be used. Thereby, only a specific user can input an instruction to the computer 4.
In step S203, the instruction detection loop process is executed, and then the process returns to step S201.

(Instruction detection loop processing)
Next, the instruction detection loop process executed by the processor 5 will be described.
As shown in FIG. 9, in step S301, the current image acquisition unit 10 acquires one frame of the video indicated by the video data output from the imaging unit 2, that is, the current image, as shown in FIG.

  In step S302, the foreground mask generation unit 12 and the foreground image generation unit 13 generate foreground information including the foreground mask and the foreground image based on the current image acquired in step S301. As the foreground information generation method, for example, the following method can be used. First, the foreground mask generation unit 12 generates a foreground mask indicating whether or not the pixel value of the pixel of the current image is different from the image value of the pixel having the same background image coordinates stored in the background image storage unit 19. To do. For example, a foreground mask is generated by giving “255” to pixels whose image values are different between pixels of the same coordinates in the current image and the background image stored in the background image storage unit 19 and giving “0” to the other pixels. To do. That is, as shown in FIG. 11, the difference between the luminance value of each pixel of the background image stored in the background image storage unit 19 and the luminance value of each pixel of the current image is calculated for each pixel, and the calculated difference is calculated. A foreground mask is obtained by giving “255” to pixels whose absolute value is equal to or greater than a predetermined value and giving “0” to other pixels.

  In the case where the background image storage unit 19 stores a plurality of background images, the background image used for generating the foreground mask is a predetermined time (for example, compared to the time of execution of step S301, that is, the acquisition of the current image). (1 second) The background image stored more than the previous time is used. Here, as described above, the instruction detection loop process is specified when a user sufficiently approaches the imaging unit 2 and a face image of a predetermined size or more is detected, or from a range within a predetermined distance from the imaging unit 2. Execution is permitted when a heat source is detected. Therefore, the background image acquired immediately before the acquisition of the current image is hardly different from the current image, and an appropriate foreground mask may not be generated. On the other hand, in the background image stored more than a predetermined time before the current image is acquired, there is a high possibility that the position of the user has changed greatly. Therefore, an appropriate foreground mask can be generated by using a background image stored a predetermined time or more before the current image is acquired.

  Subsequently, the foreground image generation unit 13 generates a foreground image by masking the current image with the foreground mask generated by the foreground mask generation unit 12. For example, as shown in FIG. 12, an image composed of pixels corresponding to a pixel whose pixel value of the foreground mask is “255” among the pixels of the current image is set as the foreground image. At this time, the foreground mask may be repeatedly subjected to expansion processing and contraction processing to remove sesame salt noise from the generated foreground image. In order to easily superimpose the foreground image on the background image, a frame having the same number of horizontal × vertical pixels as each frame of the video imaged by the imaging unit 2 (hereinafter also referred to as “imaging frame of the imaging unit 2”) is used. Alternatively, “0” may be set to the pixel value of the pixel having coordinates other than the foreground image. Then, the foreground image generation unit 13 generates foreground information including the generated foreground mask and the foreground image, and stores the generated foreground information in the foreground information storage unit 20. As the foreground image included in the foreground information, for example, an edge image representing only the contour line of the foreground image, that is, the edge portion where the luminance value or the like changes rapidly may be stored. Alternatively, the outline of the foreground image (same as the boundary line of “255” and “0” in the foreground mask) may be stored.

  Subsequently, the process proceeds to step S303, where the instruction area setting unit 14 determines whether an area for instructing a pitch change (hereinafter also referred to as “instruction area”) has been set. And when it determines with the instruction | indication area having been set (Yes), it transfers to step S310. On the other hand, if it is determined that it has not been set (No), the process proceeds to step S304.

  In step S304, the indication area setting unit 14 selects one or more pixels from the pixels not included in the pixel area corresponding to the foreground image included in the foreground information generated in step S302 among the pixels of the imaging frame of the imaging unit 2. After selecting and setting an instruction area at the coordinates, the process proceeds to step S306. For example, the instruction area is set to the coordinates of pixels not included in the pixel area corresponding to the foreground image included in the foreground information among the pixels of the current image. A predetermined operation (for example, a pitch raising operation or a pitch lowering operation) that is determined in advance is associated with the instruction area. As a setting method of the instruction area, for example, a method of setting the instruction area to the coordinates of pixels not included in the pixel area corresponding to the foreground image among the coordinates of the four corners of the frame captured by the imaging unit 2 may be used. it can. As the indication area, for example, one or a plurality of pixels, a pixel on a boundary line of a circular or polygonal area, or a pixel in the area can be used. In addition, when an instruction area icon image (described later) is overwritten in advance on an operation target image (described later), the same or similar shape as the instruction area icon image can be used.

  When the foreground information storage unit 20 stores foreground information of a plurality of frames, the foreground information (foreground image) used for setting the instruction area is not limited to one, and a plurality of foreground information may be used. In other words, the indication area may be set to the coordinates of pixels not included in any of the pixel areas corresponding to the foreground images of a plurality of frames.

  As a specific method for setting the instruction area, for example, the face image area on the foreground image is detected, and the coordinates of pixels not included in the pixel area corresponding to the foreground image among the pixels in the vicinity of the detected area. May be used. For example, if it is determined that a candidate pixel having a predetermined positional relationship with the face image area on the foreground image is a pixel that is not included in the pixel area corresponding to the foreground image, the indication area is set to the coordinates of the candidate pixel. Set. As the candidate pixels, as shown in FIG. 13, a predetermined distance (for example, twice the width of the face image) in a predetermined direction (for example, right or left) from a predetermined base point (for example, the center) of the face image area. It is preferable to select a pixel at the coordinates. For example, whether the right side or the left side is used may be determined based on the user's dominant arm, such as right-handed or left-handed. In the example of FIG. 13, the foreground image is represented by an edge extracted as a contour line, that is, a pixel whose luminance value etc. changes abruptly compared with surrounding pixels. As an algorithm for extracting the contour line, for example, the Canney method proposed by John F. Canny in 1986 is appropriate.

  As shown in FIGS. 14A and 14B, when the candidate pixel is included in the pixel area corresponding to the foreground image, the instruction area may not be set. In the example of FIGS. 14A and 14B, a chair on which a dog-type robot that was not in the background image is placed appears in the current image, and candidate pixels are included in the area where the chair is shown, corresponding to the foreground image. Candidate pixels are included in the pixel area. As a result, only the operation target image and the foreground image are displayed on the image display unit 3, and an instruction area icon image to be described later is not displayed on the image display unit 3, so that the user is aware why the instruction area icon image is not displayed. be able to. Therefore, the user can display the indication area icon image on the image display unit 3 by tilting his / her upper body slightly to the left or right to keep the position where his / her face is reflected away from the chair. As a result, it is possible to prevent erroneous detection of an instruction input that is not intended by the user, and to prevent malfunction. In the example of FIG. 14B, the foreground image is represented by a contour line as in FIG.

Incidentally, for example, even when the candidate pixel is included in the pixel area corresponding to the foreground image, in the method of setting the instruction area at the coordinates of the candidate pixel, a moving body such as a pet appears in the current image, and A pixel area corresponding to an image of a moving body such as a pet overlaps with an instruction area, and a malfunction occurs as soon as the instruction area is set.
In the input device 1 according to the embodiment of the present invention, an example is shown in which it is determined whether or not the instruction area has been set, and when it is determined that the instruction area has not been set, the instruction area is set. It is also possible to adopt the configuration. For example, the indication area may be set and updated sequentially in order to follow the change in the position of the face to be reflected.

In step S305, the pixel area overlap determination unit 15 includes the instruction area set in step S304 and the foreground image included in the foreground information generated in step S302 among the pixels of the imaging frame of the imaging unit 2, that is, the instruction area. It is determined whether or not there is an overlapping portion with a pixel area corresponding to a foreground image generated after setting (hereinafter also referred to as a “second foreground image”). Here, as described above, the foreground image has the same shape as the region of the pixel whose pixel value of the foreground mask is “255”. Therefore, as a method for determining whether or not there is an overlapping portion, for example, without using the second foreground image, a pixel area whose pixel value of the foreground mask is “255”, and a pixel corresponding to the indication area It is also possible to use a method for determining whether or not there is an overlapping portion.
As a method for determining whether or not there is an overlapping portion, for example, the pixel value of the pixel in the current area of the current image is the same as the pixel value of the pixel having the same coordinates of the current image acquired when setting the current area. When it is determined that the absolute value of the difference (hereinafter also referred to as “pixel value difference”) is greater than or equal to a predetermined value, there is an overlapping portion between the designated area and the pixel area corresponding to the second foreground image Can be used. Further, for example, when there are a plurality of pixels in the indication area, the number of pixels determined that the absolute value of the pixel value difference is equal to or greater than a predetermined value is equal to or greater than a predetermined ratio of the number of all the pixels in the indication area. In this case, a method of determining that there is an overlapping portion between the designated area and the pixel area corresponding to the second foreground image may be used. If it is determined that there are overlapping portions (Yes), the process proceeds to step S310. On the other hand, if it is determined that there is no overlapping portion (No), the process proceeds to step S306.

  In step S306, as illustrated in FIG. 15, the operation target image acquisition unit 16 acquires one frame of a karaoke movie of the karaoke system that is the operation target (hereinafter also referred to as “operation target image”). In the example of FIG. 15, one frame of a “Kimigayo” karaoke movie in which a title, a graphic of the Japanese archipelago, and lyrics are displayed is used as the operation target image. In the example of FIG. 15, the number of horizontal × vertical pixels of the operation target image is larger than the number of horizontal × vertical pixels of the imaging frame of the imaging unit 2, that is, the number of horizontal × vertical pixels of the foreground mask. Then, the horizontal × vertical number of pixels in the display area of the figure of the Japanese archipelago of the operation target image is made the same as the horizontal × vertical number of pixels in the imaging frame of the imaging unit 2 (horizontal × vertical number of pixels in the foreground mask). . When step S306 is executed once or more after the instruction detection loop processing is started, that is, when the operation target image is acquired one or more times, the next frame of the operation target image acquired immediately before is obtained. Are acquired as operation target images, and each frame is sequentially acquired.

  In the input device 1 according to the embodiment of the present invention, one frame of a karaoke movie, that is, one frame of a moving image is used as an operation target image, and each frame is sequentially acquired. It can also be adopted. For example, a configuration using still images may be used. In this case, it is necessary to generate the operation target image (still image) over and over again by generating the operation target image with the instruction area icon image overwritten, that is, the still image only once and repeatedly using the generated still image. There is no need. As a still image, for example, one page of an operation manual for factory equipment may be used, or a background image itself may be used.

  In step S <b> 307, the operation image display unit 17 generates an image for performing a pitch change operation (hereinafter also referred to as “operation image”). As an operation image generation method, for example, as shown in FIG. 16, the foreground image included in the foreground information generated in step S302 in the operation target image acquired in step S306 and the indication area set in step S304 are shown. A method of overwriting the instruction area icon image can be used. The instruction area icon image overwrites the pixel of the coordinate corresponding to the instruction area among the pixels of the operation target image. For example, an image corresponding to an action associated with the instruction area is used. In the example of FIG. 16, two instruction areas are set, and the instruction area icon images “▲” and “▼” are overwritten in each instruction area. “▲” is an icon image for pitch up operation, and “▼” is an icon image for pitch down operation. In the example of FIG. 16, the foreground image is overwritten on the graphic display area of the Japanese archipelago of the operation target image. In the example of FIG. 16, the size of the graphic display area of the Japanese archipelago matches the size of the imaging frame of the imaging unit 2, so that the foreground image and the pointing area icon image are not enlarged or reduced. The coordinates of the pixels of the foreground image and the pointing area icon image may be shifted and overwritten according to the upper left coordinates of the graphic display area.

  As a method of overwriting the foreground image, for example, a method of overwriting the foreground image as it is can be used. Further, for example, a method of overwriting a foreground image processed so that the operation target image can be seen through can be used. Furthermore, for example, a method of overwriting an image in which a pixel region corresponding to a foreground image is represented by a single color or a complementary color of the operation target image can be used. Further, for example, a method of overwriting an image in which the outline of the foreground image (the boundary line between “255” and “0” in the foreground mask) is represented by a single color or a complementary color of the operation target image can be used. Further, for example, a method of overwriting an image representing a contour line of a foreground image, that is, an edge portion extracted as a pixel whose luminance value etc. changes abruptly compared with surrounding pixels, with a single color or a complementary color of the operation target image. It can also be used. Also, for example, two or more of these methods can be used in combination.

As the contour line, for example, one extracted based on the luminance value Y of YCbCr can be used. Further, for example, it is also possible to use a contour obtained by extracting the contour lines of the three primary colors of RGB and combining (merging) the extracted three contour lines into one. In addition, for example, an image obtained by subtracting the outline of the entire background image from the outline of the entire current image is generated (if the subtracted value is a negative number, the absolute value is adopted), and the pixel value of the generated image is a predetermined value. It is also possible to use a binary value (for example, “255” and “0”) depending on whether or not it is above, and mask the image after setting with a foreground mask. In this method, even if the foreground mask cannot be successfully generated due to changes in illumination and sunlight between the background image and the current image, only the outline of the moving object remains when the difference between the outlines is obtained. There is a merit that Of course, if there is no change in illumination or sunlight between the background image and the current image, the same contour line extracted after the above-mentioned foreground image is generated can be obtained by masking with the foreground mask in the subsequent stage.
In the example of FIG. 16, the outline is overwritten as the foreground image.

  In the input device 1 according to the embodiment of the present invention, an example in which the operation target image is directly overwritten with the foreground image or the outline of the foreground image is shown, but other configurations may be employed. For example, as a foreground image, as shown in FIG. 17, a mirror image of the foreground image may be overwritten. In this case, since the user uses a familiar mirror image, the operability of the input device 1 can be improved. Furthermore, for example, by using a mirror image of the background image as the operation target image, it is possible to realize a digital mirror having a function of detecting an instruction operation of the captured user without error. The digital mirror can switch the whole body image and the close-up image of the face without changing the standing position of the user, or the user can sequentially switch the appearance of wearing various clothes. The switching instruction can be accurately performed by the function.

Further, for example, a hand image may be detected from the foreground image, and only the detected hand image may be overwritten as the foreground image. Thereby, the operation target image can be displayed more clearly.
As the instruction area icon image, for example, an image having the same shape as or similar to the shape of the instruction area can be used. Further, for example, when the instruction area is set with one or a small number of coordinates and an image having the same shape as the instruction area is used as the instruction area icon image, the instruction area icon image is difficult to see in the operation image. A circle or polygon including coordinates can also be used.

  In the input device 1 according to the embodiment of the present invention, an example in which the imaging frame of the imaging unit 2 is included in the operation target image as it is is shown, but other configurations may be adopted. For example, the size of the operation target image may be different from the size of the imaging frame of the imaging unit 2. At this time, if the aspect ratio of the operation target image is the same as the aspect ratio of the imaging frame of the imaging unit 2, the foreground image is enlarged or reduced by a predetermined number of times, that is, resized and overwritten on the operation target image. That's fine. Further, when the aspect ratio of the operation target image is different from the aspect ratio of the imaging frame of the imaging unit 2, the foreground image is resized, the drawing position is adjusted, or the like so that the foreground image does not protrude from the operation target image. The operation target image may be overwritten.

  Moreover, although the input device 1 which concerns on embodiment of this invention showed the example which overwrites an instruction | indication area icon image on the operation target image, the other structure can also be employ | adopted. For example, when an instruction area icon is drawn in advance on the operation target image, overwriting of the instruction area icon image may be omitted. In this case, as shown in FIG. 18, the foreground image is overwritten by resizing the foreground image, adjusting the drawing position, or the like so that the designated area overlaps the previously drawn designated area icon image. The shape of the instruction area is set so as to be the same as the shape of the instruction area icon image.

  Subsequently, the process proceeds to step S308, and the operation image display unit 17 outputs video data for displaying the operation image generated in step S307 to the image display unit 3. As a result, the image display unit 3 displays an operation image in which the foreground image and the instruction area icon image are overwritten on one frame of the karaoke movie, as shown in FIG. 16, in accordance with the signal.

  Subsequently, the process proceeds to step S309, and the operation image display unit 17 determines whether or not to end the instruction detection loop process based on the current image acquired in step S301. As a method for determining whether or not to end the instruction detection loop process, for example, it is determined that the instruction detection loop process is ended when a face image of a predetermined size or more is not detected from the current image for a predetermined number of times. If it is determined that the instruction detection loop process is to be terminated (Yes), the calculation process is terminated. On the other hand, if it is determined not to end the instruction detection loop process (No), the process returns to step S301.

  On the other hand, in step S310, the instruction input determination unit 18 determines that there is an instruction input for a predetermined operation, and returns to step S301 after performing the predetermined operation. As the predetermined operation, for example, a pitch lowering operation / pitch lowering operation will be described. As shown in FIG. 16, “▼” is indicated as an instruction area icon image in the instruction area associated with the pitch lowering operation, and “▲” is indicated as the instruction area icon image in the instruction area associated with the pitch raising operation. It has been overwritten. As shown in FIG. 19, when it is determined that the pixel area corresponding to the foreground image and the instruction area overwritten with “▼” are overlapped, an operation for lowering the pitch of the karaoke performance is performed on the assumption that the instruction for the pitch lowering operation has been input. . Also, for example, if it is determined that the pixel area corresponding to the foreground image and the instruction area overwritten with “重 な り” overlap, an operation for raising the pitch of the karaoke performance is performed on the assumption that an instruction for raising the pitch is received.

  Note that the input device 1 according to the embodiment of the present invention is determined to have received an instruction input for a predetermined operation associated with the instruction area when it is determined that the pixel area corresponding to the foreground image and the instruction area overlap. Although an example of executing the operation has been shown, other configurations may be adopted. For example, when a plurality of instruction areas are set and a pixel area corresponding to the foreground image overlaps two or more of the set instruction areas, the pixel area corresponding to the foreground image overlaps A configuration may be adopted in which a predetermined operation is selected according to the combination of the instruction areas, it is determined that there is an instruction input for the selected predetermined operation, and the selected predetermined operation is executed. In this case, for example, the time when the pixel area corresponding to the foreground image overlaps two or more instruction areas may be measured, and the user's movement direction may be determined based on the time difference.

  In addition, for example, using the human characteristics that can repeat the operation of the same pattern, the timing at which the designated area and the pixel area corresponding to the foreground image overlap is detected, and the pattern of the overlapping timing is detected based on the detected timing. And it is good also as a structure which determines with having received the instruction | indication input of predetermined | prescribed operation | movement, when the same pattern is detected twice or more. Thereby, for example, even if it is erroneously determined that the instruction area and the pixel area corresponding to the foreground image overlap due to the influence of noise or the like of the imaging unit 2, erroneous detection of instruction input unintended by the user can be suppressed. In addition to the overlapping timing pattern, a pixel value change pattern (for example, how much brightness has changed to how much darkness) may be added to the determination.

(Operation other)
Next, the operation of the input device 1 according to the embodiment of the present invention will be described.
It is assumed that the user moves outside the imaging range of the imaging unit 2 after performing a start operation on the computer 4. Then, the processor 5 of the computer 4 starts the background image storage process, acquires one frame of the video indicated by the video data output from the imaging unit 2, that is, a candidate image (step S101 in FIG. 4), and acquires it. When a face image of a predetermined size or larger is not detected from the candidate image, it is determined that the background image is updated and stored (step S102 “Yes” in FIG. 4).

  Subsequently, as shown in FIG. 6, the processor 5 stores the candidate image as a background image in the background image storage unit 19 (step S103 in FIG. 4). Thereby, the background image storage unit 19 stores one frame of a video in which the user's face is not reflected in a predetermined size or more as a background image. Then, the above flow is repeated every time a predetermined time elapses, and the background images are stored one after another. The background image may be updated instantaneously, or may be updated after a predetermined time (for example, 1 second) has elapsed.

  At the same time, the processor 5 starts execution permission determination processing, and acquires one frame of the video indicated by the video data output from the imaging unit 2, that is, the current image (step S201 in FIG. 7). Here, it is assumed that the user has sufficiently approached the imaging unit 2 in order to instruct the pitch change. Then, the processor 5 determines that execution of the instruction detection loop process is permitted when a face image having a size equal to or larger than a predetermined size is detected from the acquired current image (step S202 “Yes” in FIG. 7), and instruction detection is performed. A loop process is executed (step S203 in FIG. 7).

  When the instruction detection loop process is executed, the processor 5 acquires one frame of the video indicated by the video data output from the imaging unit 2, that is, the current image, as shown in FIG. 10 (step S301 in FIG. 9). ). Subsequently, as shown in FIG. 11, among the pixels of the acquired current image, “255” is set to a pixel having a different image value from a pixel having the same coordinates of the background image stored in the background image storage unit 19, and not so. A foreground mask is generated by giving “0” to the pixel. Subsequently, the foreground image is generated by masking the current image with the generated foreground mask as shown in FIG. 12, and the foreground information including the generated foreground mask and the foreground image is stored in the foreground information storage unit 20 (FIG. 12). 9 step S302).

  Subsequently, the processor 5 determines that the designated area has not been set (step S303 “No” in FIG. 9), and the pixel area corresponding to the foreground image included in the foreground information among the pixels of the imaging frame of the imaging unit 2 One or more pixels are selected from the pixels not included in the pixel and an instruction area is set at the coordinates (step S304 in FIG. 9). For example, as shown in FIG. 13, the instruction area is set to the coordinates of a pixel on the right side of the position of the face image on the foreground image that is twice the face image width.

  Subsequently, as shown in FIG. 15, the processor 5 acquires one frame of the karaoke movie of the karaoke system that is the operation target, that is, the operation target image (step S306 in FIG. 9), and as shown in FIG. The foreground image included in the foreground information and the instruction area icon image indicating the instruction area are overwritten on the acquired operation target image, and an operation image for performing a pitch change operation is generated (step S307 in FIG. 9). Subsequently, a signal for displaying the generated operation image is output to the image display unit 3. As a result, the image display unit 3 displays an operation image in which the foreground image and the instruction area icon image are overwritten on the operation target image, as shown in FIG. 16, in accordance with the signal from the processor 5.

  Subsequently, the processor 5 detects a face image having a size equal to or larger than a predetermined size from the current image, determines that the instruction detection loop processing is not ended (step S309 in FIG. 9), and repeatedly executes the above flow. Since the instruction area has already been set, it is determined that the instruction area has been set (step S303 “No” in FIG. 9), and the instruction area is not set again.

  It is assumed that the user moves his / her hand or finger while the above flow is repeated, and the pixel area corresponding to the foreground image and the instruction area overlap as shown in FIG. Then, the processor 5 determines that there is a portion that overlaps the designated area and the pixel area corresponding to the foreground image generated after setting the designated area among the pixels of the imaging frame of the imaging unit 2 (step in FIG. 9). S305 “Yes”). Subsequently, a predetermined operation associated with an instruction area having a portion overlapping with the pixel area corresponding to the foreground image is performed (step S310 in FIG. 9). In the example of FIG. 19, the pitch reduction operation is tied to the contour line of the foreground image, that is, the pointing region where the pixel region corresponding to the foreground image overlaps (the region where the pointing region icon image “▼” is overwritten). It is determined that an instruction for a pitch lowering operation has been input, and a pitch lowering operation is performed for karaoke performance.

  As described above, in the input device 1 according to the embodiment of the present invention, the current image acquisition unit 10 that acquires, as a current image, one frame of video output from the imaging unit 2 that continuously captures a predetermined space, Whether the background image storage unit 19 that stores a background image generated from an image captured at the same angle of view as the current image, and the pixel value of the pixel of the current image and the image value of the pixel having the same coordinates of the background image are different A foreground mask generating unit 12 that generates a foreground mask indicating whether or not, a foreground image generating unit 13 that generates a foreground image by masking the current image with a foreground mask, and a foreground image among pixels of an imaging frame of the imaging unit 2. An instruction area setting unit 14 that selects one or more pixels from pixels that are not included in the corresponding pixel area and sets the instruction area at the coordinates, and an instruction area in a predetermined operation target image An operation image for displaying on the image display unit 3 directed to a predetermined space an operation image formed by overwriting the instruction region icon image and the second foreground image that is the foreground image generated after setting the instruction region. A display unit 17; a pixel region overlap determining unit 15 that determines whether or not there is an overlapping portion of the pixel in the imaging frame and the pixel region corresponding to the second foreground image; When it is determined that there is an overlapping portion with the pixel region corresponding to the second foreground image, an instruction input determination unit 18 that determines that there is an instruction input for a predetermined operation is provided.

  In other words, the input device 1 includes an imaging unit 2 that continuously captures a predetermined space and an image display unit 3 that displays an operation image. The operation image is an image in which an instruction area is set. And a predetermined operation target image is overwritten with a current foreground image and an instruction area icon image indicating the instruction area, and the current foreground image is one frame of the current video output by the imaging unit 2. Among the pixels of the current image acquired from the above, an image formed by pixels having pixel values different from those of pixels having the same coordinates as those of the background image previously captured at the same angle of view as the current image. At least a part of the current foreground image that is set in the pixel area that is not included in the pixel area corresponding to the past foreground image generated at the time when the area is set and that is generated after setting the instruction area is indicated Territory Of as it used to determine whether or not to present at least a portion. Therefore, for example, a portion in which a moving body other than the user, such as a pet, is included in the foreground image, and no indication area is set in the pixel area corresponding to the foreground image, so that erroneous detection of an instruction input can be suppressed. The input device 1 can be provided.

  In addition, for example, the position of the user's eyes and the position of the hand on the image are detected, the indication area is set at an intermediate position between the detected eye position and the hand position, and the pixel value of the pixel in the indication area is set. Unlike the prior art that determines that the user's hand on the image has touched the pointing area when the change occurs, detection of the user's eye position and hand position can be omitted, and the amount of computation can be reduced. .

In addition, when the instruction area setting unit 14 determines that the candidate pixel having a predetermined positional relationship with the face image area on the foreground image is a pixel that is not included in the pixel area corresponding to the foreground image, the candidate pixel The indication area was set to the coordinates of. Therefore, the instruction area can be set to more appropriate coordinates, and erroneous detection of the instruction input can be more reliably suppressed.
Further, the candidate pixels are pixels located at coordinates at a predetermined distance in a predetermined direction from a predetermined base point (for example, the center) of the face image area on the foreground image. Therefore, the instruction area can be set to a more appropriate coordinate, and instruction input can be performed relatively easily.

Further, the image processing apparatus includes a background image storage determination unit 7 that determines whether or not to update and store a background image, and the background image storage determination unit 7 updates and stores the background image every time a predetermined time elapses. Judged whether or not. Therefore, the background image can be made more appropriate.
Further, the background image storage determination unit 7 determines whether or not to update and store the background image when the designated area is not set. Therefore, it is possible to more appropriately determine whether there is an overlapping portion between the instruction area and the pixel area corresponding to the foreground image (second foreground image).

  Further, when the background image storage determination unit 7 determines whether or not to update and store the background image, and when the background image storage determination unit 7 determines to update and store the background image, it is one frame of the video. For each pixel of the candidate image, the difference between the pixel value of each pixel and the pixel value of the pixel of the stored background image corresponding to that pixel is calculated and obtained by dividing the calculated difference by a predetermined integer value. A background image writing unit 9 that rounds up the first decimal place of the absolute value of the quotient, returns the sign, calculates an integer value, and adds the calculated integer value to the pixel value of the pixel of the background image; did. Therefore, the background image can be gradually brought closer to the candidate image.

Furthermore, a face image detection unit 8 that detects a face image from one frame of the video is provided, and the background image storage determination unit 7 updates and stores the background image when no face image is detected by the face image detection unit 8. Then I decided to judge. Therefore, a more appropriate background image can be stored.
In addition, a human detection unit 21 that detects a specific heat source from a range within a predetermined distance from the imaging unit 2 is provided, and the background image storage determination unit 7 uses a background image when the specific heat source is not detected by the human detection unit 21. Since it is determined that the image is updated and stored, a more appropriate background image can be stored.

Further, the operation image display unit 17 uses a mirror image of the foreground image as the foreground image to be overwritten on the operation target image. Therefore, the user can easily input instructions, and the operability of the input device 1 can be improved.
In addition, the operation image display unit 17 has only the foreground image overwritten on the operation target image as the outline of the foreground image, that is, the edge portion extracted as a pixel whose luminance value etc. changes abruptly compared with surrounding pixels. The image showing was used. Therefore, the area that covers the operation target image can be reduced, and the operation target image can be displayed more clearly.

Further, the operation image display unit 17 uses a foreground image processed so that the operation target image can be seen through as the foreground image to be overwritten on the operation target image. Therefore, since the operation target image can be seen through the foreground image, the operation target image can be displayed more clearly.
Further, the operation image display unit 17 uses an image in which a portion corresponding to the foreground image is represented by a single color or a complementary color of the operation target image as the foreground image to be overwritten on the operation target image. Therefore, the foreground image becomes conspicuous, and the foreground image can be displayed more clearly.

Further, the operation image display unit 17 displays the outline of the foreground image (the boundary line between “255” and “0” in the foreground mask) as a foreground image to be overwritten on the operation target image by a single color or a complementary color of the operation target image. The method of overwriting the done image was used. Therefore, since the foreground image becomes conspicuous, the foreground image can be displayed more clearly.
In addition, the pixel area overlap determination unit 15 determines a difference (pixel value difference) between a pixel value of a pixel in the designated area of the current current image and a pixel value of a pixel having the same coordinates of the current image acquired when the designated area is set. ) Is determined to be greater than or equal to a predetermined value, it is determined that there is an overlapping portion between the designated area and the pixel area corresponding to the foreground image. Therefore, it can be relatively easily determined whether or not there is an overlapping portion between the designated area and the pixel area corresponding to the foreground image.

  Further, the pixel area overlap determination unit 15 determines that the number of pixels determined that the absolute value of the pixel value difference in the instruction area is equal to or greater than a predetermined value is equal to or greater than a predetermined ratio of the number of all pixels in the instruction area. In some cases, it is determined that there is an overlapping portion between the designated area and the pixel area corresponding to the second foreground image. Therefore, it can be more appropriately determined whether or not there are overlapping portions.

DESCRIPTION OF SYMBOLS 1 ... Input device, 2 ... Imaging part, 3 ... Image display part, 4 ... Computer, 5 ... Processor, 6 ... Memory | storage device, 7 ... Background image memory | storage determination part, 8 ... Face image detection part, 9 ... Background image writing , 10 ... current image acquisition unit, 11 ... execution permission determination unit, 12 ... foreground mask generation unit, 13 ... foreground image generation unit, 14 ... designated region setting unit, 15 ... pixel region overlap determination unit, 16 ... operation target image Acquisition unit, 17 ... operation image display unit, 18 ... instruction input determination unit, 19 ... background image storage unit, 20 ... foreground information storage unit, 21 ... human detection unit, 22 ... personal authentication unit

Claims (15)

A current image acquisition unit that acquires, as a current image, one frame of video output by an imaging unit that continuously images a predetermined space;
A background image storage unit that stores a background image generated in advance from a video imaged at the same angle of view as the current image;
A foreground mask generating unit that generates a foreground mask indicating whether a pixel value of a pixel of the current image is different from an image value of a pixel having the same coordinates of the background image;
A foreground image generation unit that generates a foreground image by masking the current image with the foreground mask;
An instruction area setting unit that selects one or more pixels from pixels that are not included in the pixel area corresponding to the foreground image among the pixels of the imaging frame of the imaging unit, and sets the instruction area at the coordinates;
An operation image formed by overwriting a predetermined operation target image with an instruction area icon image indicating the instruction area and a second foreground image that is a foreground image generated after the setting of the instruction area is set in the predetermined space. An image display section for operation to be displayed on the image display section directed to
A pixel region overlap determination unit that determines whether or not there is an overlapping portion between the indication region and the pixel region corresponding to the second foreground image among the pixels of the imaging frame of the imaging unit;
An input device including an instruction input determination unit that determines that there is an instruction input for a predetermined operation when it is determined that there is an overlapping portion between the instruction area and a pixel area corresponding to the second foreground image.   The candidate area setting unit determines that the candidate pixel having a predetermined positional relationship with the face image area on the foreground image is a pixel not included in the pixel area corresponding to the foreground image. The input device according to claim 1, wherein the indication area is set in a coordinate of a pixel.   The input device according to claim 2, wherein the candidate pixel is a pixel at a predetermined distance in a predetermined direction from a predetermined base point of a face image region on the foreground image. A background image storage determination unit for determining whether to update and store the background image;
The input device according to claim 1, wherein the background image storage determination unit determines whether to update and store the background image every time a predetermined time elapses.   The input device according to claim 4, wherein the background image storage determination unit determines whether to update and store the background image when the instruction area is not set. A face image detection unit for detecting a face image from one frame of the video;
The input device according to claim 4, wherein the background image storage determination unit determines to update and store the background image when a face image of a predetermined size or larger is not detected by the face image detection unit. A human detection unit that detects a specific heat source from a range within a predetermined distance from the imaging unit;
The input device according to claim 4, wherein the background image storage determination unit determines to update and store the background image when a specific heat source is not detected by the human detection unit.   The input device according to claim 1, wherein the operation image display unit uses a mirror image of the second foreground image as a foreground image to be overwritten on the operation target image.   The input device according to claim 1, wherein the operation image display unit uses an outline of the second foreground image as a foreground image to be overwritten on the operation target image.   The said operation image display part uses the said 2nd foreground image processed so that the said operation target image may show through as a foreground image overwritten on the said operation target image. The input device described.   The operation image display unit uses, as a foreground image to be overwritten on the operation target image, an image in which a portion corresponding to the second foreground image is represented by a single color or a complementary color of the operation target image. The input device according to any one of the above.   The operation image display unit uses an image in which an outline of the second foreground image is represented by a single color or a complementary color of the operation target image as a foreground image to be overwritten on the operation target image. The input device according to claim 1.   The pixel area overlap determination unit is configured to obtain an absolute difference between a pixel value of a pixel in the designated area of the current image and a pixel value of a pixel having the same coordinate of the current image acquired when the designated area is set. 13. The device according to claim 1, wherein when it is determined that the value is equal to or greater than a predetermined value, it is determined that there is an overlapping portion between the indication region and a pixel region corresponding to the second foreground image. Input device. An imaging unit for continuously imaging a predetermined space;
An input device including an image display unit for displaying an operation image;
The operation image is an image in which an instruction area is set, and a predetermined operation target image is overwritten with a current foreground image and an instruction area icon image indicating the instruction area,
The current foreground image is a pixel and a pixel having the same coordinates as the background image captured in advance at the same angle of view as the current image among the pixels of the current image acquired from one frame of the video output by the imaging unit. It is an image formed with pixels with different values,
The indication area is set to an area that is not included in a pixel area corresponding to a past foreground image generated at the time when the indication area is set, and the current area generated after setting the indication area An input device used for determining whether or not at least part of the foreground image is present in at least part of the designated area. Acquiring one frame of video output by an imaging unit that continuously images a predetermined space as a current image;
Generating a foreground mask indicating whether a pixel value of a pixel of the current image is different from an image value of a pixel having the same coordinates of a background image captured in advance with the same angle of view as the current image;
Masking the current image with the foreground mask to generate a foreground image;
Selecting one or more pixels from pixels not included in the pixel area corresponding to the foreground image among the pixels of the imaging frame of the imaging unit, and setting an instruction area at the coordinates;
An operation image in which an instruction area icon image indicating the instruction area and a second foreground image, which is a foreground image generated after setting the instruction area, are overwritten on a predetermined operation target image in the predetermined space. Displaying on the directed image display;
Determining whether or not there is an overlapping portion between the indication area and the pixel area corresponding to the second foreground image among the pixels of the imaging frame of the imaging unit;
An input method comprising: determining that there is an instruction input for a predetermined operation when it is determined that there is an overlapping portion between the instruction area and a pixel area corresponding to the second foreground image.

Images