JP2018190144A - Information processing apparatus, control method therefor, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus, a control method therefor, and a program for enabling a user surrounding a desktop surface to perform communication at a good tempo without being affected by operation of a device.SOLUTION: When a touch by a user's hand or an object such as a pen on a capture button 503 of an operation menu 500 included in an image projected by a projector is present, the information processing apparatus specifies the object and determines whether or not the specified object is within a capture area 401. When it is determined not to be within the capture area 401, a picture is taken by a camera to perform an image capture.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an information processing apparatus, a control method therefor, and a program, and more particularly to an information processing apparatus used in a conference system that captures the appearance of a desk surface and generates an image, a control method therefor, and a program.

  Conventionally, as a conference system that displays information on a desk surface (horizontal plane), for example, there are a method of projecting an image from above by a projector and a method of displaying an image on a desktop embedded display.

  In such a conference system, it is possible to hold a conference by displaying electronic materials on the desk surface. In the case of a method of projecting an image from above with a projector, if paper or a white board is installed on the desk top surface, electronic material projected as an image can be written with a pen or a marker.

  In general, there are cases where things such as actual products, prototypes, and sample products are placed on a desk and discussed while referring to them.

  Further, such a conference system is provided with an imaging means, which can capture the state of communication on a desk and record and hold (capture) it as an image. For example, if you create an image by taking pictures of electronic materials and written contents, items placed on a desk, pointing to electronic materials and objects, etc., and recording the images, you can use them as information about the progress of the meeting and reminders. can do.

  On the other hand, the image processing apparatus described in Patent Document 1 captures a form placed on a desk surface from above and generates an image. At this time, it is determined whether or not the generated image includes an object other than a form, which is a subject such as a user's hand. If it is determined that the object is included, a time until an object other than the subject moves from the capture target region is predicted, and imaging is performed again after the time has elapsed.

  This reduces the generation of an image in which the user's hand is reflected.

  For example, in the above-described conference system, when a capture operation instruction is received using a button provided physically or virtually, depending on the positional relationship between the button and the display area, the user's hand operating the button It may appear in the image. Generation of such an image can be reduced by applying the technique of Patent Document 1.

Japanese Patent No. 5629238

  However, when the technology of Patent Document 1 is applied to the above-described conference system, the hand that performs the pointing operation that is originally intended to be included in the image and the hand that holds the object that may be included in the image are not objects. It will be detected and will not be captured.

  In view of such problems, the present invention has an object to provide an information processing apparatus, a control method therefor, and a program that allow a user surrounding a desk surface to perform good-tempo communication without being affected by operation of the device. To do.

  An information processing apparatus according to claim 1 of the present invention includes a display unit that displays an image in a target area on a desk surface, a capture unit that captures the image and generates a captured image, and capture by an object operated by a user. Detecting means for detecting presence / absence of a touch on a button, specifying means for specifying the object when the detecting means detects that the touch is present, and the object specified by the specifying means Determining means for determining whether or not the object is within the target area, and the capture means generates the captured image when the determining means determines that the object is not within the target area. And

  ADVANTAGE OF THE INVENTION According to this invention, the user who surrounds a desk surface can perform communication with good tempo without being influenced by operation of an apparatus.

1 is an external view of a conference system including an information processing apparatus according to Embodiment 1. FIG. It is an external view of the information processing apparatus of FIG. It is a hardware block diagram of the conference system of FIG. FIG. 3 is a schematic diagram illustrating a relationship between regions in Example 1. It is a figure which shows the operation menu contained in the image projected on the projection area | region of FIG. 3 is a flowchart illustrating a control processing procedure of the information processing apparatus according to the first exemplary embodiment. FIG. 7 is a flowchart showing a procedure of capture processing in step S604 of FIG. 6 according to Embodiment 1. FIG. It is a figure which shows the object image and capture image which are produced | generated in Example 1. FIG. FIG. 10 is a schematic diagram illustrating a relationship between regions in Example 2. It is a figure which shows the operation menu contained in the image projected on the projection area | region of FIG. FIG. 10 is a schematic diagram illustrating a method for determining a screen display mode according to the third embodiment. FIG. 7 is a flowchart illustrating a capture processing procedure in step S604 in FIG. 6 according to a third embodiment. FIG. 10 is an external view of a conference system including an information processing apparatus according to a fourth embodiment. 14 is a flowchart illustrating a control processing procedure of the information processing apparatus according to the fourth embodiment. 15 is a flowchart showing a capture process procedure in steps S604 and S1402 of FIG. It is a figure which shows the object image produced | generated in Example 4. FIG.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

(Example 1)
FIG. 1 is an external view of a conference system including an information processing apparatus 101 according to the present embodiment.

  In this conference system, as shown in FIG. 1A, a projector 203 projects an image onto a projection surface on a desk, and as shown in FIG. 1B, a plurality of users surround the desk. Use it as you like.

  The information processing apparatus 101 is installed on a desk and used, and includes a camera 202, a projector 203, and a depth sensor 204. In FIG. 1, the exterior of the information processing apparatus 101 is omitted in order to explain the internal configuration of the information processing apparatus 101.

  The projector 203 is a projection unit that projects an image onto the projection surface 103 on the desk. The camera 202 is an image capturing unit for capturing the projected image. In the present embodiment, the projector 203 projects an image on the desk, but the present invention is not limited to this as long as the user can visually recognize the image on the desk surface. For example, an image may be displayed on a desktop embedded display.

  The depth sensor 204 is a sensor used for extracting and acquiring a region of an object such as a user's hand or pen on the desk surface.

  A touch mat 205 installed on a desk is connected to the information processing apparatus 101. The touch mat 205 is an input interface for accepting a touch operation by the user, and detects coordinates indicating a pressed position. When using a desktop embedded display instead of the projector 203, the display and the touch mat 205 are integrated.

  The touch mat 205 is set to an area including a range in which the projector 203 can project an image (an area indicated by a broken line in FIG. 1A, hereinafter referred to as a projection area 104).

  The image projected by the projector 203 can include electronic materials and GUI (Graphical User Interface) such as messages and buttons. The user operates the information processing apparatus 101 using a hand or a pen. The information processing apparatus 101 detects a selection operation of the GUI projected on the projection surface based on the pointing result of the touch mat 205.

  FIG. 2 is an external view of the information processing apparatus 101 as viewed from a direction different from that in FIG. In FIG. 2, as in FIG. 1, the exterior is not shown in order to explain the internal configuration.

  The information processing apparatus 101 includes a base 251 and a main frame 252 fixed to the base 251. In addition, the projector 203 and the depth sensor 204 are directly fixed to the main frame 252 and the camera 202 is fixed via the camera attachment 253.

  The projector 203 includes a light source unit 258 that is a light source, a projector barrel 259 that is a projection optical system, and a projection mirror 260. A plurality of lenses are arranged inside the projector barrel 259. The light source unit 258 and the projector barrel 259 are connected via a bent portion 259a. The light source unit 258 is disposed on the back side of the bent portion 259a. A reflection mirror 261 is disposed at the bent portion 259 a and reflects light emitted from the light source unit 258 toward the projection mirror 260. The projection mirror 260 is disposed on the upper front side of the projector barrel 259 in FIG. The projection mirror 260 reflects the light projected from the light source unit 258 toward the projection plane 103 and projects an image on the projection area 104. Hereinafter, as the size of the projection region 104 in FIG. 1, the width is represented as “screen_width” and the height is represented as “screen_height”. The size of the projection area 104 in this embodiment is screen_width = 1920 dots and screen_height = 1080 dots, but is not limited thereto. In this embodiment, the projection mirror 260 of the projector 203 is a curved mirror so that the image projected on the projection area 104 has an appropriate aspect ratio (for example, 16: 9).

  Next, the configuration of the camera 202 will be described. The camera 202 includes a charge-coupled device (CCD) sensor (not shown), an imaging mirror 256, and a camera barrel 257 as an imaging device.

  The CCD sensor is mounted so as to be substantially horizontal with respect to the projection plane 103.

  The imaging mirror 256 is a concave curved mirror that is assembled to the main frame 252.

  The camera barrel portion 257 (imaging barrel portion) is an imaging optical system that incorporates a plurality of lenses as an imaging optical system, and is attached to the camera mount 255.

  The camera barrel 257 forms an image of the object on the CCD sensor inside the camera 202 described above.

  When the projection area 104 is read by the camera 202, the image of the object is reflected by the imaging mirror 256 to the camera barrel 257, passes through a plurality of lenses (not shown) provided in the camera barrel 257, and is then charged to the CCD sensor. Read the image with. An image formed on the imaging surface of the CCD sensor is converted into image data by an A / D conversion unit (not shown) inside the camera 202.

  Next, the hardware configuration of the conference system will be described.

  FIG. 3 is a diagram illustrating a hardware configuration of the conference system of FIG.

  The information processing apparatus 101 includes a control unit 301 in addition to the above-described camera 202, projector 203, and depth sensor 204.

  The control unit 301 includes a CPU 302, a RAM 303, a ROM 304, and an HDD 305 connected to the system bus. A network I / F 306, a camera I / F 307, a display controller 308, and a USB controller 309 are provided.

  The CPU 302 is a central processing unit (processor) that controls the operation of the entire control unit 301, and reads a control program stored in the ROM 304 or the HDD 305. In addition, the camera 202, the projector 203, and the depth sensor 204 included in the information processing apparatus 101 and the peripheral devices described later connected to the information processing apparatus 101 are controlled.

  A RAM 303 is a volatile memory, and is used as a temporary storage area for developing operations of the CPU 302 and various control programs stored in the ROM 304 and the HDD 305.

  A ROM 304 is a nonvolatile memory, and stores a boot program for the information processing apparatus 101 and the like.

  The HDD 305 stores a control program for the information processing apparatus 101.

  The CPU 302 executes a boot program stored in the ROM 304 when the power is turned on. The boot program reads a control program stored in the HDD 305 and expands it on the RAM 303. When the CPU 302 executes the boot program, it executes the control program developed on the RAM 303 to control the information processing apparatus 101. Further, the CPU 302 also stores data used for the operation by the control program on the RAM 303 to read / write. The HDD 305 further stores various settings necessary for operation by the control program, generated image data, and electronic material to be projected.

  The control unit 301 is connected to the network 300 via the network I / F 306. The CPU 302 transmits and receives images and information to and from external devices on the network 300. Thereby, for example, electronic material is received from the file server 102 on the network 300, or an image recorded and held as an image (hereinafter referred to as “capture”) is transmitted to the file server 102.

  The control unit 301 is connected to the depth sensor 204 via the camera I / F 307.

  The depth sensor 204 includes an infrared projection unit 315, an infrared camera 316, and a controller 314.

  The infrared projection unit 315 projects a three-dimensional measurement pattern to the object using infrared rays that are invisible to the human eye.

  The infrared camera 316 is a camera that reads a three-dimensional measurement pattern projected on an object.

  The controller 314 controls the infrared projection unit 315 and the infrared camera 316 to acquire 8-bit 1-channel distance image data indicating the distance to the object. This distance image data is transferred from the depth sensor 204 to the RAM 303 of the control unit 301 via the camera I / F 307.

  The display controller 308 controls display of an image on the display (projection area 104) in accordance with an instruction from the CPU 302. Here, the display controller 308 is connected to the projector 203.

  The control unit 301 is connected to the touch mat 205 via the USB controller 309. The CPU 302 can acquire an operation on the touch mat 205 such as a touch with a user's hand or a pen or a long press and coordinates thereof. The USB controller 309 controls an external USB device in accordance with an instruction from the CPU 302. In this embodiment, an external memory 313 such as a USB memory or an SD card is connected to the information processing apparatus 101. The user can project the electronic material stored in the external memory 313 with the projector 203 by connecting the external memory 313 to the information processing apparatus 101. In addition, an image captured by the camera 202 can be recorded in the external memory 313.

  Note that a USB, a wired LAN, a wireless LAN, a line cable, or the like can be appropriately employed for connecting the display controller 308, the USB controller 309, the camera I / F 307, and the network I / F 306 described above.

  Next, the relationship between the projection area 104 of the projector 203, the captured area 401 of the projected image, the imaging area 402 of the camera 202, and the three-dimensional shape measurement area 403 of the depth sensor 204 in this embodiment will be described.

  FIG. 4 is a schematic diagram showing the relationship between these regions in the present embodiment.

  An imaging area 402 that is an area captured by the camera 202 is an area that is wider than the projection area 104 and includes the projection area 104.

  A three-dimensional shape measurement region 403 that is a region in which the depth sensor 204 measures a three-dimensional shape is a region that is wider than the imaging region 402 and includes the imaging region 402.

  Of the images projected by the projector 203, an area captured by the camera 202 and to be captured is referred to as a capture area 401. In this embodiment, the capture area 401 is an area having the same size as the projection area 104.

  Next, the operation menu 500 included in the image projected by the projector 203 on the desk used in the conference system will be described.

  FIG. 5 is a diagram showing an operation menu 500 included in an image projected on the projection area 104.

  When the CPU 302 detects a long press (an operation to keep touching for a predetermined time) from the touch mat 205 to an arbitrary touch position (for example, 501) in the projection area 104, the CPU 302 displays the operation menu 500 at a position centered on the touch position 501. An operation other than a long press, for example, a touch or a double touch (a quick two touch operation) may be detected, and the operation menu 500 may be displayed at a position centered on the touch position. Note that when the touch position is close to the end of the projection area 104, a part of the operation menu 500 may be placed outside the projection area 104. In this case, the CPU 302 displays the operation menu 500 after correcting the center position where the operation menu 500 should be displayed to be shifted from the touch position so that the entire operation menu 500 is displayed. Thereby, the operation menu 500 can be reliably displayed inside the projection area 104. Thus, the operation menu 500 displayed in the projection area 104 includes an open button 502, a capture button 503, and an end button 504.

  When the touch mat 205 detects that the user touches the open button 502, the CPU 502 displays a material selection screen (not shown). Furthermore, when the touch mat 205 detects that the user selects (touches) an electronic material (for example, a material for presentation) 105 to be displayed from the material selection screen, the CPU 502 displays the selected electronic material 105 in the projection area 104. To display. The user can select an electronic material stored in the external memory 313, the file server 102, the HDD 305 in the information processing apparatus 101, or the like on the material selection screen.

  When the touch mat 205 detects that the user has touched the capture button 503, the CPU 502 captures an image of communication on the desk surface by the camera 202 and captures it as an image.

  When the touch mat 205 detects that the user touches the end button 504, the CPU 502 turns off the information processing apparatus 101 and ends the conference system.

  When the touch mat 205 detects that the user touches a portion inside the projection area 104 but outside the area of the operation menu 500, the CPU 502 closes the operation menu 500 (ends display).

  Next, control of the information processing apparatus 101 will be described.

  FIG. 6 is a flowchart illustrating a control processing procedure of the information processing apparatus 101 according to the present exemplary embodiment. Each step shown in the flowchart of FIG. 6 is realized by the CPU 302 reading out the control program stored in the ROM 304 or the HDD 305 to the RAM 303 and executing it. This processing is realized by appropriately cooperating the CPU 302, the display controller 308, the USB controller 309, and the like in the control unit 301 with the camera 202, the projector 203, the depth sensor 204, the touch mat 205, and the like connected to the control unit 301. .

  When the information processing apparatus 101 is powered on, the CPU 302 starts this process.

  In step S601, the CPU 302 determines whether or not an instruction to display the operation menu 500 has been issued by the user. As described above, CPU 302 determines that an instruction to display operation menu 500 has been issued when the user presses and holds an arbitrary touch position in projection area 104, and advances the process to step S602. Otherwise, the process of step S601 is performed again.

  In step S602, the CPU 302 controls the display controller 308 and the projector 203 to display the operation menu 500.

  In step S <b> 603, the CPU 302 determines whether a capture operation instruction has been made by the user, that is, whether the user has touched the capture button 503. If the capture button 503 on the operation menu 500 displayed in step S602 is touched, it is determined that a capture operation instruction has been made, and the process proceeds to step S604. Otherwise, the process proceeds to step S605.

  In step S604, the CPU 302 performs capture processing. The capture process will be described later with reference to FIG.

  In step S605, the CPU 302 determines whether or not an operation instruction to open a file has been made by the user. If the open button 502 on the operation menu 500 displayed in step S602 is touched, it is determined that an operation instruction to open the file has been issued, and the process proceeds to step S606. Otherwise, the process proceeds to step S608.

  In step S606, as described above, the CPU 302 opens the file selected by the user in response to an operation instruction from the user.

  In step S607, the CPU 302 controls the display controller 308 and the projector 203 to close the operation menu 500 and update the display content.

  In step S <b> 608, the CPU 302 determines whether an end operation instruction has been issued by the user. If the end button 504 on the operation menu 500 displayed in step S602 is touched, it is determined that an end operation instruction has been made, and the process ends (the information processing apparatus 101 is turned off). Otherwise, the process proceeds to step S609.

  In step S <b> 609, the CPU 302 determines whether or not the user has issued an instruction to end the display of the operation menu 500. As described above, the CPU 302 determines that an instruction to end the display of the operation menu 500 is given when the user touches a portion inside the projection area 104 but outside the operation menu 500, and advances the process to step S607. Otherwise, the process returns to step S603.

  Next, the capture processing in step S604 in FIG. 6 will be described using the flowchart in FIG.

  In step S <b> 701, the CPU 302 executes an object detection process for detecting an object near the projection plane 103. This process will be described with reference to the flowchart of FIG.

  In FIG. 7B, first, in step S721, the CPU 302 controls the depth sensor 204 to acquire a distance image. Hereinafter, in the present embodiment, a case where the distance image acquired by the depth sensor 204 has a size of 640 dots × 360 dots will be described. The distance image acquired in step S721 is generated by the distance image generation process of FIG. 7C executed by the controller 314 of the depth sensor 204.

  In FIG. 7C, first, in step S741, the controller 314 controls the infrared projection unit 315 to project a three-dimensional shape measurement pattern using infrared rays toward the projection plane 103.

  In step S742, the controller 314 captures the three-dimensional measurement pattern projected in the direction of the projection plane 103 in step S741 with the infrared camera 316, and acquires an infrared camera image including 8-bit 1-channel information.

  In step S743, the controller 314 extracts corresponding points between the three-dimensional measurement pattern projected in step S741 and the infrared camera image acquired in step S742. For example, one point on the infrared camera image is searched from the three-dimensional shape measurement pattern, and these are associated when the same point is detected. Alternatively, a pattern having a predetermined size including the periphery of the pixel of the infrared camera image may be searched from the three-dimensional shape measurement pattern and associated with the portion having the highest similarity. In order to facilitate this association, the three-dimensional shape measurement pattern is configured such that an arbitrary part thereof is a pattern different from any other part. Therefore, it is possible to specify from a part of the pattern by pattern matching which part of the whole is similar to which part. When a three-dimensional shape measurement pattern is projected onto a three-dimensional shape such as a hand or a finger, the pattern obtained by photographing the pattern may be greatly distorted. In this case, it may be difficult to associate with the original pattern. Therefore, it is desirable that the three-dimensional shape measurement pattern be a high-density pattern so that the position in the entire pattern can be specified from a part of the pattern projected in a very narrow range that can be approximated to a plane.

  In step S744, the controller 314 calculates the distance from the infrared camera 316 to each corresponding point extracted in step S743 by performing calculation using the principle of triangulation using the straight line connecting the infrared projection unit 315 and the infrared camera 316 as a base line. Is calculated. Of the pixels of the infrared camera image, for the pixels extracted as corresponding points in step S743, the distance from the infrared camera 316 is calculated and stored as a pixel value. On the other hand, for a pixel that could not be extracted as a corresponding point, an invalid value (for example, −1) is stored as a portion where the distance could not be measured. By performing this operation for all pixels of the infrared camera image, a distance image composed of 8-bit 1-channel information in which each pixel has a distance value is generated. In order to calculate the distance from the infrared camera 316 to each corresponding point, it is necessary to know the positional relationship between the infrared projection unit 315 and the infrared camera 316 and respective internal parameters. These are calculated by a prior calibration process. Note that the processing in step S744 is executed by an ASIC or DSP (Digital Signal Processor) (not shown) provided in the controller 314.

  In this embodiment, the depth sensor 204 employs a method of detecting a distance to an object by projecting a three-dimensional shape measurement pattern using infrared rays, but other methods can also be used. For example, a stereo method in which stereo vision is obtained with two RGB cameras, or a time until infrared rays are reflected by a subject from a phase difference between infrared projection light and reflected light, and TOF (Time of Flight) can also be used.

  Returning to the description of FIG. 7B, in step S722, the CPU 302 generates an object image from the distance image acquired in step S721 and the reference image acquired in advance. Specifically, in the object image, a pixel in which the difference between the image value of the distance image and the reference image (distance from the infrared camera 316) is a predetermined value or more is black (pixel value 255), and a pixel that is less than the predetermined value is white. It is generated by binarizing to (pixel value 0). Note that the reference image is a distance image that serves as a reference acquired in a state where an object such as a user's hand or pen does not exist by a prior calibration process. That is, in the generated object image, a portion where an object such as a user's hand or pen is present is expressed as a black pixel.

  In step S723, the CPU 302 performs image processing for noise removal on the object image obtained in step S722. For example, if there is a pixel of white (pixel value 0) even around one pixel around the pixel of interest, the CPU 302 performs contraction processing to set the value of the pixel of interest to 0, or black (pixel value 255) around the pixel of interest. ), The noise is reduced by an expansion process for setting the value of the target pixel to 255. FIG. 8A shows an example of an object image obtained by the above steps. The object image 800 includes two objects as an area formed from a continuous black (pixel value 255) pixel group. That is, the object 801 is a user's hand that touches the capture button 503 of the operation menu 500 projected onto the projection area 104. The object 802 is a user's hand pointing at the electronic material 105 projected onto the projection area 104.

  Returning to the description of FIG. 7A, in step S702, the CPU 302 performs projective transformation of the image of the region corresponding to the projection region 104 in the object image obtained in step S701 into the projection coordinate system. FIG. 8B is a diagram for explaining the projective transformation in step S702. A broken line 821 indicates an area corresponding to the projection area 104 in the object image 800. The object image 820 is an image obtained by projective transformation of the broken line 821. Projective transformation is performed so that the coordinates of the object image 800 and the object image 820 correspond to the following. The coordinates (L821, M821) of the object image 800 become the coordinates (0, 0) of the object image 820. The coordinates (L822, M822) of the object image 800 become the coordinates (0, screen_height-1) of the object image 820. The coordinates (L823, M823) of the object image 800 become the coordinates (screen_width-1, screen_height-1) of the object image 820. The coordinates (L824, M824) of the object image 800 become the coordinates (screen_width-1, 0) of the object image 820. Note that the parameters necessary for this projective transformation are acquired by prior calibration.

  In step S703, the CPU 302 identifies the object for which the capture operation instruction has been issued. Here, a broken line 822 in FIG. 8B indicates the area of the operation menu 500 in the projection area 104, and a broken line 823 indicates the area of the capture button 503. The object image 820 acquired in step S702 and the image projected on the projection area 104 have the same coordinate system. Therefore, the CPU 302 identifies an object (black pixel set, object 801 in this embodiment) that overlaps the area corresponding to the capture button 503 in the object image 820 as the object that has undergone the capture operation.

  It should be noted that the method for identifying the object for which the capture operation instruction has been issued in step S703 is not limited to this. For example, the CPU 302 may specify the coordinates of the end of each object included in the object image 820 and determine whether or not the coordinates are included in the area of the broken line 823. In this case, the coordinates of the end of the object are specified as follows. Of the four sides of the circumscribed rectangle of the object, the side that is the end side of the object image and that is inscribed most by the non-transparent pixels is determined. Then, the non-transparent pixel inscribed in the side opposite to the side is calculated as the end portion of the object (if there are a plurality of the non-transparent pixels, for example, the center pixel is selected). As shown in FIG. 8C, in the case of the object 801, the coordinates of the end are (U1, V1).

  Note that the object image 800 acquired by the object detection process in step S701 and the identification result of the object for which the capture operation instruction has been issued in step S703 are temporarily stored in the RAM 303. Here, the identification result of the object for which the capture operation instruction has been issued is, for example, information that the object for which the capture operation instruction has been performed is the leftmost object among the objects included in the object image 800 in this embodiment. Point to.

  In step S704, the CPU 302 determines whether or not the object for which the capture operation instruction specified in step S703 is in the capture area 401 (in the target area). The CPU 302 determines whether an object for which a capture operation instruction has been issued is included in an area corresponding to the capture area 401 in the object image 800. If it is determined that it is included, the process proceeds to step S705. If it is determined that it is not included, the process proceeds to step S707. Note that an area corresponding to the capture area 401 in the object image 800 is calculated by a prior calibration process.

  In step S705, the CPU 302 performs object detection processing again to acquire an object image 800a (not shown). This step is the same as step S701.

  In step S706, the CPU 302 tracks and specifies the object for which the capture operation instruction has been given in the object image 800a acquired in step S705. A certain time has elapsed since the object image 800 is acquired in step S701 and the object for which the capture operation instruction is given in the object image 800 is specified in step S703. Therefore, in the object image 800a acquired in step S705, it is assumed that the object that has issued the capture operation instruction is moving. Therefore, it is necessary to track and specify the object that has moved after the capture operation instruction is given in the object image 800a acquired in step S705. The CPU 302 reads the object image 800 temporarily stored in the RAM 303 and the identification result of the object for which the capture operation instruction has been issued. Then, the degree of overlap between each of the objects in the object image 800a acquired in step S705 and an object for which a capture operation instruction has been issued before being specified by the specifying result included in the object image 800 is evaluated. The object with the highest overlap and specificity is identified as the object that has been previously instructed to perform the capture operation and is tracked in the object image 800a acquired in step S705. In addition, the object image 800 temporarily stored in the RAM 303 is updated with the object image 800a, and the result of specifying the object for which the capture operation instruction has been issued is updated with the result of specifying the tracked object. Then, the process returns to step S704, where it is determined whether or not the object that has been subjected to the capture operation instruction tracked and specified in step S706 is within the capture area 401.

  On the other hand, as a result of the determination in step S704, if it is determined that the object for which the capture operation instruction has been issued is not in the capture area 401, the process proceeds to step S707. In step S707, the CPU 302 controls the display controller 308 and the projector 203 to temporarily hide the operation menu 500. Thereby, it is possible to prevent the operation menu 500 from being included in the content of the captured image 860 obtained as a result of photographing in the next step S708.

  In step S <b> 708, the CPU 302 controls the camera 202 to photograph the desk surface and obtain a photographed image.

  In step S <b> 709, the CPU 302 generates a captured image 860 by performing projective transformation on an image of an area corresponding to the capture area 401 in the captured image acquired in step S <b> 708. As the size of the captured image 860, the width is “image_width” and the height is “image_height”. In this embodiment, the size of the captured image 860 is image_width = 1920 dots and image_height = 1080 dots, but is not limited thereto. FIG. 8D illustrates the projective transformation in step S709. A broken line 841 indicates an area corresponding to the capture area 401 in the captured image 840. The captured image 860 is an image obtained by projective transformation of the captured image 840. Projective transformation is performed so that the coordinates of the captured image 840 and the captured image 860 correspond to the following. The coordinates (L841, M841) of the captured image 840 become the coordinates (0, 0) of the captured image 860. The coordinates (L842, M842) of the captured image 840 become the coordinates (0, image_height-1) of the captured image 860. The coordinates (L843, M843) of the captured image 840 become the coordinates (image_width−1, image_height−1) of the captured image 860. The coordinates (L844, M844) of the captured image 840 become the coordinates (image_width−1, 0) of the captured image 860. Note that the parameters necessary for the projective transformation in step 709 have been acquired by prior calibration. The generated captured image 860 is recorded in the HDD 305 or the external memory 313. Alternatively, it is transmitted to the file server 102 via the network I / F 306.

  In step S710, the CPU 302 controls the display controller 308 and the projector 203 to display the operation menu 500 again.

  It should be noted that the object detection processing in step S701 and step S705 is assumed to be executed a plurality of times until the object for which the capture operation instruction is issued moves outside the capture area 401, but the processing interval is determined by human action. The time interval is sufficiently short. For example, the processing interval is equivalent to 24 FPS (Frame Per Second).

  In this embodiment, an example in which a part of the shooting area 402 is the capture area 401 has been described. However, the shooting area 402 and the capture area 401 may be matched. That is, the entire captured image may be a captured image.

  In this embodiment, when it is determined in step S704 that the object for which the capture operation instruction has been issued is no longer included in the capture area, the image is captured in step S708. However, the present invention is not limited to this. Since the capture area 401 is included in the imaging area 402, for example, when it is determined in step S704 that the object for which the capture operation instruction has been issued is not included in the imaging area 402, the imaging may be performed in step S708.

  As described above, according to the present embodiment, it is possible to reduce an object such as a user's hand that has touched the capture button 503 displayed in the capture area 401 from appearing in the captured image. . On the other hand, a user's hand that points at the electronic material 105 or grips a sample product placed on the desk can be included in the captured image.

(Example 2)
In the present embodiment, a case where the screen of the conference system has a configuration different from that of the first embodiment will be described. Only differences from the first embodiment will be described.

  First, the conference system in the present embodiment will be described. In the first embodiment, the touch mat 205 is connected to the information processing apparatus 101. However, in this embodiment, the touch mat 205 is not connected. The touch operation by the user is received by the following method.

  In the first embodiment, the method in which the CPU 302 obtains the distance image generated by the depth sensor 204 by the distance image generation process of FIG. 7C and generates the object image 800 shown in FIG. In addition, the method for projective transformation of the object image 800 to the projection coordinate system has been described with reference to FIG. Furthermore, as described with reference to FIG. 8C, the method for acquiring the end points of the object 801 included in the object image 800 has been described. In this embodiment, the CPU 302 performs projective transformation on the distance image acquired from the depth sensor 204 into the projection coordinate system by the same method as described with reference to FIG. Then, the value of the corresponding pixel in the distance image is referred to for the end point of the object acquired by the method described with reference to FIG. It is determined whether the pixel value (distance value) is equal to or less than a predetermined threshold value, that is, whether the user's fingertip or the penpoint used by the user is near the desk. If the result of this determination is equal to or less than the threshold value and the end point of the object overlaps the broken line 823 indicating the area of the capture button 503 in FIG. 8B, a touch is made on the coordinate position of the pixel. Judge as having been done.

  The information processing apparatus 101 can detect an operation on the GUI projected on the projection plane based on the touch determination result obtained by the above-described method.

  Next, the relationship between the projection area 104, the capture area 901, the imaging area 402 of the camera 202, and the three-dimensional shape measurement area 403 of the depth sensor 204 in this embodiment will be described. FIG. 9 is a schematic diagram showing the relationship between these regions in the present embodiment.

  Unlike the first embodiment, the capture area 901 is a partial area of the projection area 104 as shown in FIG.

  Next, the operation menu 1000 included in the image projected by the projector 203 on the desk used in the conference system in the present embodiment will be described.

  FIG. 10 is a diagram showing an operation menu 1000 included in an image projected on the projection area 104.

  The operation menu 1000 is always displayed at the top of the capture area 901 as shown in FIG. The operation menu 1000 includes an open button 1001, a capture button 1002, and an end button 1003.

  When the open button 1001, the capture button 1002, and the end button 1003 are touched, the CPU 502 executes the same processing as when the open button 502, the capture button 503, and the end button 504 in FIG. Is done.

  The control of the information processing apparatus 101 in this embodiment is the same as the processing described in the first embodiment with reference to FIGS. 6 and 7. In step S603, the CPU 302 determines whether or not a capture operation instruction has been issued by the user. The touch determination on the capture button 503 is performed using the distance image acquired using the depth sensor 204 as described above. Do. In step S721, the distance image used in the touch determination in step S603 may be referred to. In the first embodiment, the operation menu 500 is not displayed in step S707. However, in this embodiment, since the operation menu 1000 is always displayed, the process proceeds to step S708 directly from step S704 without performing the process in step S707. In the first embodiment, instead of the operation menu 500, the operation menu 1000 may be displayed in the process of step S602. In this case, similarly in the first embodiment, the process of step S707 is not performed, and the process proceeds directly from step S704 to step S708.

  As described above, even when a capture area exists between the capture operation instruction means and the user, an object such as a user's hand that has made an operation instruction to the capture operation instruction means is captured in the captured image. Reflection can be reduced. On the other hand, a user's hand that points or grips an object can be included in the captured image.

Example 3
In the present embodiment, display control of the screen of the conference system from when the touch to the capture button 503 is detected until the shooting region is actually shot by the camera 202 and captured will be described. Only differences from the first embodiment will be described.

  An image projected by the projector 203 on a desk used in the conference system is as shown in FIG. In addition, in this embodiment, when a touch on the capture button 503 is detected and it is determined that a capture operation instruction has been made, the screen is displayed according to the position of an object such as a user's hand or pen that has performed the capture operation instruction. The display mode of is changed. In this embodiment, the display transparency of the operation menu 500 is determined so that the object for which the capture operation instruction has been issued is gradually displayed as the object moves away from the capture button 503. This suggests that the capture is executed when an object such as a user's hand or pen that has issued the capture operation instruction moves out of the capture area 401. An example of a method for determining the display transparency will be described.

  FIG. 11 is a schematic diagram illustrating a method for determining the display mode of the screen according to the present embodiment. Here, after the user touches the capture button 503, the hand is moved out of the capture area 401.

  The end point coordinates of the object for which the capture operation instruction is given in the projection coordinate system can be obtained by the method described with reference to FIG.

  In the present embodiment, the user moves his / her hand along the broken line 1101. Here, a broken line 1101 is a line segment that starts from the center point 1102 of the area of the capture button 503, passes through the end point 1103 of the object that has instructed the capture operation, and extends to a point 1104 that intersects the end side of the capture area 401. Here, if the distance between the center point 1102 and the end point 1103 is D1, and the distance between the point 1103 and the point 1104 is D2, the display transparency of the operation menu 500 is obtained by the following calculation formula.

Display transparency A = D1 / (D1 + D2)
As a result, the operation menu 500 is displayed more opaque as the end point of the object for which the capture operation instruction has been issued is closer to the center point of the area of the capture button 503, and is displayed more transparent as it is further away.

  It should be noted that the display transparency may be calculated by another method that suggests that the capture is executed as the object that has issued the capture operation instruction moves out of the capture area 401. For example, display transparency A1, A2, and A3 are defined (A1 <A2 <A3). When the capture button 503 is touched, the display transparency is set to A1. Then, when the end point of the object for which the capture operation instruction has been issued reaches outside the area of the capture button 503, the display transparency is set to A2. Further, when the end point of the object for which the capture operation instruction has been issued reaches the area outside the operation menu 500, the display transparency is set to A3.

  Next, control of the information processing apparatus 101 in the present embodiment will be described. Only differences from the first embodiment will be described.

  FIG. 12 is a flowchart illustrating the procedure of the capture process in step S604 of FIG. 6 according to the present embodiment.

  In FIG. 12, step S701 to step S710 are the same as the step with the same sign in FIG.

  In step S1201, the CPU 302 calculates the display transparency of the operation menu 500. The display transparency can be calculated by applying the above-described method to the object in the object image 800a acquired in step S705 and for which the capture operation instruction is tracked and specified in step S706. it can.

  In step S1202, the CPU 302 controls the display controller 308 and the projector 203 to update the display of the operation menu 500 so that the display transparency calculated in step S1201 is obtained.

  In the present embodiment, the operation menu 500 according to the first embodiment has been described as an example, but the present invention can also be applied to the operation menu 1000 according to the second embodiment. In this case, in step S1202, the CPU 302 updates the display of the operation menu 1000.

  In this embodiment, the transparency of the operation menu 500 including the capture button 503 is determined and the screen display of the conference system is updated. However, the present invention may be applied only to the capture button 503.

  As described above, according to the present invention, when an object for which a capture operation instruction has been performed moves outside the area to be captured and is shot, shooting is performed as the object moves. A screen can be displayed to suggest.

(Example 4)
In the present embodiment, a case where the capture operation instruction unit is physically provided as hardware of the information processing apparatus 101 will be described. Only differences from the first embodiment will be described.

  FIG. 13 is an external view of a conference system including the information processing apparatus 101 according to the present embodiment. The information processing apparatus 101 in FIG. 13 has the same configuration as that in FIG.

  A capture button 1301 is a hardware button of the information processing apparatus 101 arranged outside the capture area 401 (outside the target area). When pressed, the capture button 1301 can capture the contents on the desk surface and save it as an image. . In this embodiment, a capture operation instruction can be issued from both the capture button 1301 and the capture button 503.

  Next, control of the information processing apparatus 101 in the present embodiment will be described.

  FIG. 14 is a flowchart illustrating a control processing procedure of the information processing apparatus 101 according to the present exemplary embodiment. Steps S601 to S609 are the same as steps with the same reference numerals in FIG.

  In step S1401, the CPU 302 determines whether the capture button 1301 has been pressed. If the user has pressed capture button 1301, the process advances to step S1402.

  In step S1402, the CPU 302 performs capture processing. This step is the same as step S604.

  Next, the capture processing in steps S604 and S1402 in FIG. 14 will be described using the flowchart in FIG. Steps S701 to S710 are the same as the steps denoted by the same reference numerals in FIG.

  In step S <b> 1501, the CPU 302 determines whether this process has been executed by pressing the capture button 1301. If the user has pressed capture button 1301, the process advances to step S1502. Otherwise (that is, if the user has touched the capture button 503), the process proceeds to step S703.

  In step S1502, the CPU 302 depresses the capture button 1301, and estimates and specifies the object for which the capture operation instruction has been issued. FIG. 16 shows an example of a distance image 1600 acquired by the depth sensor 204 when the user presses the capture button 1301 and an object image 1620 obtained by projective transformation of a broken line portion of the distance image 1600. An object 1601 is a user's hand pressing the capture button 1301. An object 1602 is a user's hand pointing at the electronic material projected on the projection area 104. The information processing apparatus 101 can measure the three-dimensional shape of the desk top surface with the depth sensor 204, but the vicinity of the capture button 1301 provided in the apparatus does not fall within the angle of view and cannot be measured. For this reason, it is necessary to estimate an object for which the capture button 1301 is pressed. Here, among the objects in the object image 1620, an object that intersects the upper end side of the object image 1620 (the side on the information processing apparatus 101 side) and one of the other sides other than the upper end side is captured by the capture button. 1301 is estimated as the pressed object.

  As described above, the present invention can also be applied to the case where the capture operation instruction means is physically provided as hardware of the information processing apparatus.

  The object of the present invention can also be achieved by executing the following processing. That is, a storage medium that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus is stored in the storage medium. This is the process of reading the code. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code and the storage medium storing the program code constitute the present invention.

DESCRIPTION OF SYMBOLS 101 Information processing apparatus 202 Camera 203 Projector 204 Depth sensor 205 Touch mat 301 Control part 401 Capture area 402 Shooting area 403 Three-dimensional shape measurement area 500 Operation menu

Claims (13)

Display means for displaying an image in a target area on the desk surface;
Capture means for capturing the image and generating a capture image;
Detecting means for detecting whether or not the capture button is touched by an object operated by a user;
A specifying means for specifying the object when the detection means detects that the touch is present;
Determining means for determining whether the object specified by the specifying means is within the target area;
The information processing apparatus according to claim 1, wherein the capture unit generates the capture image when the determination unit determines that the object is not in the target area. The capture button is included in the displayed image,
When the determination unit determines that the object is not in the target area, the display unit further includes a display control unit that hides the capture button in the image,
The information processing apparatus according to claim 1, wherein the capture unit captures the image after the capture button is hidden. The capture button is included in the displayed image,
The information processing apparatus according to claim 1, wherein an area in which an image to be the captured image is displayed exists in the target area between the capture button and the user.   4. The apparatus according to claim 1, wherein the specifying unit includes a first generation unit that generates a first object image for the target region, and specifies the object based on the first object image. 5. The information processing apparatus according to any one of claims.   The information processing apparatus according to claim 4, further comprising: a tracking unit that tracks the object when the determination unit determines that the object is within the target region. A holding means for holding the first object image and the specifying result by the specifying means;
The tracking unit includes a second generation unit that generates a second object image for the target region, and is included in the first object image among objects included in the generated second object image. The information processing apparatus according to claim 5, wherein an object that overlaps with the object specified by the specifying result is specified as the tracked object.   Between the detection of the touch by the detection means and the generation of the captured image by the capture means, the position was tracked by the area where the detection means was provided and the tracking means. The information processing apparatus according to claim 5, further comprising a determining unit that determines a display mode of the capture button according to a distance between the objects.   The information processing apparatus according to claim 7, wherein the display mode is display transparency.   The information processing apparatus according to claim 1, wherein the detection unit is a touch mat provided in the target area.   The detection means includes third generation means for generating a third object image for the target region, and a distance value of an end point of the object included in the third object image is equal to or less than a threshold value, The information processing apparatus according to claim 1, wherein when the position overlaps the capture button, the touch is detected.   The information processing apparatus according to claim 1, wherein the detection unit is a button physically provided outside the target area. A display step for displaying an image in a target area on the desk surface;
A capture step of capturing the image and generating a capture image;
A detection step of detecting whether or not the capture button is touched by an object operated by a user;
A specifying step of specifying the object when it is detected in the detecting step that the touch is present;
A determination step of determining whether or not the object specified in the specifying step is within the target region,
The control method characterized in that the capture step generates the capture image when it is determined in the determination step that the object is not within the target region.   A program for executing the control method according to claim 12.