JP6555958B2 - Information processing apparatus, control method therefor, program, and storage medium - Google Patents

Description

  The present invention relates to a technique for recognizing an operator's operation based on a captured image.

  A region in which a predetermined object such as a user's hand is captured is extracted from an image obtained by a visible light camera, an infrared camera, a distance image sensor, or the like, and a UI (user interface) operation by a space gesture is performed according to the movement and position. Recognition technology is spreading. In particular, a gesture recognition technique based on an image instead of a touch panel has begun to be used for a table top interface for projecting an image or UI on an operation surface such as a table and touching and operating the image or UI.

  In Patent Document 1, a gesture operation is recognized by tracking the shape of a hand detected based on a difference between two frames of images acquired at predetermined time intervals from an image of a space in which a hand is present. Is disclosed. For such gesture recognition technology, calibration that defines the correspondence between the three-dimensional position coordinates of the real space and the position information in the image obtained by imaging the space is important.

  In Patent Document 2, when an optimum camera parameter is determined based on the correspondence between world coordinates defined in real space and image coordinates defined in an image captured by the camera, an error in correspondence is detected. And presenting to the user.

JP2013-257686A JP 2006-67272 A

  Actually, a conversion error occurs in the coordinate conversion between the coordinates (world coordinates) indicating the three-dimensional position in the real space and the coordinates defined in the captured image depending on the installation environment of the hardware. When the operation surface such as a table is smaller than the image capturing range of the image capturing unit or the like, when the boundary portion of the operation surface appears in the image, the operation surface boundary may be shifted due to a coordinate conversion error. As a result, there is a possibility that the operation tool cannot be stably detected in the vicinity of the boundary, or the touch of the operation surface by the operation tool is erroneously detected.

  The present invention has been made in consideration of the above-described problems, and an object of the present invention is to improve the accuracy of processing for recognizing an object in a space including a plurality of regions having different background conditions using a captured image.

In order to achieve the above object, an information processing apparatus according to the present invention includes an image acquisition unit that acquires an image of a space in which a first coordinate system is defined, and position information in the first coordinate system. Conversion means for converting into position information in the second coordinate system defined in the image acquired by the method, and at least a predetermined operation surface included in the space represented by using the position information in the first coordinate system An operation region acquisition unit that acquires information on an operation region that is set in part, and position information in the second coordinate system obtained as a result of the conversion unit converting the information acquired by the operation region acquisition unit Te, wherein searching the pixels included within the obtained image by the image obtaining means to the pixel group around a boundary of the operating area, the operating surface is captured as a subject to the each pixel Based on dolphins, and the operation region determining means for determining the position information representing the operation area in said second coordinate system,
Is provided.

According to the present invention, the accuracy of processing for recognizing an object in a space including a plurality of regions having different background conditions using a captured image is improved.

1 is a diagram illustrating an example of a system that uses an information processing apparatus 100. FIG. 2 is a block diagram illustrating an example of a hardware configuration and a functional configuration of an information processing device. FIG. The flowchart showing an example of the flow of an operation area | region determination process. The figure showing an example of the operation area data which information processor 100 acquires. The figure showing the mode of a correction process in case an operation surface and an operation area correspond. The figure showing the mode of the correction | amendment process when some distance values of an operation surface are not detected. The figure showing the mode of a correction process when an operation surface and an operation area | region do not correspond. The flowchart showing an example of the flow of an operation detection process. The figure showing an example of the movement area | region detected after correction | amendment. The figure showing an example of the operation area | region data acquired when the height of an operation surface changes with positions.

<First Embodiment>
First, as a first embodiment, an example of processing for recognizing a touch operation performed by an operator on an item projected on a table surface of a table top interface system will be described.

  FIG. 1A is an example of the appearance of a tabletop interface system in which the information processing apparatus 100 according to the present embodiment is installed. The operation surface 101 is a table portion of the table top interface, and the operator can input a touch operation by touching the operation surface 101. In the present embodiment, the distance image sensor 102 is installed above the operation surface 101 so as to look down at the operation surface. In the distance image, information corresponding to the distance D from the reference position (for example, the center of the lens) of the imaging unit that captures the distance image to the object surface captured by the pixel is reflected in the value of each pixel. It is an image. Since the size of each pixel value corresponds to the depth viewed from the imaging means, the distance image sensor may be referred to as a depth sensor (depth sensor). In the present embodiment, the pixel value of the distance image captured by the distance image sensor 102 reflects the magnitude of the distance D from the distance image sensor 102 to the operation surface 101 or the object surface existing above it. The captured distance image is input to the information processing apparatus 100 as an input image. The information processing apparatus 100 acquires the three-dimensional position of the operator's hand 105 by analyzing the input image, and recognizes the input operation. Accordingly, the operator moves a predetermined object such as a hand within a range that can be imaged by the distance image sensor 102 in an operation space (a space that includes the operation surface 101 and its periphery and can be imaged by the distance image sensor 102). It is possible to input a space gesture operation. In the present embodiment, a sensor (Time-of-Flight system) that acquires distance information using a reflection pattern (or reflection time) of infrared light is used. However, for example, an input image can be obtained by installing a stereo camera system or an infrared light emitting element and an infrared camera.

  In the present embodiment, the visible light camera 103 is installed so as to look down on the operation surface 101 from above. The information processing apparatus 100 can function as a document camera that controls the visible light camera 103 to capture an image of an object placed on the operation surface 101 and obtain a read image thereof. The information processing apparatus 100 detects and further identifies an object existing in the operation space based on a visible light image obtained by the visible light camera 103 and a distance image obtained by the distance image sensor 102. The object includes, for example, an operator's hand, a paper medium, a document such as a book, and other three-dimensional objects. However, in the case of the system illustrated in FIG. 1A, the angle of view of the distance image sensor 102 and the visible light camera 103 does not include the operator's head around the table. For this reason, in an input image captured in a state where a person inserts his / her hand into the operation space, the end of the image intersects some part of the user's arm (portion beyond the shoulder).

  The projector 104 projects an image on the upper surface of the operation surface 101. As described above, in this embodiment, the distance image acquired by the distance image sensor 102 is used for detecting the hand 105 and recognizing the operation. By using the distance image, there is an advantage that even if the color of the operator's hand changes due to the projection light of the projector 104, the distance image is hardly affected. The display device of the present system can be configured by using the operation surface 101 as a liquid crystal display instead of the projector 104. In that case, even if a method of detecting a human hand from an image by detecting a skin color region from a visible light image, the hand can be detected without being affected by the projection light. When detecting a moving area from a visible light image, the visible light camera 103 may also be used as an input image acquisition unit.

  Note that the distance image sensor 102 and the visible light camera 103 themselves are not necessarily installed above as long as an image obtained by viewing the operation surface 101 from above is obtained. For example, you may comprise so that reflected light may be imaged using a mirror. Similarly, in the example of FIG. 1A, the projector 104 also projects onto the operation surface 101 so as to look down from diagonally above. However, the projection light projected in a direction different from that shown in the figure may be guided to the operation surface 101 using a mirror or the like. Even when this embodiment is applied to a distance image sensor or a projector used in a system in which the operation surface 101 is installed along the vertical direction, an optical system including a mirror can be similarly used.

  In this embodiment, the X, Y, and Z axes shown in FIG. 1 are defined in the operation space (three-dimensional space on the operation surface 101), and position information is handled. In the example of FIG. 1A, the point 106 is set as the origin of the coordinate axes. Here, as an example, the two-dimensional parallel to the upper surface of the table is the XY plane, and the direction perpendicular to the table upper surface and extending upward is the positive direction of the Z axis. In the present embodiment, the Z-axis direction corresponds to the height direction in the world coordinate system. However, this embodiment can also be applied to a system in which a non-horizontal surface such as a whiteboard or a wall surface is used as the operation surface 101. In this case, the value in the Z-axis direction does not necessarily represent the height, but at least corresponds to the coordinate axis in the direction intersecting the operation surface.

  FIG. 1B illustrates a distance image captured by the distance image sensor 102 and a coordinate system defined in the image. An image 108 represents an example of the content of a distance image captured by the distance image sensor 102. A hatched area 109 is an area in which an image of the operator's hand 105 shown in the distance image 108 is captured. Hereinafter, it is simply referred to as a hand region 109. A point 110 is a pixel detected as a fingertip position from the hand region 109, and is a coordinate of a position estimated that the operator is pointing with the finger.

  In this system, infrared light irradiated from above is imaged by reflected light reflected by the subject surface, and the intensity of the reflected light is the magnitude of the distance D from the light source of the distance image sensor 102 to the subject surface. Is assumed to represent. In each pixel of the distance image, a value corresponding to the distance from the distance image sensor 102 to the subject is held as a pixel value d. The pixel value d reflects the magnitude of the distance D shown in FIG. Accordingly, a pixel that is originally larger in pixel value d (reflected light is stronger and brighter) is closer to the distance image sensor 102, and a pixel that is smaller in pixel value d (reflected light is weaker and darker) is farther from the distance image sensor 102. Indicates that the subject is present. However, in FIG. 1B, it is simplified and only the edge of the subject is clearly shown. In the distance image acquired in the present embodiment, a coordinate system defined by the x-axis and y-axis shown in FIG. 1B and the pixel value d is defined. For example, the position coordinates in the input image of the fingertip position 110 are represented by (x, y, d). As an example, the resolution of the input image is 640 [dot] × 450 [dot].

  In the present embodiment, the coordinates (X, Y, Z) in the operation space shown in FIG. 1A are coordinate-converted into the in-image coordinates (x, y) in the input image shown in FIG. Therefore, the position of the subject in the input image can be handled as information equivalent to the position information in the real space. Here, in the image 108, the trapezoidal region 111 is a region where the surface of the operation surface 101 in FIG. Whether the entire operation surface 101 is used as the operation region or a part of the operation surface 101 is used as the operation region is determined in advance according to the application executed in the system and the shape of the operation surface 101. The operation area is an area in which a touch operation or a gesture operation can be recognized on the operation surface 101. Furthermore, in this embodiment, it is assumed that the range on the operation surface 101 in which an image can be projected by the projector 104 matches the operation area. However, even when the entire operation surface 101 is an operation region, it may be difficult for the information processing apparatus 100 to automatically detect the edge of the operation region to acquire the boundary position of the operation region. Many. This is because an error occurs in the detection result due to a slight deviation in the position and angle of the distance image sensor 102. Therefore, the information processing apparatus 100 according to the present embodiment receives the boundary information of the operation area defined by using the coordinates (X, Y, Z) indicating the three-dimensional position information defined in the operation space by the input by the user. get.

  In the present embodiment, the information representing the boundary of the operation area is represented by using a predetermined parameter to represent the position of the pixel in the image obtained by imaging the operation space from the three-dimensional position information defined in the operation space. After conversion into position information, a conversion error is specified. Specifically, the position information indicating the position of the pixel in the image obtained as a conversion result identifies a pixel that is indicated near the boundary of the operation area from within the actually captured image, and Then, it is determined whether the operation surface 101 is actually imaged. Such a determination is performed on the entire pixel group surrounding the boundary, and a pixel for which it is determined that the operation surface 101 is actually imaged is regarded as a pixel belonging to the inside of the operation region. As a result, the portion corresponding to the pixel group determined to include the operation surface 101 is redefined as the operation region. Furthermore, with regard to the subsequent conversion processing using the error (direction and amount of deviation) between the boundary information of the operation area obtained as the conversion result of the definition information and the boundary information of the operation surface detected from the actually captured image Add corrections.

  FIG. 2A is a hardware configuration diagram of a tabletop interface including the information processing apparatus 100 according to the present embodiment. The central processing unit (CPU) 200 is connected to the system bus 204 by using the RAM 202 as a work memory, executing an OS and a program stored in the ROM 201 and the storage device 203, performing various processing calculations and logical determinations, and the like. Control each configuration. The storage device 203 is a hard disk drive, an external storage device connected by various interfaces, and the like, and stores programs and various data related to the operation recognition processing of the embodiment. The distance image sensor 102 captures a distance image of the space on the operation surface 101 under the control of the CPU 200 and outputs the captured distance image to the system bus 204. In this embodiment, the distance image acquisition method will be described based on an infrared pattern projection method that is less affected by ambient light or table surface display. However, a parallax method or infrared light reflection time is used depending on the application. It is also possible to use a method to do so. The projector 104 projects and displays an image item to be operated on the table according to the control of the CPU 200.

  In the above-described system, the visible light camera 103, the distance image sensor 102, and the projector 104 are external devices connected to the information processing apparatus 100 via an input / output interface, and cooperate with the information processing apparatus 100. Configure an information processing system. However, these devices may be integrated with the information processing apparatus 100.

  FIG. 2B is an example of a block diagram illustrating a software configuration of the information processing apparatus 100. Each of these functional units is realized by the CPU 200 developing a program stored in the ROM 201 in the RAM 202 and executing processing according to each flowchart described later. The execution result of each process is held in the RAM 202. Further, for example, when hardware is configured as an alternative to software processing using the CPU 200, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

  The image acquisition unit 210 acquires a distance image captured by the distance image sensor 102 as an input image at regular intervals, and stores the acquired distance image in the RAM 202 as needed. The object acquired by the image acquisition unit 210 and exchanged with each functional unit is actually a signal corresponding to the image data, but in this specification, simply “acquire a distance image” or “acquire an input image”. ".

  The movement area detection unit 211 performs threshold determination and noise reduction processing on each pixel of the input image acquired by the distance image acquisition unit 210, and extracts one or more movement areas from the distance image. In the case of the present embodiment, an area in which a subject is shown to be present at a position higher than the operation surface 101 that is the background by the pixel value of the distance image and an area in contact with the edge of the image is extracted as a hand area. To do. However, the moving region can be detected not only by such a background difference method but also by a method such as inter-frame difference. It is also possible to detect the moving region by searching the skin color region from the visible light image by using the visible light camera 103 also as an input image acquisition unit. The operation detection unit 212 recognizes an operation input to the information processing apparatus 100 based on a change in the position and shape of the moving area detected by the moving area detection unit 211 from the periodically acquired input image.

  The operation area acquisition unit 213 acquires information on the operation area defined using the three-dimensional coordinate system defined in the operation space. In the present embodiment, it is assumed that information on the operation area is acquired by a user input when the system is installed. However, the operation region boundary information may be acquired from history information stored in an external device or a storage device or external input. The coordinate conversion unit 214 converts coordinates in the three-dimensional coordinate system of the operation space and coordinate information in the image coordinate system defined in the input image. Conversely, mutual conversion is possible in which coordinate information in the image coordinate system defined in the input image is converted into coordinates in the three-dimensional coordinate system of the operation space. The coordinate conversion unit 214 according to the present embodiment converts the coordinate information representing the operation region acquired by the operation region acquisition unit 213 into coordinate information representing the position in the input image, so that the position of the operation region in the image get information. Also, by converting the position information detected from the content of the input image acquired by the image acquisition unit 210 into the three-dimensional coordinates defined in the operation space, position information such as the position of the operation object can be obtained. . For example, the result of converting position information of the user's fingertip position detected from the input image into three-dimensional position information in the operation space is acquired. Moreover, the actual value of the boundary information of the operation surface 101 is obtained by converting the position information obtained from the input image not including the moving area. Details of this conversion will be described later.

  The operation region determination unit 215 determines whether the operation surface 101 is captured as a subject around the boundary of the operation region in the input image acquired by the image acquisition unit 210. At that time, in order to search for the vicinity of the boundary of the operation area in the input image, the information of the operation area acquired by the operation area acquisition unit 213 is converted into the image coordinate system obtained as a result of the conversion by the coordinate conversion unit 214. Information on the operation area is used. The operation region determination unit 215 first identifies a pixel group that surrounds the operation region in the input image from the conversion result by the conversion unit 214. Then, the height information (Z coordinate) in the coordinate system defined in the operation space is obtained from the result of converting the position information in the coordinate system defined in the image of each pixel of the pixel group by the conversion unit 214. Then, it is determined whether or not the difference between the Z coordinates representing the height of the operation surface 101 is small. It is determined that the operation surface 101 is captured as a subject in a pixel from which a Z coordinate whose difference from the height of the operation surface 101 is smaller than a reference value (for example, a predetermined threshold) is obtained. The operation area determination unit 215 determines that pixels in the input image that are determined to include the operation surface 101 are inside the operation area and pixels that are not determined to include the operation surface 101 are outside the operation area. If there is, information that defines the boundary of the operation area is determined. Information on the determined operation area is stored in the RAM 202 and is referred to in processing for recognizing an operation by an operating body in the operation area.

  The correction unit 216 corrects an error occurring in the conversion process of the conversion unit 214 based on the determination result for each pixel by the operation region determination unit 215. The correction unit 216 uses, as the boundary of the operation region, the boundary between the pixel that is determined by the operation region determination unit 215 that the operation surface 101 is captured as the subject and the pixel that is not determined that the operation surface 101 is captured as the subject. decide. A difference from the conversion result of the position information input by the user input is specified as a conversion error. For example, the direction and the amount in which the coordinate shift is caused by the error are specified. In the present embodiment, the correction unit 216 adds correction based on the error to the position information that is the basis for the operation recognition by the operation detection unit 212. In the present embodiment, correction is performed by shifting the position information of the operation region obtained as a conversion result of the coordinate conversion unit 214. However, even if correction is performed by adjusting the parameter to the conversion unit 214, the correction may be performed. Good.

  The output control unit 217 uses the information stored in the ROM 201 or the storage device 203 to generate and output an image to be projected on the table 101 by the projector 104 that is the display unit of the present embodiment. For example, based on the recognition result of the multi-touch operation, at least a part of the displayed image is subjected to deformation such as enlargement / reduction and rotation, and is output to the display unit. Further, for example, the range in which the projection by the projector 104 is performed may be changed according to the operation region determined by the operation region determination unit 215. The output image is projected on the table 101 which is an operation surface by the projector 104.

  The flowchart of FIG. 3 represents the flow of the operation area | region determination process performed by this embodiment. 3 is executed as part of calibration when the information processing apparatus 100 is installed. Further, when the information processing apparatus 100 is activated, the process of the flowchart of FIG. 3 may be executed as a pre-stage of the process of detecting an operation.

  First, in step S301, the operation area acquisition unit 213 acquires operation area information in a coordinate system defined in the real space. The operation area information includes position information that defines the boundary of the operation area. Here, an example of the format of the operation region data acquired by the operation region acquisition unit 213 will be described with reference to FIG. FIG. 4A shows operation area data. FIG. 4B shows the meaning of the operation area data in FIG. 4A on the XY plane corresponding to a state where the operation space is looked down from above. In FIG. 4A, 401 is the size (size_X, size_Y) of the operation area. In this case, the values of size_X and size_Y correspond to the vertical and horizontal widths of the operation area, respectively. Reference numeral 402 denotes a shift amount (shift_X, shift_Y, shift_Z) along the XYZ axes from the origin 106 of the reference point of the operation area (the upper left point of the operation area in FIG. 4B). In particular, shift_Z represents the relative height from the origin 106 of the three-dimensional position coordinates of the operation surface. In the example of FIG. 3A, the height of the operation surface matches the height of the origin. Reference numeral 403 denotes the shape of the operation area within the size (size_X, size_Y) of the operation area. The shape 403 of the operation area is expressed by a matrix of (size_X, size_Y). A value of 0 indicates that the point is outside the operation area, and a value of 1 indicates that the point is included in the operation area. Represent.

  Returning to the flowchart of FIG. In step S302, the coordinate conversion unit 214 uses the coordinate system in the input image captured by the distance image sensor 102 (hereinafter simply referred to as image coordinates) to acquire the information on the operation area expressed in the three-dimensional position coordinate system acquired in step S301. System). A conversion formula between the three-dimensional position coordinate system and the image coordinate system is expressed by Formula (1).

  Here, d is each pixel of the input image and is a value corresponding to the distance D. r00 to r22, t0 to t2, fx, fy, cx, and cy are parameters obtained in advance by calibration when the sensor is installed. Parameters obtained by calibration are known points where the correspondence between the coordinates (X, Y, Z) representing the three-dimensional position information defined in the operation space and the coordinates (x, y) of the image coordinate system is taken. If the number is greater than or equal to the number, the value can be calculated in advance by the least square method or the like. Using these parameters, the coordinate conversion unit 214 converts each point of the operation region input in the three-dimensional position coordinate system according to Equation (1) into the image coordinate system of the distance image sensor.

  FIG. 4C shows an operation area before and after coordinate conversion. Reference numeral 404 denotes an operation area of the three-dimensional position coordinate system defined by the operation area data. Reference numeral 405 denotes an operation area of the image coordinate system converted by the coordinate conversion unit 214 using the expression (1). A region 205 is a range in which the entire square is displayed as a distance image, and a white region is an operation region set in the distance image.

  In step S303, the image acquisition unit 210 acquires a distance image as an input image. The input image is an image obtained by capturing an operation space in which the distance image sensor 102 includes at least an area defined as an operation area in the operation surface 101. Next, in step S304, the operation region determination unit 215 converts the boundary of the operation region 405 converted into the position information in the image coordinate system by the coordinate conversion unit 214 in step S302 in the input image acquired by the image acquisition unit 210. Search for pixels. Here, the search means that pixels in an area regarded as the periphery of the boundary of the operation area 405 are selected one by one as a processing target, and a value of (x, y, d) is acquired.

In step S <b> 305, the operation area determination unit 215 determines whether the operation surface 101 is regarded as a subject in the pixel selected as the processing target. In the present embodiment, the operation area determination unit 215 first converts the position information in the image coordinate system of the pixel to be processed from the coordinate conversion unit 214 into position information in the three-dimensional coordinate system of the operation space. Execute. Then, it is determined whether the information cannot be converted into the three-dimensional position information in the operation space, or whether the conversion result is different from the information obtained in step S301 as a definition. This determination corresponds to determining whether or not the height converted from the pixel value is a small difference from the height shift_Z. In addition, when it cannot convert into three-dimensional position information, when the solid object is imaged by the distance image, distance information may not be acquired well near the edge. Even when the Z-axis does not correspond to the height, the same determination may be performed based on the coordinate value on the coordinate axis intersecting the operation surface. If “ No” in step S305, the process proceeds to step S306, and the operation area determination unit 215 determines that the pixel being selected as a processing target is outside the operation area, that is, not included in the operation area. If Yes in step S305, the process proceeds to step S307, and the operation region determination unit 215 determines that the pixel being selected as a processing target is within the operation region. That is, it is determined that the pixel is included in the operation area. In step S308, the operation region determination unit 215 determines whether or not all the pixels in the region considered as the periphery of the boundary of the operation region have been searched. The processing from step S304 to step S308 is repeated until the search for all pixels is completed. When it determines with Yes by step S308, it progresses to step S309.

  In step S309, the operation region determination unit 215 determines the boundary of the operation region in the image in the image coordinate system defined for the input image. That is, the boundary between the pixel determined to be within the operation region in step S307 and the pixel determined to be outside the operation region in step S306 is determined as the boundary of the operation region, and the information is retained. At this time, the difference from the information converted in step S302 is held as a conversion error. In step S <b> 310, the movement area detection unit 211 acquires a part of the input image that is determined to be within the operation area as a background image for detecting the movement area. The background image is extracted from an image captured by the distance image sensor 102 in a state where the operation body is not included in the operation space. The above is the operation region determination processing in the present embodiment.

  Here, the difference in the operation area in the image coordinate system before and after the processing of step S304 to step S309 is executed will be described with reference to FIG. FIG. 5 shows a specific example when the operation surface 101 and the operation area match. FIG. 5A shows an input image acquired by the image acquisition unit 210 in step S303. FIG. 5A shows an operation area 405 in the image coordinate system after the coordinate conversion unit 214 performs coordinate conversion. Here, reference numeral 501 denotes an area where the operation surface 101, that is, the operation area actually exists in the distance image. When the height Z in the three-dimensional coordinate system of the operation space is calculated from the coordinates (x, y) and the pixel value d, the difference from shift_Z (information corresponding to the definition of the operation area described in the shift amount 402) is obtained. Small value. On the other hand, 502 is an area where the operation surface 101 does not exist, and the height Z does not have a small difference from shift_Z. In step S304, the operation area determination unit 215 searches for an area around the boundary.

  FIG. 5B is a diagram for explaining a region regarded as a boundary periphery. A range between the broken lines indicated by reference numeral 503 is a region regarded as the periphery of the boundary in the input image. The area considered as the periphery of the boundary in the input image includes both the inside and outside of the operation area 405. The operation region determination unit 215 acquires the three-dimensional position in step S305 for all the pixels in the region 503 regarded as the boundary periphery, and can acquire the height Z, or the difference between the height Z and shift_Z is small. Calculate whether it becomes a value. As a result, if the height is significantly different from shift_Z, it is determined that it is outside the operation region in step S306. If the difference from shift_Z is small, it is set in the operation area in step S307. FIG. 5C shows the state of the operation area after being determined in consideration of the conversion error. Reference numeral 504 denotes the determined operation area, which can acquire an operation area that matches the area where the operation surface is located in the distance image.

  FIG. 6 is another example of determining the operation region, and shows a case where a pixel portion for which no distance image is acquired exists in the input image. FIG. 6A is an input image showing the operation area of the image coordinate system. Here, reference numeral 601 denotes an area where distance information has not been acquired in the input image. Depending on the characteristics of the distance image sensor 102, an area where distance information cannot be acquired, such as 601 around the boundary of a three-dimensional object, may appear. In this embodiment, when it is determined in step S305 that distance information cannot be acquired, the pixel is determined to be outside the operation area. This is an object of reducing such misrecognition because in this area, if distance information as a background cannot be acquired in step S310, it causes an error in recognition processing of an operation executed in the subsequent stage. . 603 shown in FIG. 6B is the determined operation area. The area where the distance information cannot be acquired is excluded from the operation area.

  FIG. 7 shows an example in which the operation area is a part of the operation surface 101. Depending on the application, as described above, such a situation occurs when the entire operation surface 101 is not required as the operation region or when the operation surface 101 is larger than the angle of view of the distance image sensor 102. Even in the example of FIG. 7, in step S301, operation area data is acquired by user input. If the entire angle of view of the distance image sensor 102 is included in the operation surface 101, a portion where the operation surface 101 does not exist does not appear in the input image. However, in the process of step S301, the inclusion relationship between the operation area and the operation surface 101 is not questioned. Therefore, even in such a case, processing is performed according to the definition data acquired in step S301, as in the examples of FIGS.

  In FIG. 7A, the operation area 405 set in the image coordinate system is shown in the distance image. In FIG. 7B, an area regarded as the periphery of the boundary of the operation area 405 is indicated by 701. In the example of FIG. 7, if an area regarded as the boundary periphery is set outside the boundary, pixels other than the assumed operation area in the operation surface 101 are also determined in step S305. The determination result that it is in the operation area is obtained. Therefore, in the case of FIG. 7, the area regarded as the periphery of the boundary is set only inside the boundary of the operation area. Whether or not the operation area is only a part of the operation surface 101 is determined based on the distance of the input image in spite of the presence of the operation area boundary in the image in the shape data 403 of the operation area acquired in step S301. It can be detected because no edge appears in the value. As a result, the boundary of the operation area is set only when it is determined that the part that is defined within the operation area outside the operation surface 101 is correctly outside the operation area. . In FIG. 7C, reference numeral 702 denotes an operation region that is determined by searching the region 701 that the operation region determination unit 215 regards as the boundary periphery.

  As described above, in the present embodiment, the “operation surface” in the image information of the actually captured input image is converted into the conversion result of the position information of the “operation region” acquired as part of the calibration by the user input. The “operation area” is determined by making corrections using the method of copying. Here, the operation surface is a physical surface such as a table, a white board, or a wall, and the operation area has at least a part of the surface of the operation surface. Since the operation surface is a physical surface, the presence / absence of the presence of the operation surface can be determined from the image by analyzing the pixel information of the captured image. On the other hand, since the operation area can be arbitrarily set on the operation surface, it is difficult to determine the boundary only with the pixels of the captured image. In the system shown in FIG. 1A, one of the causes of malfunction is that the outside of the table that is the operation surface 101 is erroneously set as the operation area. If there is such an error, for example, if the user simply brings his hand close to the edge of the table, it will be erroneously recognized that the hand has touched the operation area, and an unintended operation will be performed. Therefore, in this embodiment, in order to solve such a problem, a position where it can be determined from the image information that the operation surface 101 does not exist reliably is excluded from the pixel group set as the operation area. Therefore, the operation area indicated by the information acquired before the correction considering the conversion error is not necessarily the same size as the operation area after the process for correcting the conversion error.

  In addition, in order to obtain a stable detection result when detecting an operation body such as a human finger or a stylus based on an image obtained by imaging a space with a predetermined operation surface such as a table or a wall in the background, the background is It should be static and known. In the case of the system shown in FIG. 1A, in the detection of the operating body using the captured image, it is easy to perform stable detection by taking a difference from the background in a portion where the operation surface 101 exists in the background. On the other hand, if the difference between the background information including the conversion error and the input image is taken at a portion where the operation surface 101 does not actually exist, the accuracy of the detection result of the operation tool may be lowered. Therefore, according to the above-described embodiment, the boundary between the pixels on which the operation surface 101 is captured and the pixels that are not captured in the actually captured image is specified, and the detection method of the moving region is determined for each region having different background conditions. By switching, the detection accuracy of the operating tool can be improved. In the following, a flow of recognizing an operation based on a moving area detected after such switching is performed after the operation area determination process is performed as the above-described pre-process will be described.

  FIG. 8 shows a flow of an operation detection process for detecting an operation as a subsequent process after the operation area determination process is performed as a pre-process. In the present embodiment, the flowchart of FIG. 8A is repeated every time an input image is input from the distance image sensor 102 at predetermined intervals. The predetermined period is, for example, 30 frames / 1 second and 15 frames / 1 second. First, in step S801, the image acquisition unit 210 acquires a distance image captured by the distance image sensor 102 as an input image. Next, in step S802, the movement area detection unit 211 detects a movement area. When an operating body such as a hand or pen operated by the user enters the angle of view of the distance image sensor 102, a portion where the operating body is captured is detected as a movement region.

Here, FIG. 8B is a flowchart showing the flow of the movement area detection process of the present embodiment, which is executed by the movement area detection unit 211 in step S802. FIG. 9 is a diagram illustrating the relationship between the detected moving region and the state of the hand that is the operating body. In step S811, the movement area detection unit 211 places the pixels of the input image one by one. In step S812, it is determined whether the pixel is within the operation region determined in step S309. If the pixel is within the operating area determined in step S309 (Yes in Step S 812), at step S813, it determines whether the pixel is either moving region by the background difference with input image and the background image. If the target pixel is not within the operation region determined in step S309 (No in Step S 812), the process proceeds to step S814. In step S814, the moving area detection unit 211 detects the moving area using the absolute value of the three-dimensional position coordinates defined in the operation space. That is, the pixel value obtained by converting the pixel value of the input image into the three-dimensional position coordinates defined in the operation space and obtaining a Z coordinate larger than the height Z where the operation surface should be is defined as a moving region. In step S815, it is determined whether this process has been executed for all pixels of the input image, and the process is repeated until it is determined Yes.

  FIG. 9A is an example of an input image acquired with the user's hand inserted in the operation space. FIG. 9B shows the background image acquired by the moving region detection unit 211 in step S310. However, the background image is used only within the operation area defined by the boundary information determined by the operation area determination unit 215 in the image captured in the state where the operation body does not exist in the operation space. is there. In other parts, since the operation surface 101 does not exist in the background, an error in distance information increases, and objects other than the operation body (parts other than the user's hand, other humans, etc.) are reflected. Includes noise. Therefore, no background image is generated for the area determined to be outside the operation area. This portion is shown in black in FIG. 9B.

  The moving area detection unit 211 detects a moving area based on the difference from the background image in FIG. 9B in step S813 in the portion within the operation area in FIG. On the other hand, in the portion outside the operation area, in step S814, the moving area is detected based on the value of the height Z converted using the pixel value d of each pixel. Specifically, the height Z is obtained from (x, y) and the pixel value d in the distance image based on the above equation (1). When this height is higher than shift_Z (the height of the operation surface) + α (a fixed value based on a distance error), it is detected as a moving region. As described above, in this embodiment, it is possible to detect the moving area even outside the operation area. However, the outline of the moving region may become ambiguous when compared with a portion where the background difference from the static background can be used. However, when the boundary of the region where the operation surface 101 exists in the background is not redefined according to the present embodiment, the moving region is detected by the background difference even in the region where the operation surface 101 does not actually exist (the background is not stable). . Therefore, the detection error of the outline of the moving area and the recognition error of the operation are more likely to occur. Therefore, as described above, the present invention in which the moving region detection method is divided according to the presence or absence of the background operation surface 101 is important in order to enable stable detection of the moving region. Note that the movement area detected outside the operation area may be integrated with the movement area detected within the operation area after various corrections and smoothing processing. FIG. 9C shows the result of integrating the detected moving areas. Reference numeral 901 denotes a moving area detected by background difference, and reference numeral 902 denotes a moving area detected based on a three-dimensional position.

  Returning to the flowchart of FIG. If the movement area detection unit 211 detects a movement area in step S802, the operation detection part 212 detects an operation input in the operation area in step S803. More specifically, the detected moving area is tracked and detected as an operation when a movement corresponding to a predefined command is made. The operations that can be input to the information processing apparatus 100 include a touch operation on the operation surface and a gesture operation that moves a hand in the air. The movement corresponding to the predefined command is a “tap operation” movement in which the tip of the operation body touches the operation surface 101 and leaves. Note that the position of the tip of the operating body is the position of the pixel farthest from the image edge of the input image in the moving area, or if the shape of the moving area is regarded as a hand, it is specified as the fingertip position detected from that shape. be able to. In addition to taps, the presence or absence of an operation is detected based on whether the position of the tip of the operating body, such as the fingertip, in the three-dimensional position coordinate system of the operation space and its locus are in line with those defined by the command. Is done. For example, a tap operation is detected when the height Z of the fingertip is sufficiently close to the height shift_Z of the operation surface and separated. Here, as for the tapped position, the three-dimensional position coordinates (X, Y, Z) of the fingertip are specified. Note that a touch operation including a tap is recognized as an operation input only when the touch position (X, Y) is included inside the operation region in the three-dimensional position coordinate system shown in FIG. If so, false recognition can be further suppressed.

  Next, in step S804, the correction unit 216 corrects the coordinates of the operation position related to the operation detected in step S803. Here, in the processing described in the flowchart of FIG. 3, it has already been found that there is an error in the coordinate conversion processing performed by the coordinate conversion unit 214 using Expression (1). Therefore, the position information of the moving area needs to be corrected according to the error specified in step S309. Therefore, in the present embodiment, before the operation position (for example, the position specified by the tap operation) is recognized and responded, the moving region detected from the input image at the stage of step S804 is tertiary in the operation space. The original position is corrected in consideration of the error. For example, the x and y movement amounts of the centroid positions of the operation areas before and after the correction between the operation area 405 set in the image coordinate system and the corrected operation area 504 are held as errors, and the X coordinate and the Y coordinate are stored. The value shifted by the error is used as the coordinates of the corrected operation position. The correction unit 216 notifies the operation position (for example, the tapped position) corrected in consideration of the error to a function unit that outputs a response to the operation, such as the output control unit 217.

  Note that the operation position correction method does not have to follow the order described above. For example, correction may be performed before step S803. Further, the correction method is not limited to the above, and for example, the variation in the lengths in the x and y directions of the operation area before and after correction may be corrected by multiplying by (X, Y). Alternatively, only when (X, Y) of the operation position is outside the operation area in the three-dimensional position coordinate system set in FIG. 4B, it is corrected to (X, Y) in the nearest operation area. It doesn't matter. If the operation area 504 in the corrected distance image is used, not only (X, Y) but also the height Z can be corrected. For example, the average of the height Z of each pixel of the distance image in the operation area 504 is calculated, and the height direction is also corrected by adding the difference between the value and the preset operation surface height shift_Z to Z. Things are preferable.

  After step S804, in step S805, the output control unit 217 outputs a pair to the operation input by the operation detected by the operation detection unit 212 and the operation position corrected by the correction unit 216. In this embodiment, the output image projected on the projector 104 is updated. At this time, the projection area of the output image projected on the projector 104 is moved in accordance with the determined operation area. The processing from step S801 to step S805 is repeated until the information processing apparatus 100 ends. For example, the information processing apparatus 100 ends when an end instruction is input or when a moving region is not detected for a predetermined time or longer.

  As described above, according to the present embodiment, when an arbitrary operation region is defined in the operation space, it occurs in the conversion between the coordinates in the image obtained by imaging the operation space and the three-dimensional position coordinates in the real space. The error is corrected based on information measured in the vicinity of the boundary of the operation area. Accordingly, it is possible to correct an error that occurs when detecting a position related to a moving region in which the operating tool is captured from the captured image, and reduce erroneous recognition of the operation.

<Modification>
The operation surface 101 illustrated in the first embodiment is a flat surface having a certain height. Here, as a first modification, an example will be described in which the present invention can be applied to a system having an operation surface composed of a plurality of planes having a curved surface or different heights. FIG. 10 is a diagram illustrating an example of operation area data acquired in step S301 when the height of the operation surface varies depending on the position. The size 401 of the operation area and the shift amount 402 are the same as those shown in FIG. 4 of the first embodiment. On the other hand, unlike the first embodiment, 1001 which is information indicating the shape and height of the operation area is not a flag of 0 or 1 indicating whether or not the operation area is simply used. As described above, when the shape data includes numerical values other than 0 and 1, these are values representing a difference in height from the shift amount shift_Z in the height direction. On the other hand, when the flag is not “numeric” but indicated by “n” in the figure, it indicates that the position is outside the operation area. FIG. 10B is a diagram in which the operation area data in FIG. 10A is coordinate-transformed and superimposed on the input distance image. 1002 is set in the image coordinate system based on the operation area data.

  In the first embodiment, in step S305, the operation region determination unit 215 determines whether to determine the inside of the operation region or outside the operation region based on whether or not each pixel around the boundary has the height shift_Z. Were determined. However, in the above embodiment, the height to be referenced differs depending on the position of the (X, Y) coordinates. As a method for determining the height to be referred to, in the above embodiment, the shape / height data 1001 of the operation region corresponding to the pixel is referred to, and a value added to shift_Z is obtained. For example, the height near the right boundary in FIG. 10B is (shift_Z + 5 = 5), and the height near the left boundary is (shift_Z + 10 = 10). Based on this height, it is determined whether the pixel is inside the operation area or outside the operation area. According to this modification, even if the operation surface is configured by a curved surface or a plurality of planes and the height varies depending on the position, or even when an arbitrary operation region is defined on the operation surface, the tertiary of the real space It is possible to correct an error of coordinate conversion between the original image and the image obtained by imaging the operation surface.

  In the first embodiment, the operation region determination unit 215 determines whether the three-dimensional position is different in step S305 as a reference pixel based on the operation region data in FIG. 4A or 10A. It was determined that the detected height was different from the input. However, the determination criteria may be different as long as the value obtained by converting the pixel information of the input image into the three-dimensional position coordinates in the operation space is used in the process of step S305. For example, the height Z in the vicinity of the target pixel may be searched, and a pixel in which the height in the vicinity is discontinuous may be determined as the actual boundary position of the operation region.

  In the first embodiment, only the area considered as the periphery of the boundary position in the image coordinate system is set as the processing target in step S305. However, the same result can be obtained even if processing is performed on all pixels in the operation area. Which one is to be processed may be selected according to the system resource and the purpose of the application.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

210 Image acquisition unit 211 Moving region detection unit 212 Operation detection unit 213 Operation region acquisition unit 214 Coordinate conversion unit 215 Operation region determination unit 216 Correction unit 217 Output control unit

Claims (15)

Image acquisition means for acquiring an image of a space in which the first coordinate system is defined;
Conversion means for converting position information in the first coordinate system into position information in a second coordinate system defined in an image acquired by the image acquisition means;
Operation region acquisition means for acquiring information of an operation region set on at least a part of a predetermined operation surface included in the space, expressed using position information in the first coordinate system;
Using the position information in the second coordinate system obtained as a result of the conversion means converting the information acquired by the operation area acquisition means, the boundary of the operation area in the image acquired by the image acquisition means Search for each pixel in the surrounding pixel group,
An operation area determining means for determining position information representing the operation area in the second coordinate system based on whether the operation surface is imaged as a subject in each pixel;
An information processing apparatus comprising:   Further, among the position information in the second coordinate system obtained as a result of the conversion means converting the information acquired by the operation area acquisition means, and the image acquired by the image acquisition means, the operation area determination means 2. The image processing apparatus according to claim 1, further comprising a correction unit that corrects a result of conversion by the conversion unit based on a difference from position information in the second coordinate system that represents the region determined as the operation region by the step. The information processing apparatus described in 1.   3. The information processing according to claim 1, wherein the operation area determining unit excludes, from the operation area, pixels that are not determined that the operation surface is captured as a subject in the pixel group. apparatus.   The operation region determination means determines that a pixel determined to have the operation surface captured as a subject in the pixel group is included in the operation region. The information processing apparatus according to any one of claims. The converting means can further convert position information in the second coordinate system into position information in the first coordinate system,
The operation region determination unit is configured to apply the position information of each pixel included in the pixel group in the second coordinate system to each pixel based on a result of the conversion unit converting the position information into the position information in the first coordinate system. The information processing apparatus according to claim 1, wherein it is determined whether the operation surface is captured as a subject.   The operation area determination means includes, among the position information in the first coordinate system obtained as a result of the conversion means converting position information of each pixel included in the pixel group in the second coordinate system, the operation area 6. The method according to claim 5, wherein it is determined whether or not the operation surface is captured as an object by each pixel included in the pixel group based on a coordinate value along a coordinate axis in a direction intersecting the surface. Information processing device.   The operation area determining means calculates a coordinate value along a direction intersecting the operation surface, and a height of the operation surface, out of position information in the first coordinate system corresponding to each pixel included in the pixel group. The information processing apparatus according to claim 6, wherein when the difference from the indicated coordinate value is smaller than a reference, the information processing apparatus determines that the operation surface is captured as an object at each pixel included in the pixel group.   Furthermore, an operation input to the information processing apparatus is detected based on the movement of the operating body detected from the operation area determined by the operation area determination means among the images acquired by the image acquisition means. The information processing apparatus according to claim 1, further comprising an operation detection unit.   The operation detection unit detects the operation based on the position of the operation body in the second coordinate system corrected by the correction unit. The claim according to claim 8 when dependent on claim 2 The information processing apparatus described. A moving area detecting means for detecting a moving area in which the operating body appears as a subject from the image acquired by the image acquiring means;
The moving area detecting means includes
Among the images acquired by the image acquisition means, in a portion corresponding to the operation area determined by the operation area determination means, the space is divided into an image captured in a state where the operating body is not inserted and the space. Based on the difference from the image captured with the operating tool inserted, the moving area is detected,
The operation detection unit detects a touch operation on the operation surface by the operation body based on position information of an operation position specified based on a movement region detected by the movement region detection unit. The information processing apparatus according to claim 8 or 9.   The image acquisition means is means for acquiring a distance image captured by a distance image sensor installed at a position overlooking the operation surface, which is a table surface constituting a table top interface system. The information processing apparatus according to claim 10. The moving area detecting means includes
Of the distance image acquired by the image acquisition unit, a portion that does not correspond to the operation region determined by the operation region determination unit is based on the three-dimensional position information obtained from the pixels of the distance image from the operation surface. The information processing apparatus according to claim 11, wherein an area in which an object existing above is considered to be captured is detected as the moving area. A method for controlling an information processing apparatus,
An image acquisition step of acquiring an image obtained by imaging a space in which the first coordinate system is defined by the image acquisition means;
A conversion step of converting the position information in the first coordinate system by the conversion means into the position information in the second coordinate system defined in the image acquired in the image acquisition step;
An operation region acquisition step of acquiring information of an operation region set on at least a part of a predetermined operation surface included in the space, expressed by position information in the first coordinate system, by an operation region acquisition unit; ,
By using the position information in the second coordinate system obtained as a result of converting the information acquired in the operation region acquisition step by the operation region determination means, the information acquired in the image acquisition step. Of these, searching for each pixel included in a pixel group around the boundary of the operation region,
Further, an operation region determination step of determining position information representing the operation region in the second coordinate system based on whether the operation surface is imaged as a subject at each searched pixel by the operation region determination means. When,
An information processing apparatus control method comprising: By loading and executing the computer, the computer is
Image acquisition means for acquiring an image of a space in which the first coordinate system is defined;
Conversion means for converting position information in the first coordinate system into position information in a second coordinate system defined in an image acquired by the image acquisition means;
Operation region acquisition means for acquiring information of an operation region set on at least a part of a predetermined operation surface included in the space, expressed using position information in the first coordinate system;
Using the position information in the second coordinate system obtained as a result of the conversion means converting the information acquired by the operation area acquisition means, the boundary of the operation area in the image acquired by the image acquisition means Search for each pixel in the surrounding pixel group,
An operation area determining means for determining position information representing the operation area in the second coordinate system based on whether the operation surface is imaged as a subject in each pixel;
A program that functions as an information processing apparatus.   A computer-readable storage medium storing the program according to claim 14.

Images