JP2012048393A - Information processing device and operation method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately determine a contact state of an object with a prescribed surface.SOLUTION: An operation method of an information processing device includes steps of: detecting a position of an object existing on a surface at a time of attention using a distance image sensor; specifying an edge section of the object from the detected position and a colored image obtained by photographing surroundings of the position at the time of attention; estimating the position of the specified edge section based on the position of the object; and determining a contact state of the object with the surface in accordance with the position of the edge section.

Description

  The present invention relates to a technique for determining contact.

  A user interface using a table for displaying and inputting information on the table (hereinafter referred to as an interactive table) is called a table top interface. In the table top interface, a tap (input) on the desk by a fingertip may be used. For example, Patent Document 1 discloses a technique for obtaining a finger shape and a gesture based on an image of a hand obtained by using a translucent material on a desk that is capable of diffusing projected light and transmitting the desk from below the desk. It is disclosed.

JP 2006-40271 A

  However, according to the prior art, a machine is embedded in a desk that a user contacts, and a physical load such as an impact may be applied to the machine itself embedded in the desk. Accordingly, an object of the present invention is to reduce the physical load applied to the machine itself and to accurately determine the tap (input) on the desk.

  In order to solve the above problems, an information processing apparatus according to the present invention uses a distance image sensor to detect a position of an object existing on a predetermined surface at a point of interest, and is detected at the point of interest. A specifying means for specifying an end portion of the object from a position image and a color image obtained by photographing the periphery of the position; an estimation means for estimating the position of the specified end portion based on the position of the object; Determining means for determining contact with the surface in accordance with the position of the part.

  According to the present invention, it is possible to reduce the physical load applied to the machine itself and to accurately determine the tap on the desk.

The figure which shows a mode that the interactive table etc. were installed. 1 is a functional block diagram of a system according to a first embodiment. The flowchart which shows the flow of the process which specifies an input position. The figure which shows the process with respect to an example and the distance image 205 of the distance images 205 and 207. The figure which shows the color image 218. The figure which shows the position estimated as a position of a fingertip.

(First embodiment)
Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In addition, the same code | symbol is attached | subjected about the same element and the description is abbreviate | omitted. FIG. 1 is a diagram illustrating a state in which elements (interactive table or the like) constituting the system according to the present embodiment are installed. The system according to the present embodiment includes a desk 101, a projector 104, a distance image sensor 105, a color image sensor (video camera) 106, and an arithmetic device (information processing device) 107.

  The desk 101 is a standard desk. In the present embodiment, the desk 101 has a rectangular plane and functions as a projection surface (display surface) for displaying information (characters, images, etc.) as an interactive table. The user inputs a predetermined instruction to the interactive table by bringing the fingertip 102 into contact with the desk 101. Note that a predetermined instruction may be input by bringing the tip of an object such as a pen into contact with the fingertip 102 instead of touching it. For example, when the object 103 is displayed on the desk 101, the user selects the object 103 by bringing the fingertip into contact with the area of the object 103. The projector 104 projects information on the desk 101. Note that the object 103 is an object projected by the projector 104.

  The distance image sensor 105 is installed above the desk 101 (in a direction facing the desk 101), projects an area on the desk 101, and outputs a distance image. The distance image is an image in which the brightness of each pixel in the image represents the distance from the distance image sensor to the subject reflected in the pixel. The distance image is output at each of a plurality of different time points (attention time points). The color image sensor 106 is installed above the desk 101 (in a direction facing the desk 101) and projects an area on the desk 101. The image sensor is a video camera, a camcorder, or the like. The color image output from the color image sensor is an image (video) taken by a video camera or the like, and each pixel in the image represents the color of the subject reflected in the pixel. It is assumed that the color image is output at each point of interest in the same manner as the distance image, and the corresponding color image and the distance image at the point of interest are held in association with each other.

  The arithmetic device (information processing device) 107 has a ROM, a RAM, and a CPU. The program and data stored in the ROM are expanded in the RAM, and the functions are realized by the CPU. The arithmetic device (information processing device) 107 controls the projection of information by the projector 104, and also periodically (at predetermined time intervals) the distance image from the distance image sensor 105 and the color image from the color image sensor 106. get.

  The distance image sensor 105 and the color image sensor 106 are disposed so as to include the entire surface of the desk 101 within an angle of view for acquiring an image and further from the user's fingertip 102 as viewed from the desk 101. In the present embodiment, the processing operation corresponding to each step of the flowchart to be described later is realized by software using a CPU included in the arithmetic device 107. However, part or all of the processing may be realized by hardware such as an electronic circuit, or a CPU or hardware included in the desk 101, the projector 104, the distance image sensor 105, the color image sensor 106, or the like. . Further, the present invention may be realized by changing the desk 101 to a liquid crystal display or the like without using the projector 104.

  FIG. 2 is a functional block diagram of the system according to the first embodiment. This system includes a module 201 that performs proximity determination, a module 202 that acquires fingertip coordinates, and a module 203 that estimates fingertip coordinates. First, the module 201 that determines the proximity of the desk 101 and the fingertip 102 based on the distance image will be described in detail. The distance image acquisition unit 204 accesses the distance image sensor 105 and acquires the distance image 205. FIG. 4A is a diagram illustrating an example of the distance image 205. In the present embodiment, it is assumed that the range in which the distance image 205 appears matches the desk 101. Note that if the reflected area and the desk 101 do not match, only the area corresponding to the desk 101 may be cut out by prior calibration and perspective transformation for each image acquisition. In the distance image, the brightness indicates the distance from the distance image sensor of the subject reflected in the pixel. For example, in FIG. 4A, if the optical axis of the distance image sensor and the normal of the desk are orthogonal, the brightness changes concentrically. The distance image 205 may reflect the user's arm 108. When it is detected that the user's fingertip 102 is close to the desk 101 by a certain amount (predetermined threshold) or more, the brightness of the pixel corresponding to the position of the fingertip 102 and the brightness of the pixel corresponding to the position of the desk 101 are close. . The planar image recording unit 206 acquires a distance image from the distance image sensor to the desk 101 without including the user's arm 108 and stores it as a planar image 207.

  FIG. 4B is a diagram illustrating an example of the planar image 207. In this embodiment, it is assumed that the positional relationship among the desk 101, the distance image sensor 105, and the color image sensor 106 does not change after the flat image 207 is stored. The distance image foreground / background separation unit 208 separates the area corresponding to the user's arm 108 and the other area from the distance image 205 and stores the area corresponding to the user's arm 108 as the distance image foreground 209. FIG. 4C is a diagram illustrating an example of the distance image foreground 209. The fingertip distance acquisition unit 210 obtains the fingertip coordinates 211 in the distance image and the distance 212 from the distance image acquisition means to the fingertip from the distance image 205 and the distance image foreground 209. FIG. 4C shows the coordinates 211 of the fingertip in the distance image. A distance 212 from the distance image acquisition unit to the fingertip corresponds to a distance between the distance image sensor 105 and the fingertip 102. The plane distance acquisition unit 213 obtains a distance 214 from the distance image acquisition unit to the plane from the plane image 207 and the coordinates 211 of the fingertip in the distance image. The distance 214 from the distance image acquisition means to the plane corresponds to the distance from the distance image sensor 105 to the point 109 where the line segment connecting the distance image sensor 105 and the fingertip 102 intersects the desk 101 in FIG.

  The proximity determination unit 215 calculates the fingertip from the distance 212 from the distance image acquisition means to the fingertip, the distance 214 from the distance image acquisition means to the plane, the fingertip coordinates 222 in the color image described later, and the estimated coordinates 226 of the fingertip in the color image. Information 216 indicating whether or not the planes are close to each other is obtained. The proximity determination unit 215 outputs information 216 indicating whether or not the fingertip and the plane are close to each other.

  Next, components of the module 202 that obtains the fingertip coordinates based on the color image will be described. The color image acquisition unit 217 accesses the color image sensor 106 and acquires the color image 218. FIG. 5A is a diagram illustrating an example of the color image 218. In the present embodiment, it is assumed that the color image 218 has the same range as the desk 101 in the same manner as the distance image 205. In addition, when the range shown and the range of the desk 101 do not match, only the area corresponding to the desk 101 may be cut out by calibration and perspective transformation. In the color image 218, the user's arm 108, the user's fingertip 102, and the object 103 projected by the projector may appear in the image. The color image foreground / background separation unit 219 obtains a color image foreground 220 from the color image 218 and the distance image foreground 209. FIG. 5B is a diagram illustrating an example in which a part including the fingertip 102 of the color image foreground 220 is enlarged. The fingertip coordinate acquisition unit 221 obtains the fingertip coordinates 222 in the color image from the color image foreground 220. FIG. 5B shows the fingertip coordinates 222 in the color image. The module 202 outputs the fingertip coordinates 222 in the color image.

  Next, components of the module 203 for estimating the fingertip coordinates will be described. The arm direction approximating unit 223 obtains the arm direction 224 in the distance image from the distance image foreground 209. The arm direction 224 illustrated in FIG. 4A indicates the position and orientation of the arm. The fingertip coordinate estimation unit 225 obtains the estimated coordinate 226 of the fingertip in the color image from the arm direction 224 in the distance image. FIG. 6 shows the estimated coordinates 226 of the fingertip in the color image. Note that the module 203 outputs the estimated coordinates 226 of the fingertip in the color image.

  FIG. 3 is a flowchart showing a flow of processing for specifying an input position. According to this flowchart, whether or not the user's fingertip 102 is close to the desk 101 and the coordinates of the fingertip 102 when the user's fingertip 102 is close are obtained using the table top interface configured as shown in FIG. Note that the processing in steps S301 to S313 in FIG. 3A is processing executed by the module 201. Also, the processing of steps S320 to S324 in FIG. 3B is processing executed by the module 202. Also, the processing in steps S330 to S333 in FIG. 3C is processing executed by the module 203. Each module may be realized by a device having a CPU, a ROM, a RAM, and the like.

  Steps S301 to S302 correspond to preprocessing of the module 201. First, in step S301, the positions of the distance image and the color image are calibrated. Specifically, first, a full-screen monochrome image is projected by the projector 104. Next, the distance image acquisition unit 204 and the color image acquisition unit 217 acquire a distance image and a color image, respectively. For each of the distance image and the color image, the four vertices projected by the projector 104 are detected from the image, and a matrix for perspective-transforming a rectangle connecting the four vertices into a rectangle represented by the aspect ratio of the projector is obtained. Here, it is assumed that the color image sensor has a higher resolution than the normal distance image sensor, and therefore the color image is higher in resolution than the distance image even after calibration. Thereafter, each of the distance image acquisition unit 204 and the color image acquisition unit 217 performs the perspective transformation obtained previously each time an image is acquired, and outputs a calibrated image. Next, in step S302, a planar image is acquired and stored. An example of the acquired planar image 207 is shown in FIG. Specifically, in the configuration shown in FIG. 1, a distance image is acquired from the distance image sensor 105 without the user's arm 108 on the desk 101. However, it is assumed that the calibration obtained in step S301 is also applied to the planar image 207.

  Steps S303 to S312 are the main processing of the module 201. In step S303, the object is projected onto certain coordinates. For example, suppose that the user is asked to approve some information. The object 103 including the character string “OK” is projected onto the desk 101 using the projector 104. Next, in step S304, a distance image and a color image are acquired. Here, it is assumed that the distance image 205 shown in FIG. 4A and the color image 218 shown in FIG. 5A are obtained. In the distance image 205 in FIG. 4A, it is assumed that the user's fingertip 102 is close to the desk 101 and the brightness thereof is close. Further, it is assumed that the user's arm 108 and the user's fingertip 102 are shown in the color image 218 of FIG. Next, in step S305, a background image is subtracted from the distance image and threshold processing is performed to obtain a distance image foreground. Specifically, for each pixel included in the distance image 205 shown in FIG. 4A, the brightness of the same coordinate of the planar image 207 shown in FIG. 4 is subtracted from the brightness, and the difference is set in advance. In such a case, it is assumed that the pixel is included in the distance image foreground 209. An example of the obtained distance image foreground 209 is shown in FIG. The threshold processing is intended to suppress this error because the brightness of the image is not always constant for each measurement due to a measurement error that the distance image sensor 105 generally has. On the other hand, when the distance from the desk 101 to the fingertip 102 is lower than the threshold value or the quantization accuracy of the distance image sensor, the area of the fingertip cannot be distinguished from the desk and is not included in the distance image foreground as a result. For the above reason, it is assumed that the range image foreground 209 illustrated in FIG. 5C does not include an area corresponding to the fingertip 102. Here, erosion processing and enlargement processing may be performed on the obtained image foreground to remove small noise.

  Next, in step S306, a connected pixel block is obtained for the foreground of the distance image, a connected pixel block including a finger is selected, and an approximate ellipse is obtained. The generation method of the connected pixel block is arbitrary, and may be four bonds or eight bonds. For the selection of a connected pixel block including a finger, for example, if the distance image foreground includes only one connected pixel block, it may be selected, and if a plurality of connected pixel blocks are included, the maximum one is selected. You may choose. Assume that the same area as the distance image foreground 209 is obtained as the connected pixel block 501. In order to derive the approximate ellipse of the connected pixel block 501, a known method such as a method of least square estimation of ellipse parameters for five or more points on the contour is used. Hereinafter, it is assumed that the approximate ellipse 502 is obtained by using a known method. Next, in step S307, the distance of the fingertip is obtained from the coordinates of the fingertip in the distance image and the distance image sensor. Specifically, a point where the major axis 503 of the approximate ellipse 502 intersects the approximate ellipse 502 is obtained, and the point is set as the coordinate 211 of the fingertip in the distance image. Next, the brightness of each pixel included in the range of a circle 504 having a certain radius around the fingertip coordinates 211 and included in the connected pixel block 501 is averaged to obtain a distance 212 from the distance image acquisition means to the fingertip. Next, in step S308, the distance from the distance image sensor to the desk is obtained. What is necessary is just to obtain | require the brightness in the plane image 207 of the coordinate equivalent to the coordinate 211 of the fingertip calculated | required by step S307.

  Next, in step S309, it is determined whether the fingertip is close to the desk. If the fingertip is close to the desk, the process proceeds to step S310, and if not, the process proceeds to step S303. For the determination of the proximity of the fingertip and the desk, first, when the distance from the distance image sensor to the desk calculated in step S308 is lower than the threshold set in advance from the distance 212 calculated in step S307, It is determined that the desk 101 is close. Here, since the brightness of each pixel in the distance image foreground 209 is far from the threshold used in step S305 from the desk 101, the threshold used for identifying the desk 101 and the fingertip 102 is the threshold used in step S305. Must be bigger than. Next, in step S310, the coordinates of the fingertip in the color image are obtained. The color image is 218 in FIG. 5A, and the enlarged view of the region including the fingertip in the color image (the position of the fingertip and its periphery) is shown in FIG. 5B. In this figure, the coordinates 222 of the fingertip in the color image are shown. Details of step S310 will be described later.

  Next, in step S311, the position of the fingertip in the color image is estimated from the fingertip of the distance image. FIG. 6 shows the estimated coordinates 226 of the fingertip in the color image. Details of step S311 will be described later. Next, in step S312, if the fingertip is close to the desk, the process proceeds to step S313, and if not, the process proceeds to step S303. Unlike step S309, the proximity of the fingertip and the desk is determined by calculating the distance between the fingertip coordinates 222 obtained in step S310 and the estimated fingertip coordinates 226 estimated from the distance image in step S311. In the following cases, it is determined that the desk and the fingertip are close to each other. Here, if the distance is set to infinity, it is determined that the fingertip is always close to the desk. In other words, the lower limit of performance is guaranteed.

  In step S313, processing is performed when the fingertip is close to the desk. Specifically, for example, if a button including the character string “OK” is projected in step S303, and if the fingertip coordinates 222 obtained in step S310 are included in the button, the button is pressed. It is determined that Finally, returning to step S303, the detection of the fingertip coordinates is repeated again. The information 216 indicating whether or not the fingertip is close to the desk determined in step S312 is the output of the module 201. As described above, according to the present invention, the proximity of the fingertip and the desk can be easily determined.

  Next, the module 202 will be described. The processing by the module 202 corresponds to the processing in step S310 in FIG. Hereinafter, a description will be given with reference to FIG. In FIG. 3B, a distance image foreground, a connected pixel block and its approximate ellipse, and a color image are input. FIG. 4C illustrates a distance image foreground 209, a connected pixel block 501, an approximate ellipse 502, and a color image 218 illustrated in FIG.

  First, in step S320, processing for obtaining a distance image foreground 209, a connected pixel block 501, an approximate ellipse 502, and a color image 218 is executed. Next, in step S321, a foreground / background uncertain region is set on the outline of the connected pixel block. FIG. 4D shows a connected pixel and a foreground / background uncertain region. Note that the foreground / background indeterminate region corresponds to a region where the proximity of the fingertip and the desk is detected and its surrounding region. In step S321, first, the major axis 503 of the approximate ellipse 502 of the connected pixel block 501 obtained in step S306 is extended in the fingertip direction to create an ellipse 505. Next, a logical sum of an ellipse 505 (shaded portion) and an area 506 (blackened) obtained by enlarging the connected pixel block 501 obtained in step S306 is obtained. The logical product of this logical sum and the negation of the area 507 (outlined) in which the connected pixel block 501 has been subjected to erosion processing is obtained and set as a foreground / background indeterminate area 508 (shaded area + black). Here, by setting the enlargement ratio of the approximate ellipse 502 to an appropriate value, the area of the fingertip 102 that was not included in the foreground by the threshold processing at the time of obtaining the distance image foreground in step S305 is also included in the foreground / background uncertain area 508. Can do.

  Next, in step S322, an uncertain area corresponding to the foreground / background uncertain area on the distance image is obtained on the color image. As shown in step S301, the resolution of the color image 218 is rougher than that of the distance image 205. Therefore, in order to obtain an area on the color image 218 corresponding to an area on the distance image 205, simple enlargement may be performed. Specifically, the resolution ratio between the distance image 205 and the color image 218 is obtained, and an area obtained by enlarging the area 507 obtained by performing the erosion process on the connected pixel block obtained in step S321 according to the obtained resolution ratio is set as the foreground 601. . Similarly, an area in which the foreground / background indeterminate area 508 obtained in step S321 is enlarged in accordance with the obtained resolution ratio is set as an indeterminate area 602. A region not included in both the regions is set as a background 603. FIG. 5B shows an enlarged view of the arm portion of the color image shown in FIG. Assume that one of the black squares forming the foreground 601 corresponds to one pixel on the distance image 205 of the transfer source. Similarly, one of the white squares forming the background 603 corresponds to one background pixel on the distance image 205 of the transfer source. The indeterminate region 602 includes the contours of the arm 108 and the fingertip 102.

  Next, in step S323, a foreground is obtained within an indeterminate area on the color image. FIG. 5B shows a color image foreground 220 that is separated into a shaded area and black paint in FIG. That is, the foreground 220 includes the foreground 601 obtained in step S322. Here, in order to separate the foreground from the background, a technique called Graph Cut is generally applicable. First, for each of the foreground 601 and the background 603, a color model of a pixel group included in the foreground 601 or the background 603 and close to the indeterminate region 602 is acquired. To obtain the color model, the Gaussian Mixture Model of the distribution is approximated by EM algorithm. Next, for each pixel in the uncertain region, a graph is generated in which each pixel is a node and the adjacent relationship between the pixels is a link. Next, for each pixel in the uncertain region 602, the probability belonging to the foreground 601 and the probability belonging to the background 603 are determined according to the respective Gaussian Mixture Models. The weight of each link is defined as the similarity of the color of the pixel as a node. For the above graph, add some nodes and links from the foreground 601 and background 603. The graph is cut with a maximum flow and a minimum using, for example, linear programming. Whether or not each pixel in the indeterminate region 602 belongs to the foreground 220 is determined by the Graph Cut method. A logical sum of the obtained foreground 220 and the foreground 601 is obtained to obtain the foreground 220.

  Next, in step S324, the coordinates of the fingertip in the color image are obtained based on the foreground on the color image. Specifically, for example, for the color image foreground 220, the connected pixel block is obtained, and for each pixel on the outline of the connected pixel block, the coordinates of the foot of the perpendicular to the major axis 604 are obtained, and the maximum value or the minimum Find the value farther from the edge of the image. The coordinates of the pixel on the obtained contour are taken as the fingertip coordinates 222 in the color image.

  The fingertip coordinate 222 obtained in step S324 is the output of the module 202. As described above, if the color image 218 has a higher resolution than the distance image 205, the fingertip coordinates 222 in the color image are more accurate than the fingertip coordinates 211 in the distance image. Further, by obtaining the distance image foreground 209 with high accuracy from the distance image 205 and narrowing down the processing area, the speed can be increased compared to the case where the foreground / background separation is performed on the entire color image 218. Further, by limiting the foreground / background area for generating the color model to the foreground 601 and the background 603 adjacent to the uncertain area 602, dispersion of the color model can be suppressed, and the accuracy of the foreground / background separation can be improved.

  Next, processing in the module 203 will be described. The module 203 corresponds to the details of step S311 in FIG. A flowchart is shown in steps S330 to S333 in FIG. FIG. 3C receives a distance image, a foreground connected pixel block, and an approximate ellipse thereof. 4A shows a distance image, and FIG. 4C shows a connected pixel block 501 and an approximate ellipse 502 in the foreground.

  First, in step S330, a process for obtaining a distance image foreground, a connected pixel block 501, and an approximate ellipse 502 is executed. Next, in step S331, the arm direction is obtained from the point sequence included in the connected pixel block on the long axis of the ellipse. Specifically, a straight line is approximated by the least square method or the like with respect to the brightness of each pixel on the major axis 503 of the approximate ellipse 502. FIG. 6 shows a view of the arm region of FIG. 4A viewed from a viewpoint (side direction) parallel to the plane of the desk 101. In FIG. 6, the desk 101 and the arm 108 are reflected. A lower limit 701 that is not included in the distance image foreground 209 by the threshold processing in step S305 is shown. Each pixel of the distance image foreground 209 indicates a distance from the distance image sensor 105 to the upper surface 702 of the arm 108. 6 corresponds to the upper surface 702. A straight line approximating the upper surface 702 is obtained and set as the arm direction 224. Next, in step S332, the desk direction is obtained from the point sequence on the planar image corresponding to the point sequence included in the connected pixel block on the major axis of the ellipse. Specifically, the long axis 503 in FIG. 4C approximates the brightness of each point in the point sequence 401 corresponding to the planar image by the least square method or the like. FIG. 6 shows a desired desk direction 703.

  Next, in step S333, the intersection of the desk direction and the arm direction is obtained and used as the estimated coordinates of the fingertip in the color image. That is, the intersection of the arm direction 224 and the desk direction 703 is obtained and used as the estimated coordinate 226 of the fingertip in the color image. The estimated coordinate 226 of the fingertip in the color image is the output of the module 203. As described above, in the distance image 205, the region of the fingertip 102 close to the desk 101 may be lost due to the threshold processing of the distance image foreground / background separation unit 208, and it may be difficult to obtain the region. Therefore, in this embodiment, the estimated coordinate 226 of the fingertip can be obtained. Further, since the arm direction 224 and the desk direction 703 are estimated from the brightness of a plurality of pixels of the distance image sensor 105, the influence of the distance measurement error as long as it is independent for each pixel of the distance image sensor 105 is reduced. be able to. In step S312 of the module 201 that performs proximity determination based on the distance image, when the distance between the fingertip coordinates 222 based on the color image and the estimated fingertip coordinates 226 estimated from the color image is equal to or smaller than the threshold value, the desk 101 and the fingertip 102 Is determined to be close. Therefore, by appropriately setting this threshold, it is determined that the fingertip 102 is not close to the desk 101 when it is closer to the desk 101 than the threshold used in step S309 and is not close to the desk 101. Can do.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (3)

Detecting means for detecting the position of an object existing on a predetermined surface at the time of interest using a distance image sensor;
A specifying means for specifying an end portion of the object from a color image obtained by photographing the detected position and the periphery of the position at the time of interest;
Estimating means for estimating the position of the identified end based on the position of the object;
An information processing apparatus comprising: a determination unit that determines contact with the surface according to the position of the end portion. An operation method of an information processing apparatus for determining contact with a predetermined surface,
A detection step of detecting a position of an object existing on the surface at the time of interest using a distance image sensor;
A specifying step of specifying an end of the object from a color image obtained by photographing the detected position and the periphery of the position at the time of interest;
An estimation step of estimating the position of the identified end based on the position of the object;
A determination step of determining contact with the surface in accordance with the position of the end portion.   A program for causing a computer to execute the operation method according to claim 2.