JP2017112565A - Projection device, projection method and computer program for projection - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a projection device capable of improving visibility of a user with respect to an image of a holding target object held by an object that becomes a projection target.SOLUTION: A projection device comprises: an object region detection part 21 for detecting an object region in which a target object is imaged, on a distance image that is obtained during an imaging period; a real space position calculation part 22 for calculating a position of a target object in a real space for each of pixels within the object region, the target object being imaged in that pixel; a tracking part 23 for generating a track data representing a track of the target object in the real space; a hold determination part 24 for identifying a first period in which a holding target object is being held by the target object, based on changes in a height of the holding target object within a holding object region and a height of the object after the track of the target object in the real space passes the holding object region; and a projection control part 26 for causing a projector 13 to project an image of the target object and an image indicating the holding target object that is being held, at mutually different positions on a projection screen during a second period, included in a projection period, corresponding to the first period.SELECTED DRAWING: Figure 3

Description

  The present invention relates to, for example, a projection apparatus that projects an image of an object on a predetermined projection plane, and a projection method and a computer program for projection used in such a projection apparatus.

  In recent years, work support systems that use augmented reality (AR) have been proposed (see, for example, Non-Patent Document 1). In the AR support system disclosed in Non-Patent Document 1, when a user holds a camera in front of a target to be worked on, a teacher video is superimposed and displayed on a video showing the target. At this time, the AR support system appropriately modifies the teacher video in order to match the line of sight of the teacher video with the line of sight of the video captured by the camera in accordance with the positional relationship between the camera and the operation target. The video is superimposed on the camera image.

Goto et al., “AR Support System with Teacher Video Overlay Display in Workspace”, Journal of the Institute of Image Electronics Engineers of Japan, Vol. 39, No. 6, pp.1108-1120, 2010

  Here, it is assumed that the teacher video is projected onto a predetermined projection plane using a projector. In this case, in the modification of the teacher video as in the technique described in Non-Patent Document 1, an image of an object to be grasped such as a tool or a part that can be grasped by a teacher's hand or the like shown in the teacher video is displayed on the projection plane. It may be projected on the work target object placed. However, if the work target object has a complicated structure, the projected image of the gripping target object is superimposed on the structure, which may make it very difficult to see the image. In addition, in a state where the gripping target object is gripped by a hand or the like, there is an occlusion in which a part of the gripping target object is hidden by the hand or the like, so that even if an image of the gripping target object in that state is projected, the user This makes it difficult to understand the entire object to be grasped.

  In one aspect, an object of the present invention is to provide a projection apparatus capable of improving the visibility of a user with respect to an image of a gripping target object held by an object to be projected.

  According to one embodiment, a projection device is provided. This projection apparatus displays a distance between a target object and a distance sensor obtained from a distance sensor as a pixel value during a predetermined imaging period, and a storage unit that stores a gripping object region that includes the gripping target object in real space. In the distance image, an object region detection unit that detects an object region in which the target object is captured, and in the distance image, for each pixel corresponding to the target object in the object region, the position of the target object in the pixel is represented in real space. A real space position calculating unit for calculating, a tracking unit for generating trajectory data representing the trajectory of the target object in real space based on the position of the target object in real space corresponding to the distance image, The first period in which the gripping target object is gripped by the target object is identified based on the change between the height of the gripping target object and the height of the target object after the trajectory of the target object passes through the gripping object region. You In the second period of the predetermined projection period corresponding to the first period, the image of the target object and the image indicating the target object held by the target object are projected onto the projector. A projection control unit for projecting to different positions on the surface.

  The visibility of the user with respect to the image of the gripping target object gripped by the object to be projected can be improved.

It is a hardware block diagram of the projection apparatus by one Embodiment. It is a figure which shows an example of the positional relationship of the projection surface by which the image of the distance sensor of the projection apparatus, camera, projector, and tracking target object by one Embodiment is projected. It is a functional block diagram of a control part by one embodiment. It is a figure which shows the relationship between a distance sensor coordinate system and a world coordinate system. It is a figure which shows an example of a holding | grip object area | region. It is a figure which shows an example of locus data. It is explanatory drawing of the outline | summary of a grip determination. It is an operation | movement flowchart of a holding | grip determination process. (A) is a schematic diagram of a projection plane when no gripping target object is gripped. (B) is a schematic diagram of a projection plane when any gripping target object is gripped by a tracking target object. It is an operation | movement flowchart of a projection process. It is a functional block diagram of a control part by a modification. (A) is a schematic diagram of a projection surface when no object exists in the gripping object region. (B) is a schematic diagram of a projection surface when an object different from the gripping target object being gripped exists in the gripping object region. (C) is a schematic diagram of the projection plane when the same gripping target object as the gripped target object is present in the gripping object region.

  Hereinafter, the projection apparatus will be described with reference to the drawings. This projection apparatus is a distance sensor that can measure the distance to an object located within a detection range, and continuously captures an object to be tracked (hereinafter referred to as a tracking target object) over a certain period. The projection apparatus obtains the position of the tracking target object in real space from each of a plurality of distance images obtained during the photographing. The projection apparatus is based on the change in the height of the object in the region before and after the locus of the position change in the real space of the tracking target object passes through the region including the object that the tracking target object can hold. The period during which the tracking target object is gripping the gripping target object is specified. An object that can be gripped by the tracking target object is hereinafter referred to as a gripping target object. Then, when projecting the image of the tracking target object obtained from each distance image by the projector according to the shooting order of the distance image, the projection apparatus holds the gripping target object while the tracking target object grips the gripping target object. The image of the object is also projected at a position different from the image of the tracking target object.

  In the present embodiment, the projection apparatus detects the leader's hand as a tracking target object from each of a plurality of distance images taken while the leader performs a series of operations on the work table. In this embodiment, a tool or a part gripped by the instructor's hand is set as a gripping target object. Then, when instructing the user to perform the work, the projection apparatus sequentially projects the hand image detected from each distance image on the projection plane as a teacher video according to the distance image acquisition order. In the following, the execution period of photographing by the distance sensor for a series of work by the instructor is simply referred to as an imaging period, and the execution period of the teacher video projection by the projector is hereinafter simply referred to as a projection period.

  FIG. 1 is a hardware configuration diagram of a projection apparatus according to an embodiment. FIG. 2 is a diagram illustrating an example of a positional relationship between a distance sensor, a camera, and a projector of a projection apparatus according to an embodiment and a projection plane onto which an image of a tracking target object is projected. The projection device 1 includes a distance sensor 11, a camera 12, a projector 13, a storage unit 14, and a control unit 15. The distance sensor 11, the camera 12, the projector 13, and the storage unit 14 are connected to the control unit 15 via signal lines, respectively. Further, the projection apparatus 1 may further include a communication interface (not shown) for connecting the projection apparatus 1 to other devices.

  The distance sensor 11 is an example of a distance measurement unit, and as illustrated in FIG. 2, for example, the projection direction of the projector 13 so that at least a part of the surface 100 of the work table on which the user works is included in the detection range. It is attached toward. The distance sensor 11 generates a distance image of the detection range at regular intervals (for example, 30 frames / second to 60 frames / second) during the imaging period. Then, the distance sensor 11 outputs the generated distance image to the control unit 15. Therefore, the distance sensor 11 can be a depth camera, for example. The distance image represents, for example, the distance from the distance sensor 11 to the tracking target object 101 or the like shown in the pixel for each pixel by the value of the pixel. For example, the closer the distance from the distance sensor 11 to the point of the tracking target object 101 shown in the pixel in the distance image, the larger the value of that pixel.

The camera 12 is an example of an imaging unit. The camera 12 includes, for example, a work surface 100, a tracking target object 101, a work target object 102 placed on the work surface 100, and a gripping target object 103 that the tracking target object 101 may grip. It is attached so as to be included in the shooting range of the camera 12. The camera 12 generates an image at regular intervals (for example, 30 frames / second to 60 frames / second) during the imaging period. The camera 12 outputs the generated image to the control unit 15 every time an image is generated. Alternatively, the camera 12 is configured so that the tracking target object does not exist within the imaging range of the camera 12 and the work target object 102 and all the gripping target objects 103 are placed on the surface 100 of the workbench. The shooting range may be shot only once.
Note that the image generated by the camera 12 may be a gray image or a color image. In the present embodiment, it is assumed that the camera 12 generates a color image.

The projector 13 is an example of a projection unit, and is installed so as to project an image toward a projection surface as shown in FIG. In the present embodiment, the projection plane is set on the surface 100 of the work table or the surface of the work target object 102. Therefore, the projector 13 is attached vertically above the work table.
The projector 13 is a liquid crystal projector, for example, and projects an image by displaying the image on the display surface in accordance with the image signal received from the control unit 15. In the present embodiment, the projector 13 projects an image of the tracking target object 101 on the projection plane. Further, the projector 13 has a period during the projection period (that is, the second period) corresponding to a period during which any of the gripping target objects 103 is gripped during the imaging period (that is, the first period). An image indicating the gripped target object 103 is projected onto the projection plane.

  The storage unit 14 includes, for example, a volatile or non-volatile semiconductor memory circuit. The storage unit 14 stores a distance image obtained by the distance sensor 11, an image obtained by the camera 12, a video signal representing an image projected by the projector 13, and the like. Furthermore, the storage unit 14 stores various information used in the projection processing, for example, the installation position and orientation of the distance sensor 11 in the world coordinate system, the number of pixels included in the distance image, the diagonal viewing angle of the distance sensor 11, and the like. Remember. Similarly, the storage unit 14 stores the installation position and orientation of the camera 12 and the projector 13 in the world coordinate system, the number of pixels of the camera 12, the number of pixels on the display surface of the projector 13, and the diagonal viewing angle. Further, the storage unit 14 stores various data generated during the projection process, for example, trajectory data.

  The control unit 15 includes one or a plurality of processors and their peripheral circuits. And the control part 15 controls the projection apparatus 1 whole.

Hereinafter, details of components related to the projection processing executed by the control unit 15 will be described.
FIG. 3 is a functional block diagram of the control unit 15. The control unit 15 includes a gripping object region setting unit 20, an object region detection unit 21, a real space position calculation unit 22, a tracking unit 23, a gripping determination unit 24, a display region calculation unit 25, and a projection control unit 26. And have.
Each of these units included in the control unit 15 is implemented as a functional module realized by a computer program executed on a processor included in the control unit 15, for example. Alternatively, one or a plurality of integrated circuits that realize the functions of these units may be mounted on the projection apparatus 1 separately from the control unit 15.

  Hereinafter, the processing of each unit of the control unit 15 will be described separately for the processing for the imaging period and the processing for the projection period.

(Imaging period)
The gripping object region setting unit 20 sets a gripping object region in the real space, which is a real space region including the gripping target object when it is not gripped by the tracking target object.

  For this reason, the gripping object region setting unit 20 is configured so that, for example, when the tracking target object does not exist within the detection range of the distance sensor 11 and all the gripping target objects are placed on the workbench, 11 is detected from the distance image obtained in step 11. For example, the gripping object region setting unit 20 compares the value of each pixel on the distance image with the value of the corresponding pixel on the reference distance image obtained when the work table is photographed by the distance sensor 11. . Note that the reference distance image is stored in the storage unit 14 in advance. The gripping object region setting unit 20 extracts pixels in which the absolute value of the difference between the pixel value of the distance image and the corresponding pixel value of the reference distance image is equal to or greater than a predetermined threshold value. The predetermined threshold value can be a pixel value on the distance image corresponding to 1 cm to 2 cm, for example.

  The gripping object region setting unit 20 executes, for example, a labeling process on the extracted pixels, and groups the extracted pixels so that each pixel includes the extracted pixels and is separated from each other. Find one or more candidate regions. Note that the gripping object region setting unit 20 deletes the candidate region, assuming that the candidate region included in the range in which the work target object is assumed to be captured in the distance image is irrelevant to the gripping target object. Also good. Then, the gripping object region setting unit 20 shows the gripping target object in a candidate region included in a predetermined range in which the number of pixels included in each candidate region corresponds to the size of the gripping target object. Is determined.

  The gripping object region setting unit 20 sets a region in the real space corresponding to the candidate region that is determined to include the gripping target object as the gripping object region. For this purpose, the gripping object region setting unit 20 calculates, for example, the position in real space of a point on the gripping target object corresponding to each pixel in the candidate region that is determined that the gripping target object is captured. First, the gripping object region setting unit 20 calculates the position of the point of the gripping target object reflected in the pixel in the distance sensor coordinate system for the pixel of interest. The distance sensor coordinate system is a coordinate system in real space with the distance sensor 11 as a reference. Then, the gripping object region setting unit 20 converts the coordinates of the point of the gripping target object corresponding to the pixel of interest, expressed in the distance sensor coordinate system, into coordinates in the world coordinate system based on the surface of the work table.

  FIG. 4 is a diagram showing the relationship between the distance sensor coordinate system and the world coordinate system. In the distance sensor coordinate system 400, the Zd axis is set in a direction parallel to the optical axis of the distance sensor 11, and the Xd axis and the Yd axis are parallel to the plane perpendicular to the Zd axis, that is, the sensor surface of the distance sensor 11. In the plane, they are set to be orthogonal to each other. For example, the Xd axis is set in a direction corresponding to the horizontal direction in the distance image, and the Yd axis is set in a direction corresponding to the vertical direction in the distance image. The origin of the distance sensor coordinate system 400 is set to the center of the sensor surface of the distance sensor 11, that is, the point where the sensor surface and the optical axis intersect.

  On the other hand, in the world coordinate system 410, two axes Xw axis and Yw axis orthogonal to each other are set on the surface 100 of the workbench, and the Zw axis is parallel to the normal direction of the surface 100 of the workbench. Is set. The origin of the world coordinate system 410 is set, for example, at one end of the projection range of the projector 13 or the center of the projection range on the surface 100 of the work table. The positional relationship between the distance sensor 11 and the work surface 100 is already known. That is, the coordinates of an arbitrary point on the distance sensor coordinate system 400 can be converted to the coordinates of the world coordinate system 410 by affine transformation.

The gripping object region setting unit 20 calculates the coordinates (X _d , Y _d , Z _d ) in the distance sensor coordinate system of the point of the gripping target object shown in the pixel for the pixel of interest according to the following equation.
Equation (1) is an equation according to the pinhole camera model, f _d represents the focal length of the distance sensor 11, and DW and DH respectively represent the number of horizontal pixels and the number of vertical pixels of the distance image. Represent. DfovD represents the diagonal viewing angle of the distance sensor 11. (X _d , y _d ) represents a horizontal position and a vertical position on the distance image. Z _d is the distance between the point of the tracking target object at the position corresponding to the pixel (x _d , y _d ) on the distance image and the distance sensor 11, and the value of the pixel (x _d , y _d ) Is calculated based on

Further, the gripping object region setting unit 20 determines the coordinates in the distance sensor coordinate system of the point of the gripping target object that is reflected in the pixel of interest in the coordinate system (X _W , Y _W , Z _W ).
Here, R _DW is a rotation matrix representing the rotation amount included in the affine transformation from the distance sensor coordinate system to the world coordinate system, and t _DW is a translation vector representing the translation amount included in the affine transformation. is there. DLocX, DLocY, and DLocZ are the coordinates of the center of the sensor surface of the distance sensor 11 in the Xw axis direction, the Yw axis direction, and the Zw axis direction of the world coordinate system, that is, the coordinates of the origin of the distance sensor coordinate system, respectively. . DRotX, DRotY, and DRotZ represent rotation angles in the optical axis direction of the distance sensor 11 with respect to the Xw axis, the Yw axis, and the Zw axis, respectively.

  The gripping object region setting unit 20 determines the position in real space of the point of the gripping target object corresponding to each pixel in the candidate region that is determined to include the gripping target object according to the formulas (1) and (2). calculate. As a result, the position and range of the object to be grasped in real space are obtained. Then, the gripping object region setting unit 20 sets a circumscribed rectangle of the gripping target object as the gripping object region on the XwYw plane of the world coordinate system.

  FIG. 5 is a diagram illustrating an example of a gripping object region. As shown in FIG. 5, the gripping object region 510 is set as a rectangular region that has a side parallel to the Xw axis and a side parallel to the Yw axis and circumscribes the gripping target object 500. Note that the gripping object region 510 may be set as a circumscribed ellipse of the gripping target object 500, or may be the region itself where the gripping target object 500 exists on the XwYw plane.

  Further, the gripping object region setting unit 20 is obtained by the camera 12 when the tracking target object does not exist within the detection range of the distance sensor 11 and all the gripping target objects are placed on the work table. The region corresponding to the gripping object region is specified for each gripping target object on the captured image. Then, the gripping object region setting unit 20 cuts out a region corresponding to the gripping object region from the image for each gripping target object, and stores it as a gripping object image representing the gripping target object together with the corresponding gripping object region. Save to.

In order to create a gripping object image, the gripping object region setting unit 20 converts the coordinates of the real space position of each corner of the gripping object region of interest from the coordinates of the world coordinate system to the coordinates of the camera coordinate system. At that time, the gripping object region setting unit 20 sets the coordinates (X _W , Y _W , Z _W ) of the corners in the world coordinate system for each corner according to the following equations (X _C , Y _C , Z _C )
Here, R _WC is a rotation matrix that represents the amount of rotation included in the affine transformation from the world coordinate system to the camera coordinate system, and t _WC is a translation vector that represents the amount of translation included in the affine transformation. . R _CW is a rotation matrix included in the affine transformation from the camera coordinate system to the world coordinate system, and t _CW is a translation vector representing the translation amount included in the affine transformation. CLocX, CLocY, and CLocZ are the coordinates of the center of the image sensor of the camera 12 in the Xw-axis direction, the Yw-axis direction, and the Zw-axis direction of the world coordinate system, that is, the coordinates of the origin of the camera coordinate system, respectively. CRotX, CRotY, and CRotZ represent rotation angles in the optical axis direction of the camera 12 with respect to the Xw axis, the Yw axis, and the Zw axis, respectively.

For each gripping object region, the gripping object region setting unit 20 corresponds to the coordinates (X _C , Y _C , Z _C ) in the camera coordinate system for each corner of the gripping object region on the image obtained by the camera 12. The coordinates (x _c , y _c ) of the pixel are specified according to the following equation.
Here, fc represents the focal length of the camera 12, and CW and CH represent the number of horizontal pixels and the number of vertical pixels of the image obtained by the camera 12, respectively. CfovD represents the diagonal viewing angle of the camera 12.

  For each gripping object region, the gripping object region setting unit 20 cuts out from the image a region surrounded by four points on the image corresponding to each corner of the gripping object region as a gripping object image.

  The object area detection unit 21 detects an object area that is an area in which the tracking target object is captured from each distance image acquired by the control unit 15 from the distance sensor 11 during the imaging period. In the present embodiment, the object region detection unit 21 executes the same process for each distance image, and therefore, the process for one distance image will be described below.

  For example, the object region detection unit 21 compares the value of each pixel on the distance image with the value of the corresponding pixel on the reference distance image. The object region detection unit 21 extracts pixels in which the absolute value of the difference between the pixel value of the distance image and the corresponding pixel value of the reference distance image is equal to or greater than a predetermined threshold value.

  The object region detection unit 21 executes, for example, a labeling process on the extracted pixels, and groups the extracted pixels, thereby including the extracted pixels and separating them from each other. The above candidate areas are obtained. Then, the object region detection unit 21 sets a candidate region having the maximum number of pixels included in the candidate region among the one or more candidate regions as the object region. Since both hands of the instructor may be shown in the distance image, the object region detection unit 21 selects two candidate regions in order from the one with the largest number of pixels included in the candidate region. It is good. Alternatively, the object region detection unit 21 does not include the object region in the distance image when there is no candidate region in which the number of pixels included is equal to or greater than the predetermined number of pixels, that is, the tracking target object is included in the distance image. It may be determined that is not shown.

  The object region detection unit 21 notifies the real space position calculation unit 22 of the object region on the distance image.

  For each pixel included in the object region detected from each distance image acquired from the distance sensor 11 by the control unit 15 from the distance sensor 11, the real space position calculation unit 22 in the real space of the point of the tracking target object reflected in the pixel. Calculate the position. In the present embodiment, the real space position calculation unit 22 executes the same process for each distance image, and therefore, the process for one distance image will be described below.

  First, for each pixel included in the object region, the real space position calculation unit 22 calculates the position in the distance sensor coordinate system of the point of the tracking target object shown in the pixel. Then, the real space position calculation unit 22 converts the coordinates of the point of the tracking target object corresponding to each pixel expressed in the distance sensor coordinate system into coordinates in the world coordinate system.

For each pixel included in the object region, the real space position calculation unit 22 calculates the coordinates (X _d , Y _d , Z _d ) in the distance sensor coordinate system of the point of the tracking target object reflected in the pixel as (1 ) Calculate according to the formula. Further, the real space position calculation unit 22 calculates, for each pixel included in the object region, the coordinates in the distance sensor coordinate system of the point of the tracking target object reflected in the pixel in the world coordinate system according to the equation (2). Convert to coordinates (X _W , Y _W , Z _W ).

Thus, the real space position calculation unit 22 calculates the actual position of the tracking target object in the real space by calculating the position of the point of the tracking target object shown in each pixel included in the object region in the real space. A range and a position corresponding to the size of can be obtained.
The real space position calculation unit 22 notifies the tracking unit 23 and the display region calculation unit 25 of the coordinates in the world coordinate system of each point of the tracking target object corresponding to the object region.

  The tracking unit 23 tracks the change in the position of the tracking target object in the real space during the shooting period, and obtains the locus of the position change of the tracking target object. For example, for each distance image, the tracking unit 23 obtains a position in the real space corresponding to a predetermined point in the object region detected from the distance image, and arranges the position in the order of generation of the distance image, thereby tracking target Trajectory data representing the trajectory of the position change of the object is generated. For example, the tracking unit 23 can set the tip of the hand that is the tracking target object or the tip of the gripping target object held by the hand as the predetermined point. For example, based on the positional relationship between the world coordinate system and the instructor, the tip of the hand or the tip of the object to be grasped that the hand holds is Yw in the world coordinate system in the real space region corresponding to the object region. Suppose that it is assumed to correspond to the maximum coordinate value along the axis. In this case, the tracking unit 23 uses, for each distance image, the world coordinate system of the point having the maximum coordinate value along the Yw axis in the real space region corresponding to the object region detected from the distance image. And the corresponding coordinates in the distance sensor coordinate system may be included in the trajectory data. Alternatively, the tracking unit 23 is within a predetermined range (for example, 5 cm to 10 cm) along the Yw axis direction from the point corresponding to the maximum value in the Yw axis direction in the real space area corresponding to the object area. The center of gravity of a certain partial area may be set as a predetermined point. Also in this case, the tracking unit 23 may include the coordinates of the predetermined point corresponding to each distance image in the world coordinate system and the corresponding coordinates in the distance sensor coordinate system in the trajectory data.

  When a plurality of object regions are detected from each distance image, the tracking unit 23 specifies an object region corresponding to the same tracking target object by applying a tracking process such as an optical flow to each object region. Also good. Moreover, when the tracking target object is a hand, it is not necessary to assume that the right hand and the left hand intersect each other. Therefore, the tracking unit 23 determines that the object area included in the range on the distance image corresponding to the left side of the center of the work table is the object area including the left hand, and corresponds to the right side of the center of the work table. The object region included in the range on the distance image to be determined may be determined as the object region including the right hand. Then, the tracking unit 23 generates trajectory data for each tracking target object.

  FIG. 6 is a diagram illustrating an example of trajectory data. The trajectory data 600 includes a hand ID 602 indicating whether the tracking target object is the right hand or the left hand, a coordinate 603 of the tip of the object area in the distance sensor coordinate system, and a world coordinate system for each time stamp 601 indicating the acquisition time of the distance image. Including the coordinates 604 of the tip of the object region.

  The tracking unit 23 stores the trajectory data generated for each tracking target object in the storage unit 14.

  The gripping determination unit 24 obtains a period during which the tracking target object is gripping the gripping target object for each distance image based on the trajectory data, the gripping object region, and the height information in the gripping object region.

  When the tracking target object grips the gripping target object, the tracking target object is assumed to move to a position overlapping the gripping target object. Furthermore, after the tracking target object grips the gripping target object, the gripping target object disappears from the area where the gripping target object was originally placed, so in that area, the distance from the distance sensor 11, that is, the work table It is assumed that the height from the surface changes. Therefore, the grip determination unit 24 refers to the trajectory data and determines whether or not the grip object region and the tracking target object overlap. The grip determination unit 24 then sets the height of the object in the gripping object region from the surface of the work table before and after the tracking target object overlaps the gripping object region, that is, before and after the trajectory of the tracking target object passes through the gripping object region. It is determined whether or not has changed. For example, the gripping determination unit 24 calculates the highest position from the surface of the work table in the gripping object region or the average value of the height from the surface of the worktable in the gripping object region. The height from the surface of the workbench. In the state where the gripping target object is not gripped, the gripping determination unit 24 has a height that is lower than the height before the tracking target object overlaps the gripping object region by a predetermined value or more. Then, the grip determination unit 24 determines that the grip target object is gripped. The predetermined value is set to 1 cm to 3 cm, for example. The predetermined value may be set according to the actual size of the object to be grasped. When there are a plurality of gripping target objects, a predetermined value corresponding to the size of the gripping target object may be set for each gripping target object in association with the gripping object region corresponding to the gripping target object.

  FIG. 7 is an explanatory diagram of an outline of grip determination. In this example, four gripping object regions 701 to 704 are set on the surface 100 of the work table. A locus 705 represents the locus of the tracking target object during the shooting period. In this example, the trajectory 705 overlaps the gripping object areas 702 and 704. For this reason, there is a possibility that the gripping target object placed in the gripping object region 702 or the gripping target object placed in the gripping object region 704 is gripped by the tracking target object. Therefore, for the gripping object regions 702 and 704, it is determined whether or not the height of the object in the gripping object region from the surface 100 of the work table has changed before and after the trace of the tracking target object overlaps the gripping object region. The In this example, with respect to the gripping object region 704, the height of the object in the gripping object region 704 from the surface 100 of the work table changes before and after the trace of the tracking target object overlaps the gripping object region. Therefore, it is determined that the gripping target object placed in the gripping object region 704 has been gripped.

  In addition, the gripping determination unit 24 once again determines that the tracking target object has been gripped once, after the trajectory of the tracking target object overlaps the gripping object region corresponding to the gripping target object. Changes in the height of the object from the surface of the work table may be examined. The grip determination unit 24, when the grip target object is gripped, has a height higher than a predetermined value by the height after the tracking target object overlaps the grip object area than the height before the trace target object overlaps the grip object area. If so, the grip determination unit 24 may determine that the grip target object is placed. In this case, the period from when it is determined that the gripping design object is gripped by the tracking target object to when it is determined that the gripping target object is placed is the period during which the gripping target object is gripped by the tracking target object It becomes.

  FIG. 8 is an operation flowchart of the grip determination process. The grip determination unit 24 executes grip determination processing according to this operation flowchart for each tracking target object.

  The grip determination unit 24 acquires the position of the tracking target object in the world coordinate system at the first shooting time from the trajectory data. Further, the gripping determination unit 24 sets a flag Grabbed (m) (m = 1, 2,..., N, N is the total number of gripping target objects) indicating whether or not each gripping target object is gripped. Set to False, which represents the state of not being gripped. Furthermore, the grip determination unit 24 sets a flag HandInArea indicating whether or not the tracking target object has entered the gripping object region to False indicating that the tracking target object is not in any gripping object region (step S101).

  The grip determination unit 24 determines whether or not the position of the tracking target object is located in any gripping object region (step S102). When the position of the tracking target object is located within the m-th gripping object region (step S102—Yes), the gripping determination unit 24 sets a flag ObjectNum representing the current gripping target object to m. Furthermore, the grip determination unit 24 sets the flag HandInArea to True indicating that the tracking target object has entered any gripping object region (step S103).

  On the other hand, when the position of the tracking target object is not in any gripping object region (step S102—No), the gripping determination unit 24 determines whether or not the flag HandInArea is True (step S104). When the flag HandInArea is True (step S104—Yes), it is assumed that the tracking target object has once entered any gripping object region and then exits the gripping object region. Therefore, the grip determination unit 24 starts timing and returns the flag HandInArea to False (step S105).

  The grip determination unit 24 determines whether or not the time is being measured. When the time is being measured, the height of the object in the mth gripping object area indicated by ObjectNum from the work surface is the object in the gripping object area before the tracking target object enters the gripping object area. It is determined whether or not the height has changed from the height of the work table (step S106). For example, if the difference between the height before the tracking target object enters the gripping object area and the height after the tracking target object exits the gripping object area changes by a predetermined value or more, the gripping determination unit 24 changes the object in the gripping object area. It may be determined that the height from the surface of the work table has changed. Further, the gripping determination unit 24 may change the determination condition as to whether or not the height of the object in the gripping object region from the surface of the work table has changed according to whether Grabbed (ObjectNum) is False or True. . That is, when Grabbed (ObjectNum) is False, the gripping target object specified by ObjectNum is not gripped. In this case, as described above, if the height after the tracking target object overlaps the gripping object region is lower than the height before the tracking target object overlaps the gripping object region, the gripping determination unit 24 It is determined that the height of the object in the object area from the surface of the work table has changed. On the other hand, when Grabbed (ObjectNum) is True, the gripping target object specified by ObjectNum is gripped. In this case, as described above, if the height after the tracking target object overlaps the gripping object region is higher than the height before the tracking target object overlaps the gripping object region, the gripping determination unit 24 It is determined that the height of the object in the object area from the surface of the work table has changed.

  If the grip determination unit 24 is measuring time and the height has changed between before and after the tracking target object enters the gripped object region (step S106—Yes), the flag Grabbed (ObjectNum) Is determined to be False (step S107). If the flag Grabbed (ObjectNum) is False (step S107—Yes), the grip determination unit 24 rewrites the flag Grabbed (ObjectNum) to True indicating that the gripping target object is being gripped (step S107). S108). On the other hand, when Grabbed (ObjectNum) is True (step S107—No), the grip determination unit 24 rewrites the flag Grabbed (ObjectNum) to False (step S109).

  On the other hand, if it is not timed in step S106 or if the height does not change before and after the tracking target object enters the gripping object region (step S106-No), the gripping determination unit 24 performs a predetermined time from the start of timekeeping. It is determined whether or not elapses (step S110). The predetermined time is set to, for example, a time required for the tracking target object to grip the gripping target object placed on the work table, for example, 3 to 5 seconds. When a predetermined time has elapsed from the start of time measurement (step S110-Yes), the grip determination unit 24 ends time measurement and resets the time measurement time to zero. In addition, the gripping determination unit 24 sets the flag ObjectNum to NULL that does not represent any gripping target object (step S111).

  After step S108, S109, or S111, the grip determination unit 24 stores Grabbed (ObjectNum) in association with the current shooting time for all ObjectNum (1 ≦ ObjectNum ≦ N) in the storage unit 14 (step S112). .

  Thereafter, the grip determination unit 24 determines whether or not the trajectory data has been completed (step S113). If the trajectory data has not ended (step S113-No), the gripping determination unit 24 acquires the position of the tracking target object in the world coordinate system at the next shooting time from the trajectory data (step S114). Then, the grip determination unit 24 repeats the processes after step S102. On the other hand, when the trajectory data is completed, that is, when the positions of the tracking target objects have been acquired for all shooting times included in the trajectory data (step S113-Yes), the grip determination unit 24 performs the grip determination process. Exit.

  As described above, information indicating the position of the tracking target object in the real space and whether or not any gripping target object is gripped at the time of capturing each distance image during the capturing period is obtained.

(Projection period)
Next, processing in the projection period will be described. In the projection period, the control unit 15 places the projector 13 at a position on the projection plane in the real space corresponding to the object region obtained from the distance image according to the photographing order of each distance image obtained during the photographing period. To project an image of the tracking target object. Further, the control unit 15 projects an image indicating the gripping target object by the projector 13 during a part of the projection period corresponding to a period during which any gripping target object is gripped during the imaging period. Project onto the surface.

  The display area calculation unit 25 calculates a display area representing the range of the image of the tracking target object on the display surface of the projector 13 when the projector 13 projects the image of the tracking target object on the projection plane. Further, the display area calculation unit 25 displays a second display area for displaying an image indicating the gripping target object on the display surface of the projector 13 while any gripping target object is gripped by the tracking target object. calculate.

  Note that the display area calculation unit 25 executes the same process for each distance image during the shooting period, and therefore, the process for one distance image will be described below.

In the present embodiment, the projector 13 projects the image of the tracking target object so that the size of the image of the tracking target object on the projection surface matches the actual size of the tracking target object. Therefore, the display area calculation unit 25, for each point of the tracking target object corresponding to the object area, out of the coordinates (X _W , Y _W , Z _W ) in the world coordinate system representing the position of the point in real space. The coordinate value Z _W in the Zw axis direction is replaced with the height Z _W ′ of the projection plane. Thereby, the tracking target object is virtually shifted onto the projection plane. The projection plane can be the surface of the work table (Z _w ′ = 0) or the surface of the work target object placed on the work table. When the projection plane is the surface of the work target object, the display area calculation unit 25 corresponds to the pixel in which the work target object is shown in the distance image obtained by photographing the work target object as the height of the projection plane. What is necessary is just to calculate the coordinate of the Zw-axis direction of the world coordinate system to perform according to (1) Formula and (2) Formula.

The display area calculation unit 25 calculates the coordinates on the display surface of the projector 13 corresponding to the coordinates (X _W , Y _W , Z _W ′) of each point of the tracking target object after the shift in the world coordinate system. calculate. Here, the positional relationship between the projector 13 and the projection surface is also known. Therefore, the coordinates of an arbitrary point on the world coordinate system can be converted into coordinates on the projector coordinate system based on the projector 13 by affine transformation. Note that the projector coordinate system is, for example, similar to the distance sensor coordinate system, with the center of the display surface of the projector 13 as the origin, and the two orthogonal to the optical axis direction of the projector 13 and the plane orthogonal to the optical axis direction. One direction can be a coordinate system with each axis as an axis.

Display area calculating unit 25, for each point of the tracked object, the coordinates in the world coordinate system of the point _{(X W, Y W, Z} W ') , was prepared in accordance with the following equation, the coordinates of the projector coordinate system (X _P , Y _P , Z _P ).
Here, R _WP is a rotation matrix that represents the amount of rotation included in the affine transformation from the world coordinate system to the projector coordinate system, and t _WP is a translation vector that represents the amount of translation included in the affine transformation. . PLocX, PLocY, and PLocZ are the coordinates of the center of the display surface of the projector 13 in the Xw-axis direction, the Yw-axis direction, and the Zw-axis direction of the world coordinate system, that is, the coordinates of the origin of the projector coordinate system. PRotX, ProtY, and ProtZ represent rotation angles in the optical axis direction of the projector 13 with respect to the Xw axis, the Yw axis, and the Zw axis, respectively. In this embodiment, since the projector 13 is mounted vertically downward, the rotation matrix R _WP has a value of “1” only for the diagonal component and a value of “0” for the other components. Is a matrix with

Further, the display area calculation unit 25 displays the display surface of the projector 13 corresponding to the coordinates (X _P , Y _P , Z _P ) of each point of the tracking target object in the projector coordinate system according to the pinhole model. The above coordinates (x _p , y _p ) are calculated according to the following equation.
Here, fp represents the focal length of the projector 13, and PW and PH represent the number of horizontal pixels and the number of vertical pixels on the display surface, respectively. PfovD represents the diagonal viewing angle of the projector 13.

  Further, the display area calculation unit 25 refers to the flag Grabbed (ObjectNum) stored in the storage unit 14 in association with the shooting time of the distance image for each distance image, and holds the target object represented by ObjectNum. It is determined whether or not is held. Then, when the gripping target object represented by ObjectNum is gripped, the display area calculation unit 25 selects the second display area on the display area of the projector 13 corresponding to the gripping object area corresponding to the gripping target object. It calculates according to (5) Formula and (6) Formula.

  The display area calculation unit 25 notifies the projection control unit 26 of the display area and the second display area.

  The projection control unit 26 projects the image of the tracking target object on the projection surface by displaying the image of the tracking target object in the display area on the display surface of the projector 13. At that time, for example, the projection control unit 26 sets the value of each pixel in the display area to a predetermined value set in advance.

  However, since the projection image by the projector 13 is projected as a translucent two-dimensional image on the projection surface, it is difficult for the user to recognize the difference between the height of the actual object on the projection surface and the height of the tracking target object. . For this reason, it may be difficult for the user to identify whether the tracking target object is touching an object on the projection plane or whether the tracking target object is in the air.

Therefore, according to the modification, the projection control unit 26 displays the display area of the projector 13 according to the distance from the distance sensor 11 to the tracking target object or the height from the projection surface to the tracking target object at the time of imaging. The value of the pixel inside may be adjusted. For example, when the projector 13 displays the tracking target object in gray scale, the projection control unit 26 may determine the luminance value of each pixel in the display area according to the following equation.
Here, Z _w represents the height (for example, a value in mm) of the point of the tracking target object corresponding to the pixel of interest in the display area, and L _v is the luminance of the pixel of interest. Represents a value. Alternatively, the projection control unit 26 may determine the luminance value of each pixel in the display area according to the following equation instead of the equation (7).
Here, Z _max represents an assumed maximum value of the height of the tracking target object with respect to the projection plane.

Alternatively, the projection control unit 26 may set the value of the pixel in the display area on the display surface of the projector 13 according to the following equation.
Here, R, G, and B are the values of the red component, green component, and blue component of the pixel, respectively. A is an alpha value representing transparency. Z _w represents the height of the point of the tracking target object corresponding to the pixel from the surface of the work table in the world coordinate system. Z _aveW is the world coordinate system of each point of the tracking target object existing within a predetermined range (for example, 150 mm) along the Yw axis direction from the tip of the object region in the Yw axis direction, that is, the tip of the tracking target object. The average value of the height from the surface of the work table is shown. Z _{surface W} is the average value in the Xw-axis direction and the Yw-axis direction of each point in the object region that is within a predetermined range (for example, 150 mm) along the Yw-axis direction from the tip in the Yw-axis direction of the object region. This represents the height of the projection plane (for example, the surface of the work target object) in the average value. Α and β are constants, for example, α = 1.2 and β = 128.

  By changing the color or brightness of the image of the tracking target object projected by the projector 13 at the time of projection according to the height from the projection surface to the actual tracking target object at the time of imaging, the user can It becomes easy to recognize the actual height of the tracking target object. Therefore, the user can easily identify whether the tracking target object touches an object on the projection plane or whether the tracking target object is in the air based on the image of the tracking target object. It becomes.

  Furthermore, the projection control unit 26 displays the display surface of the projector 13 corresponding to the gripping target object in a part of the projection period corresponding to the period during which any gripping target object is gripped during the imaging period. An image indicating the object to be grasped is displayed in the upper second display area. For example, the projection control unit 26 uses a gripping object image stored in the storage unit 14 as an image indicating the gripping target object. Thereby, the image of the object to be grasped is projected on the projection surface. Further, the projection control unit 26 is information indicating that the gripping target object is being gripped at a position away from the second display area by a predetermined offset in a predetermined direction, for example, “in use”. Display a string. Thereby, the information is projected on the projection surface.

  FIG. 9A is a schematic diagram of the projection plane when no gripping target object is gripped. On the other hand, FIG. 9B is a schematic diagram of the projection plane when any gripping target object is gripped by the tracking target object. As shown in FIG. 9A, when no gripping target object is gripped, only the image 901 of the tracking target object is projected onto the projection plane 900. On the other hand, as shown in FIG. 9B, when any gripping target object is gripped by the tracking target object, not only the tracking target object image 901 but also the gripping target object is gripped. An image 902 indicating the gripping target object is also projected. In this example, an image 901 of the tracking target object and an image 902 indicating the gripping target object are projected at different positions. In particular, during use of the gripping target object, the gripping object region and the tracking target object image 901 on which the image 902 indicating the gripping target object is projected do not overlap. Therefore, the visibility of the image 902 indicating the gripping target object is improved. Further, since the character string 903 “in use” is projected at a position a predetermined distance from the image 902 of the gripping target object, the user can easily identify the gripping target object being gripped.

  FIG. 10 is an operation flowchart of the projection processing executed by the control unit 15.

  The gripping object region setting unit 20 sets the gripping object region including the gripping target object to the XwYw plane of the world coordinate system for each gripping target object, and sets the height of each point in the gripping object region from the work table surface. Calculate (step S201).

  In addition, for each gripping target object, the gripping object region setting unit 20 cuts out the region on the image obtained by the camera 12 corresponding to the gripping object region from the image as the gripping target object image, and stores it in the storage unit 14. Save (step S202).

  The object region detection unit 21 detects an object region in which the tracking target object is shown on each distance image obtained from the distance sensor 11 obtained during the photographing period (step S203). Then, the object region detection unit 21 notifies the real space position calculation unit 22 of the object region.

  The real space position calculation unit 22 calculates the coordinates in the world coordinate system of the point of the tracking target object shown in the pixel for each pixel in the object region for each distance image, so that the real space of the tracking target object is calculated. The position at is calculated (step S204). Then, the real space position calculation unit 22 notifies the tracking unit 23 and the display region calculation unit 25 of the coordinates in the world coordinate system of each point of the tracking target object corresponding to the object region.

  The tracking unit 23 generates trajectory data representing a trajectory of a change in the position of the tracking target object in the real space from the position of the tracking target object in the real space for each distance image during the imaging period (step S205). The tracking unit 23 stores the trajectory data in the storage unit 14.

  The gripping determination unit 24 specifies a period during which the gripping target object is gripped by the tracking target object for each gripping target object based on the trajectory data, the gripping object region, and the height of each point in the gripping object region. (Step S206).

  For each distance image obtained during the imaging period, the display area calculation unit 25 determines, for each point of the tracking target object detected from the distance image, the height of the point from the surface of the work table in the world coordinate system. Is replaced with the height of the projection plane. Then, for each distance image obtained during the imaging period, the display area calculation unit 25 calculates the coordinates of the corresponding pixel on the display surface of the projector 13 corresponding to each point of the tracking target object detected from the distance image. Thus, the display area on the display surface is calculated (step S207).

  Further, the display area calculation unit 25 calculates a second display area on the display surface of the projector 13 corresponding to the gripping object area of the gripping target object during the period in which the gripping target object is gripped by the tracking target object. (Step S208). Then, the display area calculation unit 25 notifies the projection control unit 26 of the display area and the second display area.

  For each pixel in the display area on the display surface of the projector 13, the projection control unit 26 sets the color or luminance of the pixel according to the height of the point of the tracking target object corresponding to the pixel from the projection surface. (Step S209). Then, the projection control unit 26 projects the image of the tracking target object by displaying the image of the tracking target object on the display surface of the projector 13 so that the color or luminance of each pixel in the display area becomes a set value. Projecting onto a surface (step S210). Furthermore, the projection control unit 26 applies the gripping target object to the second display area of the display surface of the projector 13 during the projection period corresponding to the period during which the gripping target object is gripped. An image showing is displayed. The projection control unit 26 displays a character string such as “in use” at a position a predetermined distance from the second display area. Thereby, the projection control unit 26 causes the projector 13 to project an image indicating the object to be grasped and a character string thereof onto the projection plane (step S211). Then, the control unit 15 ends the projection process. Note that the control unit 15 may store the distance image in the storage unit 14 in association with the shooting time during the imaging period, and may also execute the processes of steps S201 to S206 during the projection period.

  As described above, the projection apparatus obtains the position of the tracking target object in the real space from each distance image obtained during the imaging period. The projection apparatus detects the object to be tracked by the change of the height of the object in the grasped object area from the surface of the work table before and after the locus of the object to be tracked in the real space passes through the grasped object area. It is determined whether or not it is gripped. Then, the projection apparatus has the gripping target object at the original position where the gripping target object was placed in the period during the projection period corresponding to the period during which the gripping target object is gripped by the tracking target object. An image showing is projected. As a result, the image indicating the gripping target object gripped by the tracking target object is projected so as not to overlap the work target object and the tracking target object, so that the projection device increases the visibility of the image of the gripping target object. Can be improved.

  According to the modification, the projection apparatus may determine whether or not the grip target object or another object exists in the region on the projection surface when projecting the image of the grip target object. The projection device may change the projection position of the image indicating the gripping target object or display a frame surrounding the gripping object region as an image indicating the gripping object according to the determination result.

FIG. 11 is a functional block diagram of the control unit 35 according to this modification. The control unit 35 includes a gripping object region setting unit 20, an object region detection unit 21, a real space position calculation unit 22, a tracking unit 23, a gripping determination unit 24, a display region calculation unit 25, and a projection control unit 26. And a feature extraction unit 27 and a collation unit 28.
Each of these units included in the control unit 35 is implemented as a functional module realized by a computer program executed on a processor included in the control unit 35, for example. Alternatively, one or a plurality of integrated circuits that realize the functions of these units may be mounted on the projection apparatus 1 separately from the control unit 35.

  The control unit 35 according to this modified example is different from the control unit 15 according to the above-described embodiment in that it includes a feature extraction unit 27 and a collation unit 28 and a part of the processing of the projection control unit 26. Therefore, hereinafter, the projection control unit 26, the feature extraction unit 27, the collation unit 28, and related parts will be described. For other components of the projection apparatus, refer to the description of the corresponding components in the above embodiment.

  The feature extraction unit 27 detects each gripping target from an image obtained by the camera 12 or a distance image obtained by the distance sensor 11 when each gripping target object is placed on the surface of the workbench during the shooting period. Extract object features. For example, the feature extraction unit 27 uses, as the feature amount, the number of pixels included in the candidate area detected as the area in which the gripping target object is captured from the distance image, the length of the candidate area in the longitudinal direction, or the candidate The height distribution from the surface of the work table in the area is calculated. Alternatively, the feature extraction unit 27 converts the RGB color system value from the RGB color system value to the HLS color system value for each pixel included in the region on the image corresponding to the grasped object region as the feature value, and each hue value. May be obtained. Alternatively, the feature extraction unit 27 may use a region on the image itself corresponding to the grasped object region as a feature amount. The feature extraction unit 27 may calculate a plurality of feature amounts. Then, the feature extraction unit 27 stores the obtained feature amount in the storage unit 14 together with identification information (for example, the above ObjectNum) for identifying the gripping target object. In the following, for the sake of convenience, the feature amount of the gripping target object obtained during the shooting period is referred to as a registered feature amount.

  Furthermore, when projecting an image of the gripping target object gripped by the tracking target object, the feature extraction unit 27 uses the gripping target object from the distance image obtained by the distance sensor 11 or the image obtained by the camera 12. The feature amount similar to the above is calculated from the gripped object region corresponding to. Then, the feature extraction unit 27 passes the obtained feature amount to the collation unit 28. Hereinafter, for convenience, the feature amount of the gripping target object obtained when projecting an image indicating the gripping target object gripped by the tracking target object is referred to as a matching feature amount.

  The collation unit 28 determines whether or not an object exists in the gripping object region corresponding to the gripping target object when projecting an image of the gripping target object gripped by the tracking target object during the projection period. . For example, the collation unit 28 extracts pixels whose height from the work table surface is a predetermined value (for example, 1 cm to 2 cm) or more in the area on the distance image corresponding to the grasped object area. When the number of extracted pixels is equal to or greater than a predetermined number (for example, 1/4 to 1/2 of the number of pixels included in the region on the distance image corresponding to the gripping object region), the matching unit 28 It is determined that an object exists in the area. Note that the region on the distance image corresponding to the gripping object region is obtained by applying the inverse transformations of the equations (1) and (2) to the points corresponding to the respective corners of the gripping object region.

  When the collation unit 28 determines that an object exists in the gripping object region, the collation feature 28 collates the feature value for collation for the gripping target object with the registered feature amount for the gripping target object. When the difference between the matching feature value and the registered feature value is within a predetermined allowable range, the matching unit 28 determines that the object existing in the gripping object region is the same as the gripping target object gripped by the tracking target object. Judge that there is. For example, when each feature amount is an image of a gripping target object, the matching unit 28 calculates a normalized cross-correlation value by pattern matching between the images of the two gripping target objects. If the normalized cross-correlation value is equal to or greater than a predetermined threshold, the matching unit 28 determines that the object existing in the gripping object region is the same as the gripping target object gripped by the tracking target object. Good.

  The collation unit 28 determines whether or not an object is present in the gripping object area, and whether or not the object present in the gripping object area is the same as the gripping target object gripped by the tracking target object The result is notified to the projection control unit 26.

  When projecting an image of a gripping target object that is gripped by the tracking target object during the projection period, the projection control unit 26, if no object exists in the gripping object region corresponding to the gripping target object, Similar to the embodiment, an image of a gripping target object is projected onto the gripping object region.

  On the other hand, if an object exists in the gripping object region corresponding to the gripping target object and the object is different from the gripping target object gripped by the tracking target object, the projection control unit 26 displays the image of the gripping target object. Is shifted to a position deviating from the gripping object region. Therefore, for example, the projection control unit 26 shifts the second display area corresponding to the grasped object area by the length of the second display area along the horizontal direction or the vertical direction.

  If the object exists in the gripping object area corresponding to the gripping target object and the object is the same as the gripping target object gripped by the tracking target object, the projection control unit 26 Project a frame along the outline of the projection onto the projection plane.

  FIG. 12A is a schematic diagram of the projection plane when no object is present in the gripping object region. FIG. 12B is a schematic diagram of the projection plane when an object different from the gripping target object being gripped exists in the gripping object region. FIG. 12C is a schematic diagram of the projection plane when the same gripping target object as the gripping target object being gripped exists in the gripping object region.

  As shown in FIG. 12A, when no object exists in the gripping object region 1200, an image 1201 indicating the gripping target object being gripped is projected onto the gripping object region 1200. Also, as shown in FIG. 12B, when an object 1202 that is different from the gripped target object is present in the gripped object region 1200, an image 1201 indicating the gripped target object is The image is projected at a position shifted from the gripping object region 1200 in the horizontal direction. Then, as shown in FIG. 12C, when the grip target object 1203 is the same as the grip target object being gripped, the frame line 1204 of the grip object area 1200 is projected. .

  As described above, in this modification, the projection position of the image indicating the gripping target object or the type of image indicating the gripping object is changed according to the presence or absence of the object in the gripping object region and the type of the object at the time of projection. Is done. Therefore, the projection apparatus can improve the visibility of an image indicating the gripping target object even when some object exists in the gripping object region during projection.

  The gripping target object may be a part that is assembled to the work target object. In this case, once the gripping target object is gripped by the tracking target object, the gripping target object is not returned to the original gripping object area but is moved onto the work target object. Therefore, according to another modification, once the gripping determination unit 24 determines that any one of the gripping target objects is gripped by the tracking target object, thereafter, based on the trajectory data, the position and work of the tracking target object are determined. The position where the target object overlaps is specified. Note that the position and range of the work target object in the real space are stored in the storage unit 14 in advance. Then, the gripping determination unit 24 increases the height after passing through a predetermined value corresponding to the height of the gripping target object with respect to the height before the tracking target object passes through at each specified position. In this case, it may be determined that the gripping target object is assembled at the position. In this case, after the tracking target object passes through the specified position, the grip determination unit 24 determines that the grip target object is not gripped by the tracking target object. Then, the projection control unit 26 may cause the projector 13 to project an image of the gripping target object at a position where the gripping target object is assembled.

  According to still another modified example, the projection apparatus performs a gripping operation from an image obtained by the camera 12 immediately before the gripping determination unit 24 determines that any tracking target object is gripped during the imaging period. An object image may be generated. In this case, the control unit 15 performs the same processing as that of the gripping object region setting unit 20, specifies the region on the image corresponding to the gripping object region for the gripping target object determined to be gripped, and determines the region. A grasped object image may be generated by cutting out from the image.

  According to still another modification, the projection apparatus may always project an image of the gripping target object during the projection period for the gripping target object gripped by the tracking target object at any time during the imaging period. . In this case, the projection control unit 26 determines that the grip target object whose flag Grabbed () is True at any point in the second display area on the display surface of the projector 13 during the projection period. It is sufficient to display an image indicating In this modification as well, the projection control unit 26 indicates the gripping state such as “in use” to the projector 13 only in the period during the projection period corresponding to the period in which the gripping target object is gripped by the tracking target object. A character string may be projected.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

The following supplementary notes are further disclosed regarding the embodiment described above and its modifications.
(Appendix 1)
A storage unit for storing a gripping object region including a gripping target object in real space;
In a predetermined imaging period, an object region detection unit that detects an object region in which the target object is captured in a distance image obtained from a distance sensor and representing a distance between the target object and the distance sensor as a pixel value;
In the distance image, for each pixel corresponding to the target object in the object region, a real space position calculation unit that calculates a position in real space of the target object reflected in the pixel;
A tracking unit that generates trajectory data representing a trajectory of the target object in real space based on a position of the target object in real space corresponding to the distance image;
Based on a change between the height of the gripping target object in the gripping object region and the height of the object after the trajectory passes through the gripping object region, the gripping target object is gripped by the target object. A gripping determination unit that identifies a period of 1;
In a second period of the predetermined projection period corresponding to the first period, an image of the target object and an image indicating the grip target object gripped by the target object are projected to the projector A projection control unit for projecting to different positions above,
A projection apparatus.
(Appendix 2)
The projection apparatus according to attachment 1, wherein the storage unit stores an image representing the gripping target object as an image indicating the gripping target object.
(Appendix 3)
From the distance image obtained when the gripping target object is not gripped by the target object, a region where the gripping target object is captured is detected, and the position of each point in the region is calculated in the real space An image in which the gripping target object is represented on the image obtained by setting the gripping object region corresponding to the gripping target object and capturing the gripping target object obtained by the camera. The projection apparatus according to attachment 2, further comprising: a gripping object region setting unit.
(Appendix 4)
The display surface of the projector corresponding to the gripping target object is calculated by calculating a position on the display surface of the projector corresponding to the gripping object region corresponding to the gripping target object gripped by the target object. It further has a display area calculation unit for calculating the upper display area,
The projection control unit displays an image indicating the gripping target object on the projection plane by displaying an image indicating the gripping target object in the display area on the display surface of the projector in the second period. The projection apparatus according to appendix 2 or 3, which projects.
(Appendix 5)
The storage unit further stores a registered feature amount representing a feature of the gripping object,
The gripping object corresponding to the gripping target object gripped by the target object on the distance image obtained by the distance sensor or on the image obtained by the camera in the second period A feature extraction unit that calculates a feature value for matching that represents a feature of an object reflected in an area corresponding to the area;
Whether the object and the gripping target object gripped by the target object are the same by comparing the matching feature amount with the registered feature amount of the gripping target object gripped by the target object And a verification unit for determining whether or not
When the object and the object to be gripped held by the target object are the same, the projection control unit is configured to hold the grip in the display area on the display surface of the projector in the second period. The projection apparatus according to appendix 4, wherein an image of a frame line surrounding the object region is displayed as an image indicating the gripping target object.
(Appendix 6)
When the object and the object to be gripped held by the target object are different from each other, the projection control unit causes the projector to position the image of the object to be gripped so as not to overlap the object in the second period. The projection device according to appendix 5, wherein
(Appendix 7)
The verification unit includes the object in the gripped object region corresponding to the gripping target object gripped by the target object based on the distance image obtained by the distance sensor in the second period. Whether or not
The projection apparatus according to appendix 5 or 6, wherein the projection control unit causes the projector to project an image representing the gripping target object onto the gripping object region when the object does not exist in the gripping object region.
(Appendix 8)
The projection control unit causes the projector to project information indicating that the gripping target object is gripped by the target object on the projection plane by the projector, according to any one of appendices 1 to 7. The projection device described.
(Appendix 9)
During a predetermined imaging period, an object region in which the target object is captured is detected in a distance image obtained from a distance sensor and representing a distance between the target object and the distance sensor as a pixel value;
In the distance image, for each pixel corresponding to the target object in the object region, calculate the position in real space of the target object reflected in the pixel,
Based on the position of the target object in real space corresponding to the distance image, generating trajectory data representing the trajectory of the target object in real space;
Based on the change between the height of the gripping target object in the gripping object region including the gripping target object in real space and the height of the object after the trajectory passes through the gripping object region, the gripping target object is Identify a first period of time being held by the target object;
In a second period of the predetermined projection period corresponding to the first period, an image of the target object and an image indicating the grip target object gripped by the target object are projected to the projector Project to different positions above,
A projection method comprising:
(Appendix 10)
During a predetermined imaging period, an object region in which the target object is captured is detected in a distance image obtained from a distance sensor and representing a distance between the target object and the distance sensor as a pixel value;
In the distance image, for each pixel corresponding to the target object in the object region, calculate the position in real space of the target object reflected in the pixel,
Based on the position of the target object in real space corresponding to the distance image, generating trajectory data representing the trajectory of the target object in real space;
Based on the change between the height of the gripping target object in the gripping object region including the gripping target object in real space and the height of the object after the trajectory passes through the gripping object region, the gripping target object is Identify a first period of time being held by the target object;
In a second period of the predetermined projection period corresponding to the first period, an image of the target object and an image indicating the grip target object gripped by the target object are projected to the projector Project to different positions above,
Projection computer program for causing a computer to execute the above.

DESCRIPTION OF SYMBOLS 1 Projection apparatus 11 Distance sensor 12 Camera 13 Projector 14 Memory | storage part 15,35 Control part 100 Work surface 101 Tracking target object 102 Work target object 103 Grasp target object 20 Grasp object area setting part 21 Object area detection part 22 Real space position calculation 23 Tracking unit 24 Grasping determination unit 25 Display area calculation unit 26 Projection control unit 27 Feature extraction unit 28 Verification unit

Claims (8)

A storage unit for storing a gripping object region including a gripping target object in real space;
In a predetermined imaging period, an object region detection unit that detects an object region in which the target object is captured in a distance image obtained from a distance sensor and representing a distance between the target object and the distance sensor as a pixel value;
In the distance image, for each pixel corresponding to the target object in the object region, a real space position calculation unit that calculates a position in real space of the target object reflected in the pixel;
A tracking unit that generates trajectory data representing a trajectory of the target object in real space based on a position of the target object in real space corresponding to the distance image;
Based on a change between the height of the gripping target object in the gripping object region and the height of the object after the trajectory passes through the gripping object region, the gripping target object is gripped by the target object. A gripping determination unit that identifies a period of 1;
In a second period of the predetermined projection period corresponding to the first period, an image of the target object and an image indicating the grip target object gripped by the target object are projected to the projector A projection control unit for projecting to different positions above,
A projection apparatus.   The projection apparatus according to claim 1, wherein the storage unit stores an image representing the gripping target object as an image indicating the gripping target object.   From the distance image obtained when the gripping target object is not gripped by the target object, a region where the gripping target object is captured is detected, and the position of each point in the region is calculated in the real space An image in which the gripping target object is represented on the image obtained by setting the gripping object region corresponding to the gripping target object and capturing the gripping target object obtained by the camera. The projection apparatus according to claim 2, further comprising a gripping object region setting unit. The display surface of the projector corresponding to the gripping target object is calculated by calculating a position on the display surface of the projector corresponding to the gripping object region corresponding to the gripping target object gripped by the target object. It further has a display area calculation unit for calculating the upper display area,
The projection control unit displays an image indicating the gripping target object on the projection plane by displaying an image indicating the gripping target object in the display area on the display surface of the projector in the second period. The projection device according to claim 2, wherein the projection device projects. The storage unit further stores a registered feature amount representing a feature of the gripping object,
The gripping object corresponding to the gripping target object gripped by the target object on the distance image obtained by the distance sensor or on the image obtained by the camera in the second period A feature extraction unit that calculates a feature value for matching that represents a feature of an object reflected in an area corresponding to the area;
Whether the object and the gripping target object gripped by the target object are the same by comparing the matching feature amount with the registered feature amount of the gripping target object gripped by the target object And a verification unit for determining whether or not
When the object and the object to be gripped held by the target object are the same, the projection control unit is configured to hold the grip in the display area on the display surface of the projector in the second period. The projection apparatus according to claim 4, wherein an image of a frame line surrounding an object region is displayed as an image indicating the gripping target object.   6. The projection control unit according to claim 1, wherein, in the second period, the projector projects information indicating that the gripping target object is gripped by the target object onto the projection plane. The projection device according to one item. During a predetermined imaging period, an object region in which the target object is captured is detected in a distance image obtained from a distance sensor and representing a distance between the target object and the distance sensor as a pixel value;
In the distance image, for each pixel corresponding to the target object in the object region, calculate the position in real space of the target object reflected in the pixel,
Based on the position of the target object in real space corresponding to the distance image, generating trajectory data representing the trajectory of the target object in real space;
Based on the change between the height of the gripping target object in the gripping object region including the gripping target object in real space and the height of the object after the trajectory passes through the gripping object region, the gripping target object is Identify a first period of time being held by the target object;
In a second period of the predetermined projection period corresponding to the first period, an image of the target object and an image indicating the grip target object gripped by the target object are projected to the projector Project to different positions above,
A projection method comprising: During a predetermined imaging period, an object region in which the target object is captured is detected in a distance image obtained from a distance sensor and representing a distance between the target object and the distance sensor as a pixel value;
In the distance image, for each pixel corresponding to the target object in the object region, calculate the position in real space of the target object reflected in the pixel,
Based on the position of the target object in real space corresponding to the distance image, generating trajectory data representing the trajectory of the target object in real space;
Based on the change between the height of the gripping target object in the gripping object region including the gripping target object in real space and the height of the object after the trajectory passes through the gripping object region, the gripping target object is Identify a first period of time being held by the target object;
In a second period of the predetermined projection period corresponding to the first period, an image of the target object and an image indicating the grip target object gripped by the target object are projected to the projector Project to different positions above,
Projection computer program for causing a computer to execute the above.