JP2017191351A - Work support system and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To support a worker so as to properly perform confirmation of a work result performed by the worker.SOLUTION: A user terminal 201 comprises: a storage device for storing a work space model of a work space including an attachment object body which is a work object present in a work space, a component model of a component to be attached to the attachment object body, and work manual data indicating a work procedure of one or more work steps; a depth sensor 204 for acquiring three-dimensional data of the work space which is viewed from the worker, in a period when the worker performs the work; and a central processing unit. The central processing unit outputs the work procedure of each work step, on the basis of the work manual data, compares the attachment object body and the model of the component related to the first work step, with three-dimensional data of the work space acquired by the depth sensor 204, then determines whether or not a finish condition of visual inspection confirmation work of the first work step, is satisfied, and when the finish condition is not satisfied, outputs a guidance for satisfying the finish condition.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a work support system that supports an operator by presenting information necessary for the operation to the operator in various operations such as product assembly and facility maintenance.

  In various operations such as product assembly and equipment maintenance, it is necessary for an operator to check whether the operation has been properly completed. At that time, it is necessary to provide support for confirming the necessary parts without overlooking them. In addition, the work result may be saved depending on the place where the work is performed or a photograph of the object, etc. In this case, too, the photograph of the place necessary for checking after the work is appropriately acquired. It is desirable to assist workers.

  In recent years, a technology called Augmented Reality, which is a technology that assists users in understanding the real world by presenting virtual information such as other images and characters at appropriate positions on images that reflect the real world, has attracted attention. (For example, Non-Patent Document 1). By using augmented reality, it is possible to present the position of an object to be worked in a three-dimensional manner on the work space that the worker is looking at. Since the position of the object presented in this way is also a place that needs to be confirmed, it can be used for the purpose of assisting the confirmation of the work result. Furthermore, when displaying the position of the object, three-dimensional display that guides the operator's line of sight to the place of the object is also possible.

  For example, in Patent Document 1 and Patent Document 2, the progress of work is determined by obtaining a difference between 3D CAD data or 3D target data and 3D data acquired by a 3D measuring instrument or a stereo image. The technology is described. Further, Patent Document 2 describes a technique for visualizing a difference between target data by superimposing and displaying target data and acquired three-dimensional data.

  Patent Document 3 describes a technique for comparing a photograph in a database with a newly acquired photograph and highlighting the difference.

  Patent Document 4 and Patent Document 5 describe a technique of changing the information to be superimposed on the real space according to the situation in the augmented reality technology in Non-Patent Document 1. For example, Patent Document 4 shows a technique for changing the content of information based on the position and direction of an object, and Patent Document 5 shows a technique for detecting a worker's hand and changing the display color and transparency depending on the presence or absence of the hand. Has been.

JP-A-9-133519 JP 2002-352224 A Japanese Patent No. 5740884 Japanese Patent Application No. 2009-55575 Japanese Patent Application No. 2010-269635

“Feature Augmented Reality”, Journal of Information Processing Society, Vol. 51, no. 4, 2010

  By using the technique of Non-Patent Document 1, it is possible to guide the operator's line of sight to a location or an object that needs to be confirmed. In addition, by using the techniques of Patent Document 1, Patent Document 2, and Patent Document 3, the operator can recognize the difference between the three-dimensional target data and the three-dimensional CAD data and the state of the object after the work is completed. Is possible.

  However, with these techniques, it is impossible for the worker to check which part of the specific part or target object is checked and which part is not checked, and to show the unchecked part to the worker.

  In the technique of Non-Patent Document 1, since the position of the object can be presented three-dimensionally on the work space that the worker is looking at, when the place to be confirmed can be identified, It is considered effective as a technique for guiding the line of sight to that point.

  However, when the distance between the worker and the object is short, a necessary portion may not be within the display area depending on the display device. In such a case, conversely, it may be difficult for the operator to grasp the situation.

  Further, in the technique of Non-Patent Document 1, a process for estimating the positional relationship between the worker and the work space is performed. However, if the process is not appropriately performed, there is a problem that information is not presented at all.

  In Patent Document 4 and Patent Document 5, the content of the information to be presented is changed based on the situation, but only the superimposed display on the real space is targeted, so that the above situation is sufficient. It cannot be said that it is a simple method.

  Accordingly, an object of the present invention is to provide a technique for assisting an operator so that confirmation of an operation result performed by the operator can be appropriately performed.

  Still another object of the present invention is to provide an operator with easy-to-understand information for supporting confirmation of work results.

  A work support system according to one embodiment of the present invention is a work that shows three-dimensional data of a work body to be attached and parts to be attached to the work body and work procedures of one or more work processes in the work space. A storage device that stores manual data, a three-dimensional data acquisition device that acquires three-dimensional data of a work space that is visible to the worker while the worker is working, and a central treatment device. The central processing unit is configured to output a work procedure of each work process based on work manual data stored in the storage means, and to perform a process related to the first work process stored in the storage means. The three-dimensional data of the attachment body and the part is compared with the three-dimensional data of the work space acquired by the three-dimensional data acquisition device, and it is determined whether or not the end condition of the visual confirmation work in the first work process is satisfied. A process and a process of outputting guidance for satisfying the end condition when the end condition is not satisfied in the determination are performed.

  In a preferred embodiment, the central processing unit further estimates the position and orientation of the three-dimensional data acquisition device and, based on the result of the estimation, 3 of the mounted body and the part related to the first work process. A process of matching the dimensional data with the three-dimensional data of the work space may be performed. The condition for ending the visual confirmation work may be that a difference between the three-dimensional data of the mounted body and the part related to the first work process and the three-dimensional data of the work space is not more than a predetermined threshold.

  In a preferred embodiment, the guidance for satisfying the end condition may indicate a viewpoint and a line-of-sight direction viewed by the worker.

  In a preferred embodiment, the viewpoint viewed by the worker may be a region that is in the three-dimensional data of the mounted body and the part related to the first work process but not in the three-dimensional data of the work space.

  In a preferred embodiment, the apparatus may further include a head mounted display worn by the operator. Guidance for satisfying the termination condition is displayed on the head mounted display in the work space that is visible to the worker through the head mounted display, the position of the attached body or the part viewed by the worker and the direction of the line of sight. A first display mode for displaying the position of the attached body or the part and the direction of the line of sight as seen by the worker in the CG image including the attachment body and the part on the head mounted display by computer graphics. It may be output in any one of the two display modes.

In a preferred embodiment, the guidance may be output in the second display mode when any one of (1) to (4) is satisfied.
(1) When the distance between the worker and the part is equal to or less than a predetermined value (2) When there is an object that shields or may shield the position of the mounted body or the part viewed by the worker (3) In the first display mode, when the display of the position of the attached body or the part and the direction of the line of sight seen by the worker does not fit in the head mounted display. (4) Even if the matching is performed, the first work When the deviation between the three-dimensional data of the mounted body and the part related to the process and the three-dimensional data of the work space is greater than or equal to a predetermined value

  In a preferred embodiment, the central processing unit may further output a work procedure of a second work process subsequent to the first work process when the end condition is satisfied in the determination.

  In a preferred embodiment, the central processing unit may further store the three-dimensional data of the work space in the storage device when the termination condition is satisfied in the determination.

  ADVANTAGE OF THE INVENTION According to this invention, an operator can be supported so that confirmation of the work result which an operator performs can be performed appropriately.

  Further, according to the present invention, information for supporting confirmation of the work result can be provided to the worker in an easily understandable manner.

An example of a hardware configuration of an information processing apparatus for realizing the work support system according to the first embodiment of the present invention Example of usage scene of work support system in this embodiment An example of the format of workspace data acquired by the depth sensor Example of worker's view and workspace data of corresponding area 140 Example of data format when saving part model and workspace model in the form of point cloud data Image when equipment is represented as point cloud data Example of data format when saving part model and workspace model in polygon format Example of data format of work manual data Flow chart showing work support procedure Illustration of position and orientation estimation Flow chart of processing executed by the confirmation status determination program Explanatory drawing of the processing to identify the area on the part model that lacks work space data Explanatory drawing of processing to determine the direction to be viewed by the operator An example of a method for presenting the position and direction to be viewed to the worker The block diagram of the work assistance system which concerns on the 2nd Embodiment of this invention. Example of work history data format Sequence diagram showing the flow of processing executed in the second embodiment

  Hereinafter, a work support system according to an embodiment of the present invention will be described with reference to the drawings. The work support system according to the present embodiment supports work by providing information such as work procedures to a worker who performs assembly work, for example.

  FIG. 1 shows an example of a hardware configuration of an information processing apparatus for realizing a work support system according to a first embodiment to which the present invention is applied.

  In FIG. 1, the information processing apparatus includes a central processing unit 101, an input device 102, an output device 103, and a storage device 104.

  The central processing unit 101 executes various programs.

  The input device 102 is a device for inputting data relating to a space including equipment and objects for performing work as work space data. In the present embodiment, the device 102 for inputting work space data may be a depth sensor 204 that acquires a distance image having a pixel value as a distance to an object. The input device 102 may include an input device in a general computer such as a keyboard, a mouse, and a touch panel.

  The output device 103 is a device that presents information to the worker, and may be a head-mounted display device called an HMD (head mounted display) 205.

  The storage device 104 stores a work management program 105, a position / orientation estimation program 106, a confirmation state determination program 107, and a display content generation program 108. Each program is executed by the central processing unit 101 to realize the functions described below.

  The storage device 104 further includes work manual data 800, a part model 120, a work space model 130, and work space data 140.

  The work management program 105 is a program that controls execution of the position / orientation estimation program 106, the confirmation state determination program 107, and the display content generation program 108 and manages information presented to the worker. For example, the work management program 105 may output the work procedure of each work process based on the work manual data 800. The work management program 105 may output the work procedure of the next work process when the end condition of one work process is satisfied.

  The position / orientation estimation program 106 is a program that estimates the positional relationship between the depth sensor 204 and the work space by obtaining the correspondence between the work space data 1400 and the work space model 1300. The position / orientation estimation program 106 estimates the position and orientation of the depth sensor 204, and matches the three-dimensional data of the mounted body and the part with the three-dimensional data of the work space based on the estimation result.

  The confirmation state determination program 107 is a program for estimating a portion where the worker has confirmed the object based on the positional relationship between the depth sensor 204 and the work space estimated by the position / orientation estimation program 106. For example, the confirmation state determination program 107 compares the three-dimensional data of the mounted body and the part with the three-dimensional data of the work space acquired by the depth sensor 204, and whether or not the end condition of the visual confirmation work of each work process is satisfied. It may be determined. As a condition for ending the visual confirmation work, a difference between the three-dimensional data of the mounted body and the part and the three-dimensional data of the work space may be a predetermined threshold or less.

  The display content generation program 108 is a program that determines the position and direction to be viewed by the worker based on the result of the confirmation state determination program 107 and generates information to be presented to the worker. For example, when the confirmation state determination program 107 determines that the end condition of the visual confirmation work is not satisfied, the display content generation program 108 may output guidance for satisfying the end condition. This guidance may indicate the viewpoint and the direction of the line of sight viewed by the worker. The viewpoint seen by the worker may be an area that is in the three-dimensional data of the mounted body and the part but not in the three-dimensional data of the work space.

  The part model 120 is a three-dimensional model that represents a part to be attached to an attached body. The part model 120 can be expressed in the form of point cloud data. In addition, the part model 120 may be a three-dimensional model using polygons (polygons) in order to present the location and orientation of the parts to the worker using computer graphics (CG) technology. The polygon can be converted into a point cloud model by dividing the polygon to a predetermined size and extracting the vertex. A part model of a plurality of parts may be registered in the storage device 104 in advance.

  The work space model 130 is a three-dimensional model that represents the entire space in which work is performed. The work space model 130 includes a three-dimensional model of a work target attached body in the work space. The workspace model 130 can be expressed in the form of point cloud data, similar to the part model 120. A plurality of work space models 130 corresponding to a plurality of work space states may be stored in the storage device 104 in the form of point cloud data.

  The work manual data 800 indicates the work procedure of each work process for one or more work processes. For example, the work manual data 800 includes information on work processes and work procedures of each work process, information on parts used in the work, and information on positions and orientations of parts in the work space model.

  FIG. 2 shows an example of a usage scene of the work support system in the present embodiment.

  In FIG. 2, 201 is a user terminal. The user terminal 201 is, for example, a portable terminal that performs various processes of the work support system, and may be a small computer, a tablet terminal, a smartphone, or the like. The user terminal 201 is equipped with a touch panel 202, and the work support system may operate according to a user operation received by the touch panel 202. For example, the touch panel 202 receives a user operation when starting or ending the system, or when returning to the original work process when the system erroneously changes to the next work process. In the present embodiment, the information processing apparatus in FIG. That is, in this embodiment, the user terminal 201 implements the functions of the work management program 105, the position / orientation estimation program 106, the confirmation state determination program 107, and the display content generation program 108.

  A helmet 203 is worn by the worker during work. The helmet 203 includes, for example, a depth sensor 204 for inputting work space data as the input device 102 and a head mounted display (HMD) 205 as a head-mounted display device as the output device 103.

  The depth sensor 204 is a three-dimensional data acquisition device that acquires three-dimensional data of a work space that is visible to the worker while the worker is working. The depth sensor 204 acquires work space data 140 representing the work space of the line of sight of the worker who is working. Therefore, the depth sensor 204 may be installed in the HMD 205. Or you may install the depth sensor 204 in the arbitrary places which can acquire the information of the target space, such as a worker's shoulder and chest.

  The distance image acquired by the depth sensor 204 can be converted into point cloud data representing information on the target space as a set of points on the three-dimensional space by using the focal length and the center position of the sensor. The workspace data 140 in this embodiment uses data expressed as point cloud data. Further, as the depth sensor 204, a sensor capable of acquiring a normal color image can be used. In this case, point cloud data to which color information is added can be acquired. In the present embodiment, point cloud data to which color information is added is used.

  206 is equipment arranged in the work space, and parts 207, 208, 209 and 210 to be attached to the equipment 206 are attached. Although FIG. 2 shows a state where the attachment has already been completed, the components 207 to 210 are not installed on the facility 206 at the start of the work.

  In the present embodiment, a case where a part is attached to the facility 206 in FIG. 2 will be described as an example. It can be applied to the work of

  FIG. 3 shows an example of the format of the work space data 140 acquired by the depth sensor 204.

  The work space data 140 includes, as data items, the number of points 1401, point position coordinates 1402, and point color information 1403. The work space data 140 includes point position coordinates 1402 and color information 1403 corresponding to the number of points indicated by the number 1401 of points.

  The position coordinates 1402 can be expressed by a combination of coordinate values on the x-axis, y-axis, and z-axis that are generally used. The position coordinates 1402 may be expressed in another coordinate system.

  The color information 1403 can be expressed by, for example, a combination of the values of the three primary colors (red, green, and blue). The color information 1403 may be expressed using another color space.

  FIG. 4A shows a worker's view 401, which is a work space that the worker is looking through the HMD 205. The worker's view 401 includes a part 209 and a part of the facility 206.

  FIG. 4B shows a drawing drawn based on the work space data 140 (point cloud data) of the area corresponding to the worker's view 401. As shown in FIG. 4B, when the point cloud data acquired from the depth sensor 204 is displayed on the space, the surfaces of the equipment 206 and the part 209 in the work space are expressed as a set of points.

  FIG. 5 shows an example of a data format when the part model 120 and the work space model 130 are stored in the form of point cloud data. The part model 120 and the workspace model 130 may have a common format. Here, the part model 120 and the work space model 130 in the form of point cloud data are referred to as a three-dimensional model (point cloud data) 500.

  The three-dimensional model (point cloud data) 500 includes a name 501, the number of points 502, point position coordinates 503, and point color information 504 as data items. The three-dimensional model (point group data) 500 includes position coordinates 503 and color information 504 of a plurality of points.

  A name 501 is a name of a part or work space, and is an identifier. An arbitrary character string can be used as the name 501.

  The number of points 502 is the number of points constituting a three-dimensional model of a part or work space.

  Point position coordinates 503 are position coordinates of points constituting a three-dimensional model of a part or a work space. The method of expressing the point position coordinates 503 may be the same as the position coordinates 1402.

  The point color information 504 is color information of a point constituting a three-dimensional model of a part or work space. The expression method of the point color information 504 may be the same as the color information 1403.

  FIG. 6 shows an equipment image 601 when the work space model 130 has point cloud data indicating the equipment 206 of FIG. 2 in the form of a three-dimensional model (point cloud data) 500, which is displayed on the space.

  FIG. 7 shows an example of a data format when the part model 120 and the work space model 130 are stored in a polygon (polygon) format. The part model 120 and the workspace model 130 may have a common format. Here, the polygonal part model 120 and the work space model 130 are referred to as a three-dimensional model (polygon data) 700.

  The three-dimensional model (polygon data) 700 includes, as data items, a name 701, the number of polygons 702, the number of vertexes 703 of the polygons, the position coordinates 704 of the vertexes, the normal direction 706, and polygon color information 707.

  The name 701 is a name given to distinguish each component and work space, and is an identifier. An arbitrary character string can be used as the name 701.

  The number of polygons 702 is the number of polygons constituting a part or work space. The three-dimensional model (polygon data) 700 has polygon vertex numbers 703 to polygon color information 707 by the number of polygons 702.

  The number of vertexes 703 of the polygon is the number of vertices constituting the polygon, and a numerical value such as 3 is registered if the polygon is a triangle and 4 if the polygon is a quadrangle. The three-dimensional model (polygon data) 700 has vertex position coordinates 704 corresponding to the number of polygon vertices 703.

  A vertex position coordinate 704 is a position coordinate of each vertex constituting the polygon. Vertex position coordinates 704 have data corresponding to the number of polygon vertices 703. The method for expressing the position coordinates may be the same as the position coordinates 1402.

  The normal direction 706 is information regarding a vector indicating the direction of the normal line of the polygon. The normal direction 706 can be represented by the values of components in the x-axis, y-axis, and z-axis directions of the vector.

  Polygon color information 707 indicates the color of the polygon. The method of expressing the polygon color information 707 may be the same as the color information 1403.

  FIG. 8 shows an example of the data format of the work manual data 800.

  The work manual data 800 includes information on the work procedure of each process for each of a plurality of work processes constituting the work, information on a part to be attached to the mounted body in the work, and information on the position and orientation of the part in the work space model. Is included.

  The work name 801 is a name given to distinguish each work when there are a plurality of works, and is an identifier. An arbitrary character string can be used for the work name 801.

  The number of work steps 802 is the number of steps constituting the work.

  The work process name 803 is the name of each work process. An arbitrary character string can be used for the work process name 803.

  The work process content 804 is the content of the work process performed in each work process. An arbitrary character string can be used for the content 804 of the work process. The work process content 804 can include an image or a moving image showing a specific work procedure.

  The workspace model name 805 is the name of the workspace model 130 related to each work process. Since the work space changes as the work progresses, the data format shown in FIG. 8 allows a different work space model 130 to be specified for each work process.

  The part model name 806 is the name of the part model of the part attached in each work process.

  The part position and orientation 807 is information indicating the position and orientation of the part of the part model name 806, and includes information on the position and orientation. The position may be a combination of commonly used coordinate values on the x-axis, y-axis, and z-axis. The direction can be expressed by rotation angles around the x-axis, y-axis, and z-axis. As a method for expressing the position and orientation, any expression may be used as long as it has a similar function.

  Further, in the format shown in FIG. 8, there is one part (part model name 806 and part position and orientation 807) that is a target in each work process, but a plurality of parts may be registered.

  A procedure of work support performed by the work support system according to the present embodiment will be described with reference to the flowchart of FIG.

  In step 901 in FIG. 9, the work management program 105 reads information on the work process selected by the worker from the work manual data 800 into the system. As a method for selecting a work process, for example, a list of work processes may be displayed on the screen of the user terminal 201 so that the work process can be selected using a touch panel or the like.

  In step 902, the content of the work process in the read work process is set as a work procedure to be executed.

  In step 903, the system state is set to “working”.

  In step 904, the work management program 105 acquires work space data (point cloud data) viewed by the worker from the depth sensor 204.

  In step 905, the posture estimation program 106 estimates the position and posture of the depth sensor 204 with respect to the work space model 130. The posture estimation program 106 may be activated by the work management program 105.

  The posture estimation program 106 uses the work space data 140 acquired in step 904 and the point cloud data of the work space model 130 corresponding to the work space model name 805 of the work process being performed, to determine the position and posture of the depth sensor 204. May be estimated. For example, the position / orientation estimation program 106 is configured so that the work space data 140 acquired from the depth sensor 204 and the point cloud data of the work space model 130 corresponding to the work space model name 805 of the work process being executed most closely match. In addition, the position and orientation of the depth sensor 204 may be estimated by obtaining a transformation matrix that translates and rotates one of the point cloud data.

  With reference to FIG. 10, the estimation of the position and orientation will be specifically described.

  FIG. 3A shows a work space 1001 drawn based on the work space data 140 acquired from the depth sensor 204. FIG. B is a work space model 601 in which the point cloud data of the work space model 130 corresponding to the work space model name 805 of the work process being executed is drawn.

  The position / orientation estimation program 106 obtains, for example, a transformation matrix that translates and rotates the work space 1001 so that the work space 1001 matches the corresponding location 1002 in the work space model 601. As a method for obtaining a transformation matrix that best matches the work space 1001 and the corresponding portion 1002, a method well known as alignment of point cloud data, for example, the ICP (Iterative Closest Point) algorithm (S. Rusinkiewicz and M. Levoy, “Efficient Variants of the ICP Algorithm”, Third International Conference on 3D Digital Imaging and Modeling, 2001) and NDT (Normal Distributions Transformation) (P. Biber and W. StraBer, “The Normal Distributions Transform: A New Approach to Laser ScanMatching Proceedings of the 2003 IEEE / RSJ International Conference on Intelligent Robots and Systems, 2003) can be used.

  As a method for estimating the position and orientation of the depth sensor 204, any method other than this may be used as long as it is a method for obtaining a rotation matrix in which the two point cloud data most closely match.

  Returning to FIG. 9, in step 906, the work management program 105 determines whether the system state is “working” or “confirming”. If the system status is “working”, the process proceeds to step 907, and if it is “confirming”, the process proceeds to step 910.

  In step 907, the work management program 105 may extract information on the contents of the work process being performed from the work manual data 800 and display it on the HMD 205. In this case, the displayed information may be a work procedure name 803 and a work process content 804.

  At this time, the work management program 105 uses the part model 120 of the part model name 806, the position and orientation 807 of the part, and the position and orientation of the depth sensor 204 estimated in step 905 in the worker's view. A CG image of a part may be displayed on the HMD 205 so as to be superimposed on the part in the work space. Such display on the HMD 205 can be performed, for example, by using a technique well known as augmented reality.

  In step 908, the work management program 105 determines whether or not there is an instruction to start confirmation from the worker. Here, the confirmation is a confirmation that an operator visually confirms whether or not the work of the process is completed as specified after the work procedure of each process is completed. When this visual confirmation end condition is satisfied, the process proceeds to the next work process in step 914.

  For the confirmation start instruction, for example, a confirmation start button is displayed on the screen of the user terminal 201, and a button displayed using a keyboard, a mouse, a touch panel, or the like provided as the input device 102 is pressed. Can be performed. Alternatively, any method may be used as long as it is a method of instructing the system, such as a method of pressing a predetermined keyboard or button or a predetermined gesture detected from point cloud data acquired by the depth sensor 204. But it can also be used.

  If the operator has not instructed confirmation start, the process returns to step 904 as it is.

  When the operator gives an instruction to start confirmation, the work management program 105 sets the system state to “confirming” in step 909 and then returns to step 904.

  If the system status is “confirming” in step 906, the process proceeds to step 910.

  In step 910, the confirmation state determination program 107 started by the work management program 105 determines the confirmation state. For example, the confirmation state determination program 107 determines whether or not the confirmation status being performed by the worker satisfies the end condition of the work process. If the confirmation state determination program 107 further determines that there is an unconfirmed part area, the confirmation state determination program 107 sets the area as an area that needs to be confirmed by the operator, that is, the position and direction viewed by the worker.

  Processing executed by the confirmation state determination program 107 will be described with reference to FIGS.

  The confirmation state determination program 107 extracts point cloud data corresponding to parts related to the work process being performed from the work space data 140 acquired from the depth sensor 204 in step 1101.

  In this process, first, the confirmation state determination program 107 may specify a part area in the work space model 130. For example, the component area may be identified by coordinate-transforming the component model 120 identified by the component model name 806 of the work manual data 800 using information identified by the position and orientation 807. Next, the confirmation state determination program 107 may perform coordinate conversion of the work space data acquired by the depth sensor 204 into the same coordinate system as the work space model, using the position and orientation of the depth sensor 204 estimated in step 905. . Then, the confirmation state determination program 107 may extract the point group data of the component area from the work space data subjected to coordinate conversion. When the part region has already been extracted, the newly extracted point group data may be added (synthesized) to the point group data of the extracted part region.

  This will be described using a specific example shown in FIG.

  In the figure, reference numeral 1201 denotes a work space in which the point cloud data of the work space data 140 is drawn after the work space data 140 is coordinate-transformed into the same coordinate system as the work space model 130. Reference numeral 1202 denotes an area of the part specified in the work space 1201 by the position and orientation 807 of the part. At this time, point cloud data 1203 included in the component area 1202 is extracted as point cloud data for the component.

  The extracted point cloud data is coordinate-transformed into the same coordinate system as the work space model. Therefore, the point cloud data can be synthesized by adding the point data further acquired by the depth sensor 204 to the extracted point cloud data. In addition, in the point group data after synthesis, there are cases where the points overlap at the same position, or there are cases where the points are concentrated in a specific range, so that by performing processing such as extracting points at specific intervals, It may be converted into point cloud data in which points are distributed.

  Returning to FIG. 11, in step 1102, the confirmation state determination program 107 determines whether or not the point of the part area extracted in step 1101 is greater than or equal to a predetermined value. For example, the confirmation state determination program 107 determines whether or not the difference between the point cloud data of the extracted part and the part model is equal to or smaller than a predetermined value. For example, the confirmation state determination program 107 may determine whether there is an area that exists in the part model but does not exist in the point cloud data of the target part acquired by the depth sensor 204. This determination may be an end condition for the visual confirmation work.

  This processing may be performed by comparing the point group data obtained by synthesizing the point group data of the part area extracted in step 1101 and the point group data of the part area extracted before that with the part model.

  When the part model is point cloud data, the determination in step 1102 may be performed as follows. For example, the confirmation state determination program 107 obtains a point of the part model that is closest to each point in the extracted point group data of the part, and calculates the distance between the points. Then, the confirmation state determination program 107 extracts points on the part model where the distance between the points is larger than a predetermined threshold. Thereby, the area where the extracted points are collected may be an area on the part model in which the point cloud data acquired by the depth sensor 204 is insufficient.

  If the part model is in the form of a polygon, the determination in step 1102 may be performed as follows. For example, the confirmation state determination program 107 searches for a polygon on the part model in which the perpendicular drawn from each point in the extracted point cloud data of the part is the shortest, and finds the perpendicular distance and the intersection of the perpendicular and the polygon. Ask. Then, the confirmation state determination program 107 extracts the intersection of the perpendicular and the polygon where the distance of the perpendicular is larger than a predetermined threshold. As in the case of point cloud data, the area where the extracted points are collected may be the area on the part model for which the point cloud data acquired by the depth sensor 204 is insufficient.

  In FIG. 12, reference numeral 1204 denotes a part model stored as point cloud data. At this time, when the component model 1204 and the point cloud data 1203 are compared by the above method, the region of the component indicated by 1205 is specified as the region where the point cloud data is insufficient. When the size of the specified area is larger than a predetermined threshold, it is determined that the point cloud data for the part is not sufficiently acquired. Alternatively, the ratio between the size of the area where the point cloud data is insufficient and the surface area of the part is obtained, and if the obtained ratio is larger than a predetermined threshold, it is determined that the point cloud data for the part is not sufficiently acquired. May be.

  Returning to FIG. 11, if it is determined in step 1103 that the point cloud data for the part is insufficient in step 1102, the process proceeds to step 1104. If the point cloud data for the part is sufficiently acquired, the process is terminated.

  In step 1104, the operator's viewpoint and line-of-sight direction are determined for the area where the point cloud data is insufficient.

  The position (viewpoint) that the operator should look at may be, for example, the center or the center of gravity of an area where point cloud data is insufficient. When the corresponding area is larger than a predetermined threshold, the corresponding area may be divided into smaller areas, and the center, the center of gravity, etc. may be obtained for one of the obtained small areas. As a method of dividing the region, for example, when dividing into two, the region may be mapped on a specific coordinate axis and divided by a plane passing through an intermediate point of the mapped region.

  The confirmation state determination program 107 determines the viewing direction after the operator determines the position to be viewed.

  A method for determining the direction (the direction of the line of sight) that the operator should look at will be described with reference to FIG.

  In FIG. 13, it is assumed that 1301 is an area where it is determined that the point cloud data is insufficient, and 1302 is a position determined as a position to be viewed by the operator. Here, the confirmation state determination program 107 may set the inverse vector of the normal line of the region 1301 as the reference direction 1303 in which the operator should look at the position 1302.

  Next, the confirmation state determination program 107 may verify whether or not a straight line in the reference direction 1303 and a polygon constituting another part intersect. Thereby, when the worker looks at the position 1302 from the reference direction 1303, it can be determined whether there are other parts that block the field of view. Alternatively, the presence / absence of a part that may obstruct the field of view by verifying whether or not the confirmation state determination program 107 is at a distance within a predetermined threshold between a straight line in the reference direction 1303 and another part. May be determined. In the case of FIG. 13, when viewing the position 1302 from the reference direction 1303, it is determined that another part 1304 blocks the field of view. When it is determined that there is no other part that blocks the field of view, the reference direction 1303 may be determined as the direction in which the operator should look at the position 1302. When it is determined that there is another part that blocks the field of view, the confirmation state determination program 107 uses the same method as described above for the direction in which the angle is changed around the reference direction 1303 (for example, 1305 or 1306 in the drawing). Thus, when the position 1302 is viewed, the presence or absence of other parts that block the field of view is determined, and the direction in which it is determined that there is no other part that blocks the field of view may be determined as the direction in which the operator should view the position 1302 .

  As a method of changing the angle of the line-of-sight direction, a width for changing the angle may be set in advance, and the angle may be converted using the set width. Further, the determination of the presence or absence of other parts that block the view may be determined that there are no other parts that block the view when there are no polygons or points within a predetermined range from the straight line representing the direction to be viewed. . In the case of FIG. 13, a direction 1305 in which there is no other part that blocks the view may be selected.

  Returning to FIG. 9, in step 911, the display content generation program 108 is activated. Then, the display content generation program 108 generates a guidance image indicating the position and viewing direction of the region that the operator should see for confirmation. The guidance image may be generated based on the region of the part that is not viewed by the operator and the direction in which the region is viewed, determined in step 910.

  The guidance image may be an image showing a position and a direction on the HMD 205 so as to overlap an actual part that an operator is looking through the HMD 205 by using a well-known augmented reality technology. The guidance image may include characters.

  The guidance image may be, for example, an image indicating the position of the attached body or part viewed by the worker and the direction of the line of sight in the work space that is visible to the worker through the HMD 205. This may be realized using augmented reality technology. In addition, the guidance image may be an image indicating the position of the mounted body or the part viewed by the operator and the direction of the line of sight in the CG image including the mounted body and the part.

  For example, as shown in FIG. 14A, the display content generation program 108 displays an image 1401 in which an arrow indicating the position and direction and a message for prompting confirmation are combined in a work space (worker's view) actually viewed by the worker. May be displayed. Alternatively, as shown in FIG. 14B, instead of a message, an image 1403 such as an eye, a face, a camera, etc. that recalls visual observation may be presented in combination with CG. Alternatively, an animation using an arrow or the like that moves to a viewing position can be used. Furthermore, in addition to performing superimposition with actual parts based on augmented reality, a CG image 1404 as shown in FIG. 14C may be displayed. The CG image 1404 may display the position and direction that the operator should see in the CG of the work space and parts generated based on the part model 120 and the work space model 130. The CG image 1404 may be presented at a position in the HMD 205 that does not block the operator's field of view, for example, on the upper left and right sides.

  In this case, by using augmented reality technology, the display content generation program 108 can display in real time a CG image of a part model and a workspace model viewed from the same viewpoint as the worker is viewing an actual part and workspace. You may make it produce | generate. As a method for presenting the position and direction of the region to be viewed to the operator, any method may be used as long as the method can recognize the position and direction of the region to be viewed.

  Further, the display content generation program 108 may detect the operator's hand or the tool held by the operator in the work space data acquired by the depth sensor 204. If the detected operator's hand or tool is in a position where it can be seen to overlap with the target part, or if it is within a predetermined range, the display content generation program 108 selects the CG. A presentation method using an image may be adopted, and in other cases, a presentation method using augmented reality technology may be adopted. This is because the operator's hand or tool may shield or shield the position of a part or the like that the operator should see.

  When there is something that shields the part, and when the distance between the worker and the part is short, in many cases, the estimation of the position and orientation in step 905 in FIG. 9 is not properly completed. Therefore, a method using augmented reality technology may be used when position and orientation estimation is appropriate, and a method using CG images may be used otherwise.

  Here, the determination of whether or not the estimation of the position and orientation is properly completed may be performed as follows, for example. That is, after the work space data acquired from the depth sensor 204 using the estimated position and orientation is coordinate-converted into the coordinate system of the work space model, the distance between the converted work space data and the work space model is determined in advance. If it is within a predetermined threshold, it may be determined that the estimation of the position and orientation is appropriate. The CG image when the position and orientation estimation is not appropriate may be, for example, a CG image generated using the position and orientation obtained when the position and orientation estimation is completed properly at the end, or A CG image generated using a predetermined position and orientation may be used.

  Further, the display content generation program 108 may switch between a presentation method using augmented reality shown in FIGS. 14A and 14B and a presentation method using CG images shown in FIG. 14C. For example, (1) When the distance between the worker and the part is less than or equal to a predetermined value, (2) When there is an object that shields or may shield the position of the mounted body or part that the worker sees, and (3) Expansion In the image using reality, when the display of the position of the attached body or part and the direction of the line of sight that the operator sees does not fit in the HMD 205, (4) the attached body according to the first work process after coordinate conversion In any case where the distance between the three-dimensional data of the part and the three-dimensional data of the work space is a predetermined distance or more, a method using a CG image may be used.

  Returning to FIG. 9, if it is determined in step 912 that the confirmation is completed in step 910 (end condition is satisfied), the process proceeds to step 913. If it is determined that the confirmation has not been completed (end condition is not satisfied), the process returns to step 904.

  Before proceeding to step 913, information for notifying the operator that the confirmation has been completed or the next procedure may be performed may be presented to the HMD 205. In this case, the operator's response to the notification may be acquired from the input device 102, and the process may proceed to step 913 after the response is acquired.

  In step 913, the work management program 105 determines whether all work processes have been completed. When all the work steps are finished, the process is finished. When the process is ended, the work space data (including point cloud data of each part) 140 acquired at the time of confirmation may be stored in the storage device 104. Furthermore, the start time, confirmation start time, confirmation end time, etc. of each work procedure may be stored together.

  If all work processes have not been completed, the process proceeds to the next work process in step 914, the system state is set to “working” in step 915, and the process returns to step 904.

  Note that the present system may have a function of determining timeout. In other words, when the confirmation of the target part is not completed in step 912 and the elapsed time from the start of the confirmation exceeds a predetermined time, the work management program 105 forcibly performs the processing after step 903. It may be. At this time, the work management program 105 may present information to the HMD 205 instructing the worker to redo the work. Furthermore, the operator's response to the instruction to redo the work may be acquired by the input device 102, and the process may return to step 903 after the response is acquired.

  Furthermore, this system may have a function of forcibly resetting when the degree of conformity between the work space data and the part model is low. That is, in step 912, when the distance between the work space data and the part model is larger than a predetermined threshold, the work management program 105 may forcibly perform the processing from step 903 onward. At this time, the work management program 105 may present information to the HMD 205 instructing the worker to redo the work. Furthermore, the operator's response to the instruction to redo the work may be acquired by the input device 102, and the process may return to step 903 after the response is acquired.

  In the present embodiment, the example in which the HMD 205 is used as the output device has been described. However, the user terminal 201 may be used as the output device. For example, the HMD 205 is preferably used as an output device for the purpose of assisting a worker who is working, and the user terminal 201 is used as an output device for the purpose of confirming work results after work. Is preferred.

  When the user terminal 201 is an output device, the depth sensor 204 may be integrated with the user terminal 201 or may be attached to the user terminal 201. Various processes and display methods may be the same as the processes and display methods on the assumption that the HMD is used.

  Further, in the present embodiment, the description has been made on the attachment work, but the present invention can also be applied to confirmation of completion of confirmation work in a more general manufacturing process, construction / renovation work, inspection, or the like.

  Furthermore, as a method for presenting information to the worker, a method of switching between a method using a CG image and a method using augmented reality can be applied not only during a confirmation work but also during a normal work such as installation. It is.

  According to the present embodiment, when the worker confirms the work result, the location where the confirmation is insufficient is automatically detected, and the operator's line of sight is guided to the detected location, so that the work result can be reliably confirmed. It is possible to assist the worker so that the confirmation can be performed. Furthermore, information that makes it easy for the worker to understand the situation can be presented according to the worker or the situation of the workspace when the work result is being confirmed.

  Next, a work support system according to a second embodiment of the present invention will be described with reference to FIGS.

  FIG. 15 is a configuration diagram of a work support system according to the second embodiment.

  The work support system according to the present embodiment includes a worker terminal 1501 and a work information management terminal 1517, which are connected via a network 1516. The work information management terminal 1517 manages data used in the present system. The worker terminal 1501 obtains data from the work information management terminal 1517 and supports the worker. In the following description, description of functions or configurations similar to those of the first embodiment may be omitted.

  The worker terminal 1501 includes a central processing unit 1502, an input device 1503, an output device 1504, and a storage device 1506. These are the same as the central processing unit 101, the input device 102, the output device 103, and the storage device 104 in FIG.

  The storage device 1506 includes a transmission / reception program 1507, a work management program 1508, a position / orientation estimation program 1509, a confirmation state determination program 1510, a display content generation program 1511, a part model 1513, a work space model 1514, a work Manual data 1515 is stored.

  The work management program 1508 has the same function as the work management program 105 in the first embodiment, and further has a function of transmitting and receiving data to and from the work information management terminal 1517.

  The position / orientation estimation program 1509, the confirmation state determination program 1510, and the display content generation program 1511 are the same as the position / orientation estimation program 106, the confirmation state determination program 107, and the display content generation program 108 in the first embodiment, respectively.

  The part model 1513, the work space model 1514, and the work manual data 1515 are the same as the part model 120, the work space model 130, and the work manual data 800 in the first embodiment.

  The worker terminal 1501 further includes a communication device 1505. The communication device 1505 is a device for exchanging information with the work information management terminal 1517.

  The work result 1512 stores information on work time such as work start time and confirmation completion time, and work space data (point cloud data) acquired as a result of checking each part.

  The transmission / reception program 1507 stored in the storage device 1506 receives work manual data related to the work selected by the worker and the accompanying part model and work space model from the work information management terminal 1517, and the work space model 1513, The information is stored in the part model 1514 and the work manual data 1515, and after completion of the work in progress, information on the work time stored in the work result 1512 and point cloud data of each part are transmitted to the work information management terminal 1517. It is a program to do.

  The work information management terminal 1517 is a work information management terminal that collectively manages information related to work procedures and the accompanying part models and work space models.

  Reference numeral 1518 denotes a central processing unit for executing various programs.

  Reference numeral 1519 denotes an input device in a general computer such as a keyboard, a mouse, or a touch panel for controlling various programs.

  Reference numeral 1520 denotes an output device used in a general computer such as a display.

  Reference numeral 1521 denotes a communication device for exchanging information with the worker terminal 1501.

  The storage device 1522 stores a transmission / reception program 1523, a work procedure registration program 1524, a work history browsing program 1525, a part model 1526, a work space model 1527, work manual data 1528, and a work history 1529.

  The transmission / reception program 1523 transmits the work manual data and the accompanying part model and work space model to the worker terminal 1501, and also transmits information on the work time and point cloud data of each part transmitted from the worker terminal 1501. This program is received and stored in the work history 1529.

  The work procedure registration program 1524 is a program for adding, modifying, and deleting work manual data and the accompanying part model and work space model.

  The work history browsing program 1525 is a program for displaying the contents of the work history 1529 on the output device 1520 and confirming the contents of the work history and adding / modifying information.

  The part model 1526, work space model 1527, and work manual data 1528 are the same as the part model 120, work space model 130, and work manual data 800, respectively.

  The work history 1529 stores time information of each work procedure in work performed in the past and work space data acquired at the time of confirmation as work execution history. The data format of the work history 1529 is the same as the work result data 1600 stored in the work result 1512.

  FIG. 16 shows an example of the data format of the work result data 1600.

  Reference numeral 1601 denotes a work name, which is an identifier. An arbitrary character string can be used for the work name 1601.

  Reference numeral 1602 denotes the date and time when the work was performed. The work execution date 1602 has a date and time. The time can include one or both of a work start time and an end time.

  Reference numeral 1603 denotes the number of work items.

  Reference numeral 1604 denotes the completion time of the work process.

  Reference numeral 1605 denotes work space data at the time when the confirmation of the work process is completed. It may be the point cloud data of the work space that has been acquired when the visual confirmation of the work of each process is completed.

  The work result data 1600 includes work process completion times 1604 and work space data 1605 corresponding to the number of work items 1603.

  In the data format of FIG. 16, only the completion time of each work process is described, but the start time of the work process and the start time of confirmation can also be included.

  FIG. 17 is a sequence diagram showing the flow of processing in this embodiment.

  When the worker activates the system (1701), the worker terminal 1501 uses the work management program 1508 to request the work information management terminal 1517 for a list of information regarding work (1702).

  The work information management terminal 1517 receives the request from the worker terminal 1501, and transmits a list of information related to the work to the worker terminal 1501 in response to the request (1703).

  The worker terminal 1501 receives a list of information related to the work, and displays the received work list on the output device 1504 (1704).

  When the worker selects a work process to be executed (1705), the worker terminal 1501 sends work manual data related to the work process selected by the worker to the work information management terminal 1517 and the three-dimensional data attached to the work process. A model is requested (1706).

  The work information management terminal 1517 transmits work manual data and a three-dimensional model based on a request from the worker terminal 1501 (1707).

  The worker terminal 1501 displays information on the first work process on the output device 1504 based on the received work manual data and the three-dimensional model by the work management program 1508 (1708).

  When the worker notifies the worker terminal 1501 of the start of confirmation of the first work process (1709), the worker terminal 1501 determines the position and orientation estimation program 1509, the confirmation state determination, as in the first embodiment. By using the program 1510 and the display content generation program 1511, the operator's confirmation work is supported. When the confirmation is completed, information on the second work process is displayed on the output device 1504 (1710).

  The worker is notified of the start of confirmation regarding the nth work process, which is the last work process (1711), and when the confirmation is completed, the worker terminal 1501 displays a notice of work completion on the output device 1504 ( 1712).

  Further, the worker terminal 1501 transmits time information such as the start time and end time of each process and work space data including each part acquired at the time of confirmation to the work information management terminal 1517 as a work result (1713). ). The work information management terminal 1517 stores the received work result in the work history 1529.

  In the second embodiment, the work information management terminal 1517 performs only management of the work manual data, the part model, and the work space model, but the work information management terminal 1517 performs part or all of the work support function. You may do it. In this case, the worker terminal 1501 transmits the point cloud data acquired from the input device 1503 to the work information management terminal 1517, and displays the display contents received from the work information management terminal 1517 on the output device 1504. Only the work result 1512, the part model 1513, the work space model 1514, and the work manual data 1515 may be deleted from the worker terminal 1501.

  The work support system according to the second embodiment can support the worker in the same manner as in the first embodiment. Furthermore, in the second embodiment, work manual data is centrally managed by the work information management terminal, and only necessary work manual data is read into the worker terminal, thereby providing work support according to each worker. It becomes possible. Further, by consolidating the work history into the work information management terminal, it becomes easy for a person other than the operator to check the work result, to perform a comparative analysis of the work result, and the like.

105 Work Management Program 106 Position / Orientation Estimation Program 107 Confirmation State Determination Program 108 Display Content Generation Program 120 Part Model 130 Work Space Model 130 Work Space Model 140 Work Space Data 201 User Terminal 202 Touch Panel 203 Helmet 204 Depth Sensor 205 Head Mount Display (HMD)

Claims (9)

A storage device for storing three-dimensional data of a work target attached body and a part attached to the work attached in the work space, and work manual data indicating a work procedure of one or more work steps;
A three-dimensional data acquisition device for acquiring three-dimensional data of a work space that is visible to the worker while the worker is working;
A work support system comprising a central treatment device,
The central processing unit is
Based on the work manual data stored in the storage means, processing to output the work procedure of each work process,
A comparison is made between the three-dimensional data of the mounted body and the part related to the first work process stored in the storage means and the three-dimensional data of the work space acquired by the three-dimensional data acquisition device, A process of determining whether or not the end condition of the visual confirmation work of the work process is satisfied,
And a process for outputting guidance for satisfying the end condition when the end condition is not satisfied in the determination. The central processing unit further includes:
Estimating the position and orientation of the three-dimensional data acquisition device, and matching the three-dimensional data of the mounted body and parts related to the first work process with the three-dimensional data of the work space based on the estimation result Process
2. The end condition of the visual confirmation work is that the difference between the three-dimensional data of the attached body and the part according to the first work process and the three-dimensional data of the work space is a predetermined threshold value or less. Work support system.   The work support system according to claim 2, wherein the guidance for satisfying the end condition indicates a viewpoint and a line-of-sight direction viewed by the worker.   The work support system according to claim 3, wherein the viewpoint viewed by the worker is an area that is in the three-dimensional data of the mounted body and the part related to the first work process but not in the three-dimensional data of the work space. Further comprising a head mounted display worn by the operator,
Guidance for satisfying the termination condition is as follows:
A first display mode for displaying a position and a line-of-sight direction of the attached body or the component viewed by the worker on the head-mounted display in a work space that is visible to the worker through the head-mounted display;
Any one of the second display modes in which the position and the direction of the line of sight of the mounted body or the part viewed by the worker in the CG image including the mounting body and the part are displayed on the head mounted display by computer graphics. The work support system according to claim 2, wherein the work support system is output in such a display mode. The work support system according to claim 5, wherein the guidance is output in the second display mode when any one of (1) to (4) is satisfied.
(1) When the distance between the worker and the part is equal to or less than a predetermined value (2) When there is an object that shields or may shield the position of the mounted body or the part viewed by the worker (3) In the first display mode, when the display of the position of the attached body or the part and the direction of the line of sight seen by the worker does not fit in the head mounted display. (4) Even if the matching is performed, the first work When the deviation between the three-dimensional data of the mounted body and the part related to the process and the three-dimensional data of the work space is greater than or equal to a predetermined value The central processing unit further includes:
The work support system according to claim 1, wherein when the end condition is satisfied in the determination, a process for outputting a work procedure of a second work process subsequent to the first work process is performed. The central processing unit further includes:
The work support system according to claim 1, wherein when the end condition is satisfied in the determination, a process of saving the three-dimensional data of the work space in the storage device is performed. A storage device for storing three-dimensional data of a work target attached body and a part attached to the work attached in the work space, and work manual data indicating a work procedure of one or more work steps;
A three-dimensional data acquisition device for acquiring three-dimensional data of a work space that is visible to the worker while the worker is working;
A work support method performed by a work support system comprising a central treatment device,
The central processing unit is
Based on the work manual data stored in the storage means, processing to output the work procedure of each work process,
A comparison is made between the three-dimensional data of the mounted body and the part related to the first work process stored in the storage means and the three-dimensional data of the work space acquired by the three-dimensional data acquisition device, A process of determining whether or not the end condition of the visual confirmation work of the work process is satisfied,
And a process of outputting guidance for satisfying the end condition when the end condition is not satisfied in the determination.