JP6643170B2 - Work support system and method - Google Patents

Description

  The present invention relates to a work support system that assists a worker by presenting information necessary for the work to the worker in various works such as assembly of a product and maintenance of equipment.

  2. Description of the Related Art In various operations such as assembly of products and maintenance of facilities, it is necessary for an operator to confirm whether or not the operation has been properly completed. At that time, support is needed to confirm necessary parts without overlooking them. In some cases, the work result may be stored according to the place where the work is performed or a photograph of the target object. In this case, a photograph or the like of a place necessary for performing a check after the work is appropriately acquired. It is desirable to support workers.

  In recent years, a technology called augmented reality, which is a technology that assists users in understanding the real world by presenting virtual information such as other images and characters at appropriate positions on the video that reflects the real world, has attracted attention (For example, Non-Patent Document 1). By using the augmented reality, it is possible to three-dimensionally present the position of the object on which the work is performed on the work space viewed by the worker. Since the position of the object presented in this manner is also a place where confirmation is required, it can be used for the purpose of supporting confirmation of the work result. Further, when displaying the position of the target, a three-dimensional display that guides the operator's line of sight to the position of the target is also possible.

  For example, in Patent Literature 1 and Patent Literature 2, the progress of work is determined by calculating a difference between three-dimensional CAD data or three-dimensional target data and three-dimensional data acquired by a three-dimensional measuring device or a stereo image. The technology is described. Further, Patent Literature 2 discloses a technique of superimposing and displaying target data and acquired three-dimensional data to visualize a difference from target data.

  Patent Literature 3 describes a technique in which a photograph in a database is compared with a newly acquired photograph, and a difference is highlighted.

  Patent Literature 4 and Patent Literature 5 describe, in the augmented reality technology disclosed in Non-Patent Literature 1, a technology of changing information to be superimposed on a real space according to a situation. For example, Patent Literature 4 discloses a technique for changing the content of information based on the position and direction of a target, and Patent Literature 5 discloses a technique for detecting a worker's hand and changing the display color and transparency depending on the presence or absence of the hand. Have been.

JP-A-9-133519 JP-A-2002-352224 Japanese Patent No. 5740884 Japanese Patent Application No. 2009-55575 Japanese Patent Application No. 2010-269635

"Special Feature Augmented Reality", Journal of Information Processing Society of Japan, Vol. 51, No. 4, 2010

  By using the technology of Non-Patent Document 1, it is possible to guide the operator's line of sight to a location or an object that needs to be confirmed. Further, by using the techniques of Patent Literature 1, Patent Literature 2, and Patent Literature 3, the worker can recognize the difference between the three-dimensional target data and the three-dimensional CAD data and the state of the target after the completion of the work. It is possible.

  However, these techniques do not allow a worker to check a specific part or a part of an object, determine which part has not been confirmed, and present the unconfirmed part to the worker.

  In the technique of Non-Patent Document 1, the position of the target object can be three-dimensionally presented on the work space viewed by the worker. It is considered to be effective as a technique for guiding the gaze to the location.

  However, when the distance between the worker and the object is short, a necessary portion may not be included in the display area depending on the display device. In such a case, on the contrary, it is difficult for the operator to grasp the situation.

  In the technique of Non-Patent Document 1, a process of estimating a positional relationship between a worker and a work space is performed. However, if the process is not properly performed, there is a problem that no information is presented at all.

  In Patent Literature 4 and Patent Literature 5, the content of the information to be presented is changed based on the situation. However, since it is intended only for superimposed display on the real space, it is not sufficient for the above situation. This is not an effective method.

  Therefore, an object of the present invention is to provide a technique for assisting a worker so that the worker can appropriately confirm the work result.

  Still another object of the present invention is to provide information for assisting confirmation of a work result to an operator in an easy-to-understand manner.

  A work support system according to an embodiment of the present invention provides a work support system which includes three-dimensional data of a work to be mounted and components to be mounted on the work to be mounted in a work space and work procedures of one or more work processes. A storage device for storing manual data, a three-dimensional data acquisition device for acquiring three-dimensional data of a work space seen by the worker while the worker is working, and a central treatment device. The central processing unit is configured to output a work procedure of each work process based on work manual data stored in the storage unit, and to perform processing related to a first work process stored in the storage unit. The three-dimensional data of the mounting body and the parts are compared with the three-dimensional data of the work space acquired by the three-dimensional data acquisition device, and it is determined whether or not a condition for ending the visual check operation in the first operation process is satisfied. And performing a process of outputting guidance for satisfying the termination condition when the termination condition is not satisfied in the determination.

  In a preferred embodiment, the central processing unit further estimates the position and orientation of the three-dimensional data acquisition device, and based on a result of the estimation, determines whether the three-dimensional data acquisition device and the component to be attached and the component related to the first work process are to be used. A process of matching the dimensional data with the three-dimensional data of the work space may be performed. The condition for ending the visual check operation may be such that a difference between the three-dimensional data of the attached body and the component related to the first operation process and the three-dimensional data of the work space is equal to or less than a predetermined threshold.

  In a preferred embodiment, the guidance for satisfying the termination condition may indicate a viewpoint and a line of sight of the worker.

  In a preferred embodiment, the viewpoint seen by the worker may be an area in the three-dimensional data of the attached body and the component related to the first work process but not in the three-dimensional data of the work space.

  In a preferred embodiment, the information processing apparatus may further include a head mounted display worn by the worker. The guidance for satisfying the termination condition is that, in a work space that is visible to the worker through the head mounted display, a position and a line of sight of the attached body or the component viewed by the worker are displayed on the head mounted display. A first display mode for displaying the position and the line of sight of the attached body or the part seen by the worker in the CG image including the attached body and the part on the head mounted display by computer graphics. The information may be output in any one of the two display modes.

In a preferred embodiment, the guidance may be output in the second display mode when any of (1) to (4) is satisfied.
(1) When the distance between the worker and the component is equal to or less than a predetermined value (2) When the position of the attached body or the component viewed by the worker is shielded or there is an object that may shield the position (3) In the first display mode, when the display of the position of the attached body or the component and the direction of the line of sight seen by the worker does not fit on the head mounted display. (4) The first work is performed even if the matching is performed. When the deviation between the three-dimensional data of the attached body and the part related to the process and the three-dimensional data of the work space is equal to or more than a predetermined value.

  In a preferred embodiment, the central processing unit may further output a work procedure of a second work process subsequent to the first work process when the termination condition is satisfied in the determination.

  In a preferred embodiment, the central processing unit may further store the three-dimensional data of the work space in the storage device when the termination condition is satisfied in the determination.

  ADVANTAGE OF THE INVENTION According to this invention, a worker can be assisted so that the confirmation of the work result which a worker performs can be performed appropriately.

  Further, according to the present invention, information for supporting confirmation of the work result can be provided to the worker in an easy-to-understand manner.

An example of a hardware configuration of an information processing device for realizing the work support system according to the first embodiment of the present invention An example of a use scene of the work support system in the present embodiment Example of Workspace Data Format Acquired by Depth Sensor Example 140 of Workspace Data of Worker's View and Area Corresponding to It Example of data format when saving part model and workspace model in point cloud data format Image of equipment as point cloud data Example of data format when saving part model and workspace model in polygon format Example of data format of work manual data Flow chart showing the procedure of work support Illustration of estimation of position and orientation Flowchart of processing executed by the confirmation state determination program Explanatory drawing of the process to identify the area on the part model that lacks the work space data Explanatory drawing of the process for determining the direction that the operator should see An example of a method of presenting a position and a direction to be viewed to an operator Configuration diagram of a work support system according to a second embodiment of the present invention Example of data format of work history Sequence diagram showing a flow of a process executed in the second embodiment

  Hereinafter, a work support system according to an embodiment of the present invention will be described with reference to the drawings. The work support system according to the present embodiment, for example, provides information such as work procedures to a worker performing an assembly work to support the work.

  FIG. 1 shows an example of a hardware configuration of an information processing apparatus for realizing a work support system according to a first embodiment to which the present invention has been applied.

  1, the information processing apparatus includes a central processing unit 101, an input device 102, an output device 103, and a storage device 104.

  The central processing unit 101 executes various programs.

  The input device 102 is a device for inputting data on a space including a facility for performing a work or an object as work space data. In the present embodiment, the device 102 that inputs the work space data may be the depth sensor 204 that acquires a distance image in which the distance to the target is a pixel value. The input device 102 may include an input device in a general computer such as a keyboard, a mouse, and a touch panel.

  The output device 103 is a device that presents information to an operator, and may be a head-mounted display device called an HMD (head-mounted display) 205.

  The storage device 104 stores a work management program 105, a position and orientation estimation program 106, a confirmation state determination program 107, and a display content generation program 108. Each program is executed by the central processing unit 101 to realize the functions described below.

  The storage device 104 further includes work manual data 800, a part model 120, a work space model 130, and work space data 140.

  The work management program 105 is a program that controls the execution of the position and orientation estimation program 106, the confirmation state determination program 107, and the display content generation program 108, and manages information presented to the worker. The work management program 105 may output the work procedure of each work process based on the work manual data 800, for example. When the end condition of one work process is satisfied, the work management program 105 may output the work procedure of the next work process.

  The position and orientation estimation program 106 is a program for estimating the positional relationship between the depth sensor 204 and the work space by obtaining the correspondence between the work space data 1400 and the work space model 1300. The position and orientation estimation program 106 estimates the position and orientation of the depth sensor 204, and matches the three-dimensional data of the mounted body and the parts with the three-dimensional data of the work space based on the estimation result.

  The confirmation state determination program 107 is a program for estimating a portion where the worker has confirmed the object based on the positional relationship between the depth sensor 204 and the work space estimated by the position and orientation estimation program 106. For example, the check state determination program 107 compares the three-dimensional data of the attached body and the part with the three-dimensional data of the work space acquired by the depth sensor 204, and determines whether or not the end condition of the visual check work in each work process is satisfied. May be determined. The end condition of the visual check operation may be such that a difference between the three-dimensional data of the attached body and the part and the three-dimensional data of the work space is equal to or less than a predetermined threshold.

  The display content generation program 108 is a program that determines a position and a direction to be viewed by an operator based on the result of the confirmation state determination program 107 and generates information to be presented to the operator. For example, when the confirmation state determination program 107 determines that the termination condition of the visual confirmation work is not satisfied, the display content generation program 108 may output guidance for satisfying the termination condition. This guidance may indicate the viewpoint and the direction of the line of sight seen by the worker. The viewpoint seen by the worker may be an area that is in the three-dimensional data of the attached body and the component but not in the three-dimensional data of the work space.

  The part model 120 is a three-dimensional model that represents a part to be attached to an object. The part model 120 can be represented in the form of point cloud data. Further, the component model 120 may be a three-dimensional model using polygons in order to present the location and the orientation of the component to the operator using computer graphics (CG) technology. A polygon can be converted into a point cloud model by dividing it into a predetermined size and extracting its vertices. Component models of a plurality of components may be registered in the storage device 104 in advance.

  The work space model 130 is a three-dimensional model representing the entire space in which work is performed. The work space model 130 includes a three-dimensional model of the work to be mounted in the work space. The work space model 130 can be expressed in the form of point cloud data, like the component model 120. The storage device 104 may store a plurality of workspace models 130 corresponding to a plurality of workspace states in the form of point cloud data.

  The work manual data 800 indicates a work procedure of each work process for one or more work processes. For example, the work manual data 800 includes information on work processes and work procedures of each work process, information on components used in the work, and information on the position and orientation of the components in the work space model.

  FIG. 2 shows an example of a use scene of the work support system in the present embodiment.

  In FIG. 2, reference numeral 201 denotes a user terminal. The user terminal 201 is, for example, a portable terminal that performs various processes of the work support system, and may be a small computer, a tablet terminal, a smartphone, or the like. The user terminal 201 is provided with a touch panel 202, and the work support system may operate according to a user operation received by the touch panel 202. For example, the touch panel 202 receives a user operation when starting or ending the system, or when returning to the original work process when the system has erroneously transitioned to the next work process. In the present embodiment, the information processing device in FIG. 1 may be the user terminal 201. That is, in the present embodiment, the user terminal 201 implements the functions of the work management program 105, the position and orientation estimation program 106, the confirmation state determination program 107, and the display content generation program 108.

  Reference numeral 203 denotes a helmet worn by a worker during work. The helmet 203 includes, for example, a depth sensor 204 for inputting working space data as the input device 102 and a head-mounted display (HMD) 205 as a head-mounted display device as the output device 103.

  The depth sensor 204 is a three-dimensional data acquisition device that acquires three-dimensional data of a work space seen by the worker while the worker is working. The depth sensor 204 acquires the work space data 140 representing the work space of the line of sight of the working worker. Therefore, the depth sensor 204 may be installed in the HMD 205. Alternatively, the depth sensor 204 may be installed at an arbitrary location, such as the shoulder or the chest of the worker, at which information on the target space can be obtained.

  The distance image acquired by the depth sensor 204 can be converted into point group data representing information on the target space as a set of points in a three-dimensional space by using the focal length and the center position of the sensor. The work space data 140 in the present embodiment uses data expressed as point cloud data. As the depth sensor 204, a sensor that can also acquire a normal color image can be used. In this case, it is possible to acquire the point cloud data to which the color information is added. In this embodiment, point group data to which color information is added is used.

  Reference numeral 206 denotes equipment provided in the work space, and components 207, 208, 209, and 210 for mounting the equipment 206 are mounted. Although FIG. 2 shows a state in which the attachment has already been completed, the parts 207 to 210 are not installed on the facility 206 at the start of the work.

  In the present embodiment, a case in which parts are attached to the equipment 206 in FIG. 2 will be described as an example. Can be applied to the work.

  FIG. 3 shows an example of a format of the work space data 140 acquired by the depth sensor 204.

  The work space data 140 has a number of points 1401, point position coordinates 1402, and point color information 1403 as data items. The work space data 140 has point position coordinates 1402 and color information 1403 for the number of points indicated by the point number 1401.

  The position coordinates 1402 can be represented by a combination of generally used coordinate values on the x-axis, y-axis, and z-axis. The position coordinates 1402 may be expressed in another coordinate system.

  The color information 1403 can be represented, for example, by a combination of values of three primary colors (red, green, and blue). The color information 1403 may be represented using another color space.

  FIG. 4A shows a view 401 of the worker, which is a work space viewed by the worker through the HMD 205. The worker's view 401 includes a part 209 and a part of the equipment 206.

  FIG. 4B shows a drawing based on the work space data 140 (point cloud data) of the area corresponding to the worker's view 401. As shown in FIG. 4B, when the point cloud data acquired from the depth sensor 204 is displayed on the space, the surfaces of the equipment 206 and the component 209 in the work space are represented as a set of points.

  FIG. 5 shows an example of a data format when the part model 120 and the work space model 130 are stored in the form of point cloud data. The part model 120 and the work space model 130 may have a common format. Here, the part model 120 and the work space model 130 in the form of point cloud data are referred to as a three-dimensional model (point cloud data) 500.

  The three-dimensional model (point group data) 500 has a name 501, the number 502 of points, position coordinates 503 of points, and color information 504 of points as data items. The three-dimensional model (point group data) 500 has position coordinates 503 and color information 504 of a plurality of points.

  The name 501 is a name of a part or a work space, and is an identifier. Any character string can be used as the name 501.

  The number of points 502 is the number of points constituting a three-dimensional model of a part or a work space.

  The position coordinates 503 of the points are the position coordinates of points constituting a three-dimensional model of a part or a work space. The method of expressing the position coordinates 503 of the point may be the same as that of the position coordinates 1402.

  The point color information 504 is the color information of points forming a three-dimensional model of a part or a work space. The method of expressing the color information 504 of the point may be the same as that of the color information 1403.

  FIG. 6 shows an equipment image 601 when the work space model 130 has point cloud data indicating the equipment 206 of FIG. 2 in a three-dimensional model (point cloud data) 500 format and displays it in the space.

  FIG. 7 shows an example of a data format when the component model 120 and the work space model 130 are stored in a polygon (polygon) format. The part model 120 and the work space model 130 may have a common format. Here, the component model 120 and the work space model 130 in the polygon format are referred to as a three-dimensional model (polygon data) 700.

  The three-dimensional model (polygon data) 700 has, as data items, a name 701, a polygon number 702, a polygon vertex number 703, a vertex position coordinate 704, a normal direction 706, and polygon color information 707.

  The name 701 is a name given to distinguish each part or work space, and is an identifier. An arbitrary character string can be used as the name 701.

  The number 702 of polygons is the number of polygons constituting a part or a work space. The three-dimensional model (polygon data) 700 has the number of polygon vertices 703 to the polygon color information 707 by the number of polygons 702.

  The number of vertices 703 of the polygon is the number of vertices constituting the polygon, and a numerical value such as 3 when the polygon is a triangle and 4 when the polygon is a quadrangle is registered. The three-dimensional model (polygon data) 700 has vertex position coordinates 704 corresponding to the number 703 of polygon vertices.

  The vertex position coordinates 704 are the position coordinates of each vertex constituting the polygon. The vertex position coordinates 704 have data corresponding to the number of vertices 703 of the polygon. The method of expressing the position coordinates may be the same as the position coordinates 1402.

  The normal direction 706 is information on a vector indicating the direction of the normal of the polygon. The normal direction 706 can be represented by values of components of the vector in the x-axis, y-axis, and z-axis directions.

  The polygon color information 707 indicates the color of the polygon. The expression method of the polygon color information 707 may be the same as that of the color information 1403.

  FIG. 8 shows an example of the data format of the work manual data 800.

  The work manual data 800 includes information on work procedures of each process for each of a plurality of work processes constituting the work, information on components to be attached to the attached body in the work, and information on the position and orientation of the components in the work space model. Is included.

  The work name 801 is a name given to distinguish each work when there are a plurality of works, and is an identifier. An arbitrary character string can be used as the work name 801.

  The number 802 of operation steps is the number of steps constituting an operation.

  The work process name 803 is the name of each work process. An arbitrary character string can be used for the name 803 of the work process.

  The work process contents 804 are contents of work processes performed in each work process. An arbitrary character string can be used as the content 804 of the work process. The work process contents 804 can include an image or a moving image indicating a specific work procedure.

  The work space model name 805 is the name of the work space model 130 related to each work process. Since the work space changes as the work progresses, the data format shown in FIG. 8 allows a different work space model 130 to be designated for each work process.

  The part model name 806 is the name of the part model of the part attached in each work process.

  The position and orientation 807 of the component is information indicating the position and orientation of the component having the component model name 806, and includes information on the position and orientation. The position may be a combination of commonly used coordinate values on the x-axis, y-axis and z-axis. The orientation can be represented by angles of rotation about the x, y, and z axes. As a method of expressing the position and the orientation, any expression may be used as long as it has a similar function.

  Further, in the format shown in FIG. 8, only one component (component model name 806 and component position and orientation 807) is targeted in each work process, but a plurality of components may be registered.

  The procedure of work support performed by the work support system according to the present embodiment will be described with reference to the flowchart of FIG.

  In step 901 of FIG. 9, the work management program 105 reads information on the work process selected by the worker from the work manual data 800 into the system. As a method of selecting a work process, for example, a list of work processes may be displayed on the screen of the user terminal 201 so as to be selectable using a touch panel or the like.

  In step 902, the content of the work process in the read work process is set as a work procedure to be implemented.

  In step 903, the system state is set to "working".

  In step 904, the work management program 105 acquires data (point cloud data) of the work space viewed by the worker from the depth sensor 204.

  In step 905, the posture estimation program 106 estimates the position and posture of the depth sensor 204 with respect to the work space model 130. The posture estimation program 106 may be started by the work management program 105.

  The posture estimation program 106 uses the work space data 140 captured in step 904 and the point cloud data of the work space model 130 corresponding to the work space model name 805 of the work process under execution to determine the position and posture of the depth sensor 204. May be estimated. For example, the position and orientation estimation program 106 determines that the work space data 140 acquired from the depth sensor 204 and the point group data of the work space model 130 corresponding to the work space model name 805 of the work process being executed best match. Then, the position and orientation of the depth sensor 204 may be estimated by obtaining a transformation matrix for performing translation and rotation of one point group data.

  With reference to FIG. 10, the estimation of the position and orientation will be specifically described.

  FIG. 1A shows a work space 1001 drawn based on the work space data 140 acquired from the depth sensor 204. FIG. B is a work space model 601 in which point group data of the work space model 130 corresponding to the work space model name 805 of the working process being executed is drawn.

  The position and orientation estimation program 106 obtains, for example, a transformation matrix for translating and rotating the work space 1001 so that the work space 1001 matches the corresponding location 1002 in the work space model 601. As a method of obtaining a transformation matrix in which the work space 1001 and the corresponding portion 1002 best match, a well-known method of aligning point cloud data, for example, an ICP (Iterative Closest Point) algorithm (S. Rusinkiewicz and M. Levoy, “Efficient Variants of the ICP Algorithm”, Third International Conference on 3D Digital Imaging and Modeling, 2001) and NDT (Normal Distributions Transformation) (P. Biber and W. StraBer, “The Normal Distributions Transform: A New Approach to Laser” ScanMatching Proceedings of the 2003 IEEE / RSJ International Conference on Intelligent Robots and Systems, 2003) can be used.

  As a method of estimating the position and orientation of the depth sensor 204, any other method may be used as long as a method of obtaining a rotation matrix that best matches the two point cloud data.

  Returning to FIG. 9, in step 906, the work management program 105 determines whether the system state is “working” or “confirming”. If the system state is “working”, the process proceeds to step 907;

  In step 907, the work management program 105 may extract information on the contents of the work process being performed from the work manual data 800 and display the information on the HMD 205. In this case, the displayed information may be the name 803 of the work procedure and the contents 804 of the work process.

  At this time, based on the part model 120 of the part model name 806, the position and posture 807 of the part, and the position and posture of the depth sensor 204 estimated in step 905, the work management program 105 The CG image of the component may be displayed on the HMD 205 so as to be superimposed on the component in the work space. Such display on the HMD 205 can be performed, for example, by using a technique well known as Augmented Reality.

  In step 908, the work management program 105 determines whether or not the worker has given an instruction to start checking. Here, the confirmation is a confirmation visually performed by an operator as to whether or not the work of the process is completed as specified after the work procedure of each process is completed. When the condition for terminating the visual check is satisfied, the process proceeds to the next operation step in step 914.

  The confirmation start instruction is, for example, to display a confirmation start button on the screen of the user terminal 201, and to press the displayed button using a keyboard, a mouse, a touch panel, or the like provided as the input device 102. Can be performed. Alternatively, any method that instructs the system, such as a method of pressing a predetermined keyboard or button, or a method of detecting a predetermined gesture from point cloud data acquired by the depth sensor 204, may be used. However, it can be used.

  If the operator has not instructed to start confirmation, the process returns to step 904 as it is.

  If the worker has given an instruction to start confirmation, the work management program 105 sets the system state to “under confirmation” in step 909, and then returns to step 904.

  If the system state is “confirming” in step 906, the process proceeds to step 910.

  In step 910, the confirmation state determination program 107 started by the work management program 105 determines the confirmation state. For example, the confirmation state determination program 107 determines whether the state of the confirmation performed by the worker satisfies the condition for terminating the work process. When it is determined that there is an area of a component that has not been confirmed, the confirmation state determination program 107 sets the area as an area that needs to be confirmed by the operator, that is, a position and a direction that the operator views.

  The processing executed by the confirmation state determination program 107 will be described with reference to FIGS.

  The check state determination program 107 extracts point cloud data corresponding to components related to the work process being performed from the work space data 140 acquired from the depth sensor 204 in step 1101.

  In this process, first, the confirmation state determination program 107 may specify an area of a component in the work space model 130. This part area may be specified by performing coordinate conversion on the part model 120 specified by the part model name 806 of the work manual data 800 using information specified by the position and orientation 807, for example. Next, the confirmation state determination program 107 may use the position and orientation of the depth sensor 204 estimated in step 905 to perform coordinate transformation of the work space data acquired by the depth sensor 204 into the same coordinate system as the work space model. . Then, the confirmation state determination program 107 may extract the point cloud data of the component area from the coordinate-converted work space data. If the component region has already been extracted, the newly extracted point cloud data may be added (combined) to the point cloud data of the extracted component region.

  This will be described using a specific example shown in FIG.

  In the figure, reference numeral 1201 denotes a work space in which the work space data 140 is transformed into the same coordinate system as the work space model 130 and then the point group data of the work space data 140 is drawn. Reference numeral 1202 denotes a component area specified in the work space 1201 by the position and orientation 807 of the component. At this time, point cloud data 1203 included in the component area 1202 is extracted as point cloud data for the component.

  The extracted point cloud data has been coordinate-transformed into the same coordinate system as the work space model. Therefore, by adding the point data further acquired by the depth sensor 204 to the extracted point group data, the point group data can be synthesized. Note that, in the combined point cloud data, points may overlap at the same position, or points may be densely packed in a specific range. May be converted to point cloud data in which points are distributed.

  Returning to FIG. 11, in step 1102, the confirmation state determination program 107 determines whether the number of points in the component area extracted in step 1101 is equal to or greater than a predetermined value. For example, the confirmation state determination program 107 determines whether the difference between the extracted point group data of the component and the component model is equal to or smaller than a predetermined value. For example, the confirmation state determination program 107 may determine whether there is an area that exists in the part model but does not exist in the point cloud data of the target part acquired by the depth sensor 204. This determination may be a termination condition of the visual check operation.

  This process may be performed by comparing the point cloud data obtained by combining the point cloud data of the component area extracted in step 1101 with the point cloud data of the component area that has been extracted before that, with the component model.

  If the part model is point cloud data, the determination in step 1102 may be performed as follows. For example, the confirmation state determination program 107 obtains the point of the part model closest to each point in the extracted point group data of the part, and calculates the distance between the points. Then, the confirmation state determination program 107 extracts points on the part model where the distance between the points is larger than a predetermined threshold. Thereby, the area where the extracted points are gathered may be the area on the part model where the point group data acquired by the depth sensor 204 is insufficient.

  If the component model is in the form of a polygon, the determination in step 1102 may be made as follows. For example, the confirmation state determination program 107 searches for a polygon on the part model in which the perpendicular drawn from each point in the extracted part point data is the shortest, and determines the distance of the perpendicular and the intersection of the perpendicular with the polygon. Ask. Then, the confirmation state determination program 107 extracts the intersection of the perpendicular and the polygon at which the distance of the perpendicular is larger than a predetermined threshold. Similarly to the case of the point cloud data, the area where the extracted points are gathered may be an area on the part model where the point cloud data acquired by the depth sensor 204 is insufficient.

  In FIG. 12, reference numeral 1204 denotes a part model stored as point cloud data. At this time, when the part model 1204 and the point cloud data 1203 are compared by the above method, the area of the part indicated by 1205 is specified as the area where the point cloud data is insufficient. If the size of the specified area is larger than a predetermined threshold, it is determined that the point cloud data for the component has not been sufficiently acquired. Alternatively, the ratio between the size of the area in which the point cloud data is insufficient and the surface area of the component is obtained. If the obtained ratio is larger than a predetermined threshold, it is determined that the point cloud data for the component is not sufficiently acquired. May be.

  Returning to FIG. 11, when it is determined in step 1103 that the point cloud data for the component is insufficient in step 1102, the process proceeds to step 1104. If the point cloud data for the part has been sufficiently acquired, the process ends.

  In step 1104, the viewpoint and the direction of the line of sight of the worker are determined for the region where the point cloud data is insufficient.

  The position (viewpoint) to be viewed by the operator may be, for example, the center or the center of gravity of the area where the point cloud data is insufficient. If the corresponding area is larger than a predetermined threshold, the corresponding area may be divided into smaller areas, and the center, the center of gravity, or the like may be obtained for one of the obtained small areas. As a method of dividing a region, for example, in the case of dividing into two regions, the region may be mapped on a specific coordinate axis, and may be divided on a plane passing through an intermediate point of the mapped region.

  After determining the position to be viewed by the operator, the confirmation state determination program 107 determines the viewing direction of the operator.

  With reference to FIG. 13, a method of determining a direction (a direction of a line of sight) to be viewed by an operator will be described.

  In FIG. 13, it is assumed that reference numeral 1301 denotes an area where point cloud data is determined to be insufficient, and reference numeral 1302 denotes a position determined as a position to be viewed by an operator. Here, the confirmation state determination program 107 may set the inverse vector of the normal line of the area 1301 as the reference direction 1303 of the direction in which the operator should look at the position 1302.

  Next, the confirmation state determination program 107 may verify whether or not a straight line in the reference direction 1303 intersects a polygon constituting another component. This makes it possible to determine whether there is another component that blocks the view when the operator views the position 1302 from the reference direction 1303. Alternatively, the confirmation state determination program 107 verifies whether a point constituting the straight line in the reference direction 1303 and another component is within a predetermined threshold or less, thereby determining whether there is a component that may obstruct the view. May be determined. In the case of FIG. 13, when viewing the position 1302 from the reference direction 1303, it is determined that another component 1304 blocks the view. If it is determined that there is no other part that blocks the view, the reference direction 1303 may be determined as the direction in which the worker should look at the position 1302. When it is determined that there is another component that blocks the view, the confirmation state determination program 107 performs the same method as described above for a direction (for example, 1305 or 1306 in the drawing) whose angle is changed around the reference direction 1303. Then, the presence or absence of another component that blocks the view when viewing the position 1302 may be determined, and the direction in which it is determined that there is no other component that blocks the view may be determined as the direction in which the worker should view the position 1302. .

  As a method of changing the angle of the line of sight, a width for changing the angle may be set in advance, and the angle may be converted using the set width. In addition, the determination of the presence or absence of another component that blocks the view may be performed when there is no polygon or point within a predetermined range from the straight line indicating the viewing direction, and it may be determined that there is no other component that blocks the view. . In the case of FIG. 13, the direction 1305 in which no other part that blocks the view is present may be selected.

  Returning to FIG. 9, in step 911, the display content generation program 108 is activated. Then, the display content generation program 108 generates a guidance image indicating the position and the viewing direction of the area to be checked by the operator for confirmation. The guidance image may be generated based on the region of the component that is not viewed by the operator and the direction in which the region is viewed, determined in step 910.

  The guidance image may be an image indicating a position and a direction on the HMD 205 so as to overlap an actual part viewed by the operator through the HMD 205 by using a well-known augmented reality technology. The guidance image may include characters.

  The guidance image may be, for example, an image indicating the position of the attached body or the component and the direction of the line of sight seen by the worker in a work space viewed from the worker through the HMD 205. This may be realized using augmented reality technology. Further, the guidance image may be an image indicating the position of the attached body or the component and the direction of the line of sight seen by the worker in the CG image including the attached body and the component.

  For example, as shown in FIG. 14A, the display content generation program 108 converts the image 1401 in which an arrow indicating the position and direction and a message urging confirmation are combined into a work space (view of the worker) actually viewed by the worker into the HMD 205. May be displayed. Alternatively, as shown in FIG. 14B, instead of the message, an image 1403 of eyes, a face, a camera, or the like that reminds the viewer may be presented in combination with the CG. Alternatively, animation using an arrow or the like that moves to a position to be viewed can be used. Further, in addition to performing superposition using actual components based on augmented reality, a CG image 1404 as shown in FIG. 14C may be displayed. The CG image 1404 may display a position and a direction to be viewed by an operator in a CG of a work space and a part generated based on the part model 120 and the work space model 130. The CG image 1404 may be presented at a position in the HMD 205 that does not obstruct the field of view of the worker, for example, at the upper left or right.

  In this case, by using the augmented reality technology, the display content generation program 108 allows the CG image of the part model and the work space model to be viewed in real time from the same viewpoint as when the worker is looking at the actual parts and the work space. It may be generated. As a method of presenting the position and direction of the region to be viewed to the operator, any method may be used as long as the method can recognize the position and direction of the region to be viewed.

  Furthermore, the display content generation program 108 may detect the hand of the worker or the tool held by the worker in the work space data acquired by the depth sensor 204. If the detected worker's hand or tool is located at a position where it can be seen overlapping with the target component, or if the detected hand or tool exists within a predetermined range, the display content generation program 108 sets the CG A presentation method using an image may be adopted, and in other cases, a presentation method using augmented reality technology may be adopted. This is because the hand or tool of the worker may shield or block the position of a part or the like to be viewed by the worker.

  When there is an object that shields the component, or when the distance between the operator and the component is short, the estimation of the position and orientation in step 905 in FIG. 9 is often not completed properly. Therefore, a method using augmented reality technology may be used when the position and orientation estimation is appropriate, and a method using a CG image may be used otherwise.

  Here, the determination as to whether or not the estimation of the position and orientation has been appropriately completed may be made, for example, as follows. That is, after the workspace data acquired from the depth sensor 204 using the estimated position and orientation is coordinate-transformed to the coordinate system of the workspace model, the distance between the transformed workspace data and the workspace model is determined in advance. If it is within the determined threshold, the position and orientation estimation may be determined to be appropriate. As the CG image when the estimation of the position and orientation is not appropriate, for example, a CG image generated using the position and orientation obtained when the estimation of the position and orientation is finally completed properly, or Alternatively, a CG image generated using a predetermined position and orientation may be used.

  The display content generation program 108 may switch between the presentation method using augmented reality shown in FIGS. 14A and 14B and the presentation method using a CG image shown in FIG. 14C. For example, (1) when the distance between the worker and the component is equal to or less than a predetermined value, (2) when the position of the attached body or the component viewed by the worker is or is likely to be shielded, (3) expansion In the image using the sense of reality, when the display of the position and the direction of the line of sight of the attached body or the part seen by the operator does not fit on the HMD 205, (4) the attached body and the first working process after the coordinate conversion When the distance between the three-dimensional data of the part and the three-dimensional data of the work space is equal to or longer than a predetermined value, a method using a CG image may be used.

  Returning to FIG. 9, in step 912, if it is determined in step 910 that the confirmation has been completed (the termination condition has been satisfied), the process proceeds to step 913. If it is determined that the confirmation is not completed (the termination condition is not satisfied), the process returns to step 904.

  Before proceeding to step 913, the HMD 205 may be presented with information notifying the worker that the confirmation has been completed or proceeding to the next procedure. In this case, the operator's response to the notification may be further obtained from the input device 102, and the process may proceed to step 913 after obtaining the response.

  In step 913, the work management program 105 determines whether or not all work processes have been completed. When all the work steps have been completed, the processing is terminated. When ending the processing, the work space data (including the point cloud data of each component) 140 acquired at the time of confirmation may be stored in the storage device 104. Further, the start time, confirmation start time, confirmation end time, etc. of each work procedure may be stored together.

  If all the work processes have not been completed, the process proceeds to the next work process in step 914, the system state is set to “working” in step 915, and the process returns to step 904.

  Note that the present system may have a function of determining a timeout. That is, if the confirmation of the target component has not been completed in step 912 and the elapsed time from the start of the confirmation is equal to or longer than a predetermined time, the work management program 105 forcibly performs the processing after step 903. It may be. At this time, the work management program 105 may present information for instructing the worker to perform the work again to the HMD 205. Further, a response of the worker to the instruction to redo the work may be obtained by the input device 102, and the process may return to step 903 after obtaining the response.

  Further, the present system may have a function of forcibly resetting the work space data and the part model when the degree of conformity is low. That is, in step 912, when the distance between the work space data and the part model is larger than a predetermined threshold, the work management program 105 may forcibly perform the processing of step 903 and subsequent steps. At this time, the work management program 105 may present information for instructing the worker to perform the work again to the HMD 205. Further, a response of the worker to the instruction to redo the work may be obtained by the input device 102, and the process may return to step 903 after obtaining the response.

  In the present embodiment, an example in which the HMD 205 is used as the output device has been described, but the user terminal 201 may be used as the output device. For example, it is preferable to use the HMD 205 as an output device when the purpose is to support the worker during the work, and to use the user terminal 201 as the output device when the purpose is to confirm the work result after the work. Is preferred.

  When the user terminal 201 is used as an output device, the depth sensor 204 may be integrated with the user terminal 201 or may be mounted on the user terminal 201. Various processes and display methods may be the same as the processes and display methods on the assumption that the HMD is used.

  Further, in the present embodiment, the description has been made with respect to the mounting work, but the present invention can also be applied to a more general manufacturing process, construction / renovation work, confirmation of completion of check work in inspection and the like.

  Furthermore, as a method of presenting information to a worker, the method of switching between the method using CG images and the method using augmented reality can be applied not only at the time of checking work but also at the time of normal work such as mounting. It is.

  According to the present embodiment, when the worker confirms the work result, the part where the confirmation is insufficient is automatically detected, and the worker's line of sight is guided to the detected part, so that the work result can be surely confirmed. It is possible to assist the worker so that the confirmation can be performed. Furthermore, according to the situation of the worker and the work space when checking the work result, it is possible to present information that allows the worker to easily understand the situation.

  Next, a work support system according to a second embodiment of the present invention will be described with reference to FIGS.

  FIG. 15 is a configuration diagram of a work support system according to the second embodiment.

  The work support system according to the present embodiment includes a worker terminal 1501 and a work information management terminal 1517, which are connected via a network 1516. Then, the work information management terminal 1517 manages data used in the present system. The worker terminal 1501 obtains data from the work information management terminal 1517 and supports the worker. In the following description, the description of the same function or configuration as that of the first embodiment may be omitted.

  The worker terminal 1501 includes a central processing unit 1502, an input device 1503, an output device 1504, and a storage device 1506. These are the same as the central processing unit 101, the input device 102, the output device 103, and the storage device 104 in FIG.

  The storage device 1506 stores a transmission / reception program 1507, a work management program 1508, a position and orientation estimation program 1509, a confirmation state determination program 1510, a display content generation program 1511, a component model 1513, a work space model 1514, Manual data 1515 is stored.

  The work management program 1508 has the same functions as the work management program 105 in the first embodiment, and further has a function of transmitting and receiving data to and from the work information management terminal 1517.

  The position and orientation estimation program 1509, the confirmation state determination program 1510, and the display content generation program 1511 are respectively the same as the position and orientation estimation program 106, the confirmation state determination program 107, and the display content generation program 108 in the first embodiment.

  The part model 1513, the work space model 1514, and the work manual data 1515 are respectively similar to the part model 120, the work space model 130, and the work manual data 800 in the first embodiment.

  The worker terminal 1501 further includes a communication device 1505. The communication device 1505 is a device for exchanging information with the work information management terminal 1517.

  The work result 1512 stores information on the work time such as the work start time and the confirmation completion time, and work space data (point cloud data) obtained as a result of checking each component.

  Further, the transmission / reception program 1507 stored in the storage device 1506 receives work manual data relating to the work selected by the worker and component models and work space models accompanying the work manual data from the work information management terminal 1517, and receives the work space model 1513, After being stored in the part model 1514 and the work manual data 1515, and after completion of the work being performed, the information on the work time and the point cloud data of each part stored in the work result 1512 are transmitted to the work information management terminal 1517. It is a program to do.

  The work information management terminal 1517 is a work information management terminal that collectively manages information related to work procedures and component models and work space models associated therewith.

  Reference numeral 1518 denotes a central processing unit for executing various programs.

  Reference numeral 1519 denotes an input device in a general computer such as a keyboard, a mouse, and a touch panel for controlling various programs.

  Reference numeral 1520 denotes an output device such as a display used in a general computer.

  Reference numeral 1521 denotes a communication device for exchanging information with the worker terminal 1501.

  The storage device 1522 stores a transmission / reception program 1523, a work procedure registration program 1524, a work history browsing program 1525, a part model 1526, a work space model 1527, work manual data 1528, and a work history 1529.

  The transmission / reception program 1523 transmits the work manual data and the part model and the work space model accompanying the work manual data to the worker terminal 1501, and also transmits the information on the work time and the point cloud data of each part transmitted from the worker terminal 1501. The program is received and stored in the work history 1529.

  The work procedure registration program 1524 is a program for adding, modifying, and deleting work manual data and its associated part models and work space models.

  The work history browsing program 1525 is a program for displaying the contents of the work history 1529 on the output device 1520, confirming the contents of the work history, and adding / modifying information.

  The part model 1526, the work space model 1527, and the work manual data 1528 are similar to the part model 120, the work space model 130, and the work manual data 800, respectively.

  The work history 1529 stores time information of each work procedure in work performed in the past and work space data acquired at the time of confirmation as work execution history. The data format of the work history 1529 is the same as the work result data 1600 stored in the work result 1512.

  FIG. 16 shows an example of the data format of the work result data 1600.

  Reference numeral 1601 denotes an operation name, which is an identifier. An arbitrary character string can be used as the work name 1601.

  Reference numeral 1602 denotes the date and time when the work was performed. The work execution date and time 1602 has a date and a time. The time may include one or both of the start time and the end time of the task.

  Reference numeral 1603 denotes the number of work items.

  Reference numeral 1604 denotes a completion time of the work process.

  Reference numeral 1605 denotes work space data at the time when the confirmation of the work process is completed. The point cloud data of the work space acquired at the time when the visual confirmation of the work of each process is completed may be used.

  The work result data 1600 includes work process completion times 1604 and work space data 1605 for the number 1603 of work items.

  Although only the completion time of each work process is described in the data format of FIG. 16, the start time of the work process and the start time of the confirmation may be included.

  FIG. 17 is a sequence diagram illustrating a processing flow in the present embodiment.

  When the worker starts the system (1701), the worker terminal 1501 requests the work information management terminal 1517 for a list of information on the work using the work management program 1508 (1702).

  The work information management terminal 1517 receives the request from the worker terminal 1501, and transmits a list of information on the work to the worker terminal 1501 in response to the request (1703).

  The worker's terminal 1501 receives the list of information on the work, and displays the received work list on the output device 1504 (1704).

  When the worker selects a work process to be performed (1705), the worker terminal 1501 stores work manual data on the work process selected by the worker and three-dimensional data attached to the work process on the work information management terminal 1517. Request the model (1706).

  The work information management terminal 1517 transmits the work manual data and the three-dimensional model based on the request from the worker terminal 1501 (1707).

  The worker terminal 1501 uses the work management program 1508 to display information about the first work process on the output device 1504 based on the received work manual data and the three-dimensional model (1708).

  When the worker notifies the start of confirmation of the first work process to the worker terminal 1501 (1709), the worker terminal 1501 uses the position and orientation estimation program 1509 and the confirmation state determination as in the first embodiment. By using the program 1510 and the display content generation program 1511, the confirmation work of the operator is supported. When the confirmation is completed, information on the second work process is displayed on the output device 1504 (1710).

  The worker is notified of the start of the n-th work step, which is the last work step (1711), and when the check is completed, the worker terminal 1501 displays a work completion notification on the output device 1504 ( 1712).

  Further, the worker terminal 1501 transmits time information such as a start time and an end time of each process and work space data including each part acquired at the time of confirmation to the work information management terminal 1517 as a work result (1713). ). The work information management terminal 1517 stores the received work result in the work history 1529.

  In the second embodiment, the work information management terminal 1517 performs only the management of the work manual data, the part model, and the work space model, but the work information management terminal 1517 performs part or all of the work support function. You may do it. In this case, the worker terminal 1501 transmits the point cloud data acquired from the input device 1503 to the work information management terminal 1517, and displays the display content received from the work information management terminal 1517 on the output device 1504. Only the work result 1512, the part model 1513, the work space model 1514, and the work manual data 1515 may be deleted from the worker terminal 1501.

  With the work support system according to the second embodiment, it is possible to support the worker as in the first embodiment. Further, in the second embodiment, the work manual data is centrally managed by the work information management terminal, and only the necessary work manual data is read into the worker terminal, thereby performing work support according to each worker. It becomes possible. In addition, by collecting the work histories on the work information management terminal, it becomes easy for anyone other than the worker to confirm the work results and to perform a comparative analysis of the work results.

105 Work management program 106 Position and orientation estimation program 107 Confirmation state determination program 108 Display contents generation program 120 Part model 130 Work space model 130 Work space model 140 Work space data 201 User terminal 202 Touch panel 203 Helmet 204 Depth sensor 205 Head mounted display (HMD)

Claims (9)

A storage device that stores, in the work space, three-dimensional data of the work to be mounted and parts to be mounted on the work to be mounted, and work manual data indicating work procedures of one or more work processes;
A three-dimensional data acquisition device that acquires three-dimensional data of a work space seen by the worker while the worker is performing work;
And a central processing unit, a work support system with a,
The central processing unit,
A process of outputting a work procedure of each work process based on work manual data stored in the storage device ;
Comparing the three-dimensional data of the attached body and the component related to the first work process stored in the storage device with the three-dimensional data of the work space acquired by the three-dimensional data acquisition device, A process of determining whether or not an end condition of a visual check operation of a work process is satisfied;
And a process of outputting guidance for satisfying the end condition when the end condition is not satisfied in the determination. The central processing unit further includes:
The position and orientation of the three-dimensional data acquisition device are estimated, and based on the result of the estimation, the three-dimensional data of the attached body and the component related to the first work process are matched with the three-dimensional data of the work space. Do the process to make
2. The ending condition of the visual check operation according to claim 1, wherein a difference between three-dimensional data of an attached body and a part related to the first operation process and three-dimensional data of the work space is equal to or less than a predetermined threshold. Work support system.   The work support system according to claim 2, wherein the guidance for satisfying the end condition indicates a viewpoint and a line of sight of the worker.   4. The work support system according to claim 3, wherein the viewpoint viewed by the worker is an area that is in the three-dimensional data of the attached body and the component related to the first work process but is not in the three-dimensional data of the work space. Further comprising a head mounted display worn by the worker,
Guidance for meeting the termination conditions is:
In a work space that is visible to the worker through the head-mounted display, a first display mode in which a position and a line of sight of the attached body or the component viewed by the worker are displayed on the head-mounted display,
A second display mode in which the position and the direction of the line of sight of the attached body or the part viewed by the worker in the CG image including the attached body and the part are displayed on the head mounted display by computer graphics. The work support system according to any one of claims 2 to 4, which is output in any one of display modes. The work support system according to claim 5, wherein the guidance is output in the second display mode when any of (1) to (4) is satisfied.
(1) When the distance between the worker and the component is equal to or less than a predetermined value. (2) When the position of the attached body or the component seen by the worker is shielded or there is an object that may be shielded. In the first display mode, when the display of the position of the attached body or the component and the direction of the line of sight seen by the worker does not fit on the head mounted display. (4) The first work is performed even if the matching is performed. When the deviation between the three-dimensional data of the attached body and the part related to the process and the three-dimensional data of the work space is equal to or more than a predetermined value. The central processing unit further includes:
The work support system according to claim 1, wherein when the end condition is satisfied in the determination, a process of outputting a work procedure of a second work process subsequent to the first work process is performed. The central processing unit further includes:
The work support system according to any one of claims 1 to 7, wherein when the end condition is satisfied in the determination, a process of storing the three-dimensional data of the work space in the storage device is performed. A storage device that stores, in the work space, three-dimensional data of a work piece to be mounted and parts to be attached to the work piece, and work manual data indicating work procedures of one or more work processes;
A three-dimensional data acquisition device that acquires three-dimensional data of a work space seen by the worker while the worker is performing work;
A central processing unit, a work supporting method performed by the work support system with a,
The central processing unit,
A process of outputting a work procedure of each work process based on work manual data stored in the storage device ;
Comparing the three-dimensional data of the attached body and the component related to the first work process stored in the storage device with the three-dimensional data of the work space acquired by the three-dimensional data acquisition device, A process of determining whether or not an end condition of a visual check operation of a work process is satisfied;
Outputting the guidance for satisfying the end condition when the end condition is not satisfied in the determination.