JP7374908B2 - Manufacturing support systems, methods, programs - Google Patents

Description

The present invention relates to a manufacturing support system, method, and program for supporting parts assembly work by a worker.

Traditionally, work support has been provided at production sites in the manufacturing industry by having workers wear light-transmissive head-mounted displays that display information such as work content and work procedure instructions superimposed on the actual scene (superimposed display). There are systems known to do this. For example, a technology has been proposed that supports inspection work by displaying a passing image (an image of an inspection object that has been determined to pass in the past) near the actual inspection object (real image) during product inspection. . It has also been proposed to improve work efficiency by enclosing a specific location with a circle to emphasize it or by displaying the wiring route for wiring materials in the process of assembling parts. By implementing these supports, the efficiency of visual inspection and assembly work can be improved (see Patent Documents 1 and 2).

International Publication No. 2017/033561 Japanese Patent Application Publication No. 2018-014218

By the way, at assembly work sites, work details are sometimes instructed using technical terms or codes that are difficult to decipher (combining multiple symbols to convey information), and beginners may not be able to understand the work details. There is also the aspect that it is difficult to even grasp accurately. Furthermore, in a work in which a plurality of parts are assembled to an assembly target, the layout of the parts may look different depending on the posture and orientation of the assembly target when viewed from the operator. Therefore, even if the assembled state of the parts appears to be different from the accepted image, it may actually be in an appropriate state. Furthermore, since multiple parts are checked at the same time, improper assembly may be overlooked during visual inspection. As described above, conventional work support systems have the problem that assembly errors are likely to occur during work in which a plurality of parts are assembled, making it difficult to improve work efficiency.

One of the purposes of this invention, which was created in view of the above-mentioned problems, is to provide a manufacturing support system, method, and program that can improve work efficiency regarding parts assembly work. In addition, this purpose is not limited to this purpose, and it is also possible to achieve effects derived from each configuration shown in "Details for Carrying Out the Invention" that will be described later, which cannot be obtained with conventional techniques. It can be positioned as a purpose.

(1) The disclosed manufacturing support system is a manufacturing support system for supporting parts assembly work by a worker. This system includes a photographing device that photographs a real scene within the visual field of the worker, a display device that displays a virtual image superimposed on the real scene, and smart glasses that are worn on the worker's head. The apparatus also includes a first database that defines a relationship between an assembly pattern including information on the assembly positions and orientations of the plurality of parts to be assembled and identification information corresponding thereto. Further, the second database includes a storage location of the plurality of parts to be assembled to the assembly object, a usage number that is the number of each of the parts, a relationship between the part number of each of the parts and the identification information. . Furthermore, a control device is provided that causes the display device to display the virtual image of the component so as to match the coordinate system of the actual scene within the visual field of the worker.

The control device is provided with an extraction section, an acquisition section, a recognition section, and a drawing section. The extraction unit extracts the identification information included in a work instruction sheet in which the contents of the assembly work are described from among the images taken by the photographing device. The acquisition unit acquires an assembly pattern including information on an assembly position and assembly direction of the component corresponding to the identification information, based on the first database. The recognition unit recognizes the assembly target from among images taken by the photographing device. The drawing unit draws virtual images of the plurality of parts in the positions and orientations at the time of completion when the plurality of parts are assembled to the assembly target. Further, the acquisition unit acquires, based on the second database, information on the storage location of the part and information on the number of parts corresponding to the identification information, and the part number and number corresponding to the identification information. The information on the number of parts used is acquired, the recognition unit recognizes the storage location from the image taken by the imaging device, and the drawing unit displays information on the number of parts in the vicinity of the storage location. At the same time, information on the part number and the number of parts used is drawn near the assembly target.

(3) It is preferable that the recognition unit recognizes a marker installed at the storage location in the photographed image.
(4) Preferably, the control device includes a determination section and a highlighting section. In this case, the determination unit determines whether the assembled state of the parts matches the assembly pattern based on the photographed image of the photographing device and the identification information. Further, the highlighting section highlights a portion where the assembled state of the parts does not match the assembly pattern so as to match the coordinate system of the actual scene within the field of view of the worker.

By extracting the identification information included in the work instructions and drawing a virtual image of the parts in the corresponding position and orientation, it is possible to eliminate the possibility that the type, number, and layout of parts to be assembled may differ depending on the work instructions. Even if the robot is not installed properly, it can show a model for assembly in the correct position and direction, and can support assembly work. Further, it is no longer necessary for the worker to memorize the assembly state of different parts for each work instruction, reducing assembly errors and improving productivity (work efficiency).

FIG. 2 is a diagram showing an application target of the manufacturing support system. FIG. 2 is a diagram for explaining the configuration of a manufacturing support system. FIG. 2 is a block diagram showing the hardware configuration of the manufacturing support system. FIG. 2 is a block diagram showing the software configuration of a manufacturing support program. (A) and (B) are examples of database configurations. (A) to (C) are diagrams for explaining display examples of virtual images of parts. (A) and (B) are diagrams for explaining display examples of the number of parts. It is a figure for explaining the example of a highlighted display of a test result. It is a flow chart for explaining a manufacturing support method in an assembly process. It is a flow chart for explaining a manufacturing support method in an inspection process. (A) and (B) are diagrams for explaining a modification of the manufacturing support system.

Hereinafter, a manufacturing support system, method, and program as embodiments will be described with reference to the drawings. The embodiments shown below are merely illustrative, and there is no intention to exclude the application of various modifications and techniques not specified in the embodiments below. Each structure of this embodiment can be modified and implemented in various ways without departing from the spirit thereof. Further, they can be selected or combined as necessary.

The manufacturing support system of this embodiment is a system that uses a wearable gadget to support parts assembly work by a worker 51, and is applied to the manufacturing process of an automobile 50 as shown in FIG. Here, support for the worker 51 to take out parts from the material storage shelf 53 and assemble them into the automobile 50 will be explained. Specific examples of parts that can be assembled include electrical components such as relays, switches, and lamps, as well as glass, bumpers, power trains, seats, and tires. Further, specific examples of parts to be assembled (devices to which parts are assembled) include relay boxes, switch boxes, instrument panels, doors, and vehicle bodies. The objects and types of parts to be assembled in parts assembly work are wide-ranging, and the manufacturing support system, method, and program of the present invention are applicable to all kinds of parts assembly work.

The content of the work support is the provision of visual information using smart glasses 10 worn on the head of the worker 51. As shown in FIG. 2, the smart glasses 10 are fixed to a helmet 52 of a worker 51. The smart glasses 10 include a display device 13 and a photographing device 16. The photographing device 16 is a camera that photographs the actual scene within the field of view of the worker 51, the work instructions 54, and the like. Further, the display device 13 is a device that displays a virtual image superimposed on the real scene within the field of view of the worker 51. The display method may be a method of projecting a virtual image onto the retina of the worker 51 (retinal projection type), a method of projecting a virtual image onto the surface of a half mirror (transmission type), or a method of projecting a virtual image on the surface of a half mirror (transmission type), or a method of projecting a virtual image onto the retina of the worker 51 (transmission type), or a method of projecting a virtual image on the surface of a half mirror (transmission type), or a method of projecting a virtual image onto the retina of the worker 51 (transmission type), or a method of projecting a virtual image on the surface of a half mirror (transmission type), or a method of projecting a virtual image on the surface of a half mirror (transmission type). It is also possible to use a method (non-transparent type) that displays a video that is a composite of a real scene and a virtual image.

In either method, various virtual images are displayed so as to match the coordinate system of the actual scene seen from the viewpoint of the worker 51. This virtual image is a perspective projection image (or a deformed version of the perspective projection image) that conforms to the coordinate system of the actual scene, and is rendered as a simple polygon model based on three-dimensional CAD (Computer-Aided Design) data of the part, for example. be done. Note that in existing display methods such as those disclosed in Patent Documents 1 and 2, the drawn object is a two-dimensional object. In contrast, in the display method of the present invention, the drawn objects are three-dimensional objects, and each drawn object is virtually arranged within the coordinate system of the actual scene.

The content of work support is controlled by an arbitrary control device (electronic computer, computer). The "electronic control device" here may be built into the smart glasses 10 or may be prepared separately from the smart glasses 10. For example, a system may be used in which a program is installed on a server 30 or a workstation on the network 19, and display contents calculated there are displayed on the smart glasses 10. In this case, as shown in FIG. 2, a server 30 connected to the network 19 via a repeater 36 may be used. Alternatively, a program that operates even when not connected to the network 19 may be installed in the smart glasses 10.

[1. Hardware configuration]
FIG. 3 is a block diagram showing the hardware configuration of a manufacturing support system realized by cooperation between the smart glasses 10 and the server 30. The smart glasses 10 include a recording medium drive 11 for reading a program recorded on a recording medium 18 (removable medium), a storage device 12, a display device 13, an input device 14, a communication device 15, a photographing device 16 (camera), A microphone 17 and an electronic control device 20 are provided. Similarly, the server 30 also includes a recording medium drive 31, a storage device 32, a display device 33, an input device 34, a communication device 35, and an electronic control device 40. The function of each element is substantially the same as the element with the same name built into the smart glasses 10, so the explanation will be omitted.

The recording medium 18 is, for example, a flash memory card, a semiconductor memory device compliant with the universal serial bus standard, an optical disk, or the like. The recording medium drive 11 is a reading device that reads information (programs and data) recorded and saved on the recording medium 18 . The storage device 12 is a memory device that stores data and programs that are retained for a long period of time, and includes, for example, non-volatile memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), and solid state drives. It will be done.

The display device 13 is one of the devices for providing visual information to the worker 51. In a retinal projection type device, the retina of the worker 51 serves as an imaging plane. On the other hand, transmissive and non-transmissive devices include half mirrors and displays placed within the field of view of the worker 51 (e.g., liquid crystal displays (LCDs) and organic electroluminescence displays (OELDs)). ) becomes the imaging plane.

The input device 14 is one of pointing devices for inputting input signals to the electronic control devices 20 and 40. As a specific example of the input device 14, devices such as a wireless keyboard and a wireless mouse can be connected to the smart glasses 10. Further, an input device 14 such as a push button or a physical key is provided on the temple (handle, side part) of the smart glasses 10. Furthermore, the microphone 17 is a device for inputting the voice of the worker 51 as an input signal.

The communication device 15 is a device that manages communication with the outside of the smart glasses 10, and is a device for performing wireless communication with the server 30 via the network 19, for example. The repeater 36 shown in FIG. 2 can be considered to be included in the communication device 15. When input/output devices conforming to the short-range wireless communication standard are connected to the smart glasses 10, those input/output devices are connected via the communication device 15.

The photographing device 16 is a digital video camera incorporating an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The photographing device 16 is attached to the rim (the periphery of the lenses) or the bridge (the connecting part between the left and right lenses) of the smart glasses 10, and functions to photograph the field of view of the worker 51 wearing the smart glasses 10. Information on the photographed image photographed by the photographing device 16 is transmitted to the electronic control device 20 and the server 30. Note that the "photographed image" herein includes not only still images but also one frame (one frame) image included in a moving image.

The position and photographing direction of the photographing device 16 are fixed with respect to the smart glasses 10, and the position of the display device 13 is also fixed with respect to the smart glasses 10. Therefore, as long as the relationship between the smart glasses 10 and the line of sight of the worker 51 is kept constant, the virtual image appears as if it exists in the same environment as the real scene photographed by the photographing device 16 (i.e., the coordinate system is A three-dimensional object that matches the actual scene) can be drawn on the display device 13. Note that a known AR (Augmented Reality) development library can be used to calculate the position and orientation in three-dimensional space of a subject in an image photographed by the photographing device 16. Additionally, a known three-dimensional renderer or AR drawing engine can be used to draw a three-dimensional object that matches the actual scene.

The electronic control device 20 includes a processor 21 (central processing unit), a memory 22 (main memory, main storage device), an auxiliary storage device 23, an interface device 24, and the like, which are connected to each other via a bus 25 so as to be able to communicate with each other. Ru. The processor 21 is a central processing unit that includes a control unit (control circuit), an arithmetic unit (arithmetic circuit), a cache memory (register group), and the like. Further, the memory 22 is a storage device in which programs and data being worked on are stored, and includes, for example, ROM (Read Only Memory) and RAM (Random Access Memory).

The auxiliary storage device 23 is a memory device that stores data and firmware that are retained longer than the memory 22, and includes, for example, nonvolatile memory such as flash memory and EEPROM. Further, the interface device 24 controls input and output (I/O) between the electronic control device 20 and the outside. The electronic control device 20 is connected to the recording medium drive 11, the storage device 12, the display device 13, the input device 14, the communication device 15, etc. via the interface device 24.

Note that the electronic control unit 40 of the server 30 also includes a processor 41 (central processing unit), a memory 42 (main memory, main storage device), an auxiliary storage device 43, an interface device 44, etc., and communicates with each other via a bus 45. Connected for communication. The functions of each element are almost the same as the elements with the same name built into the electronic control device 20, so the explanation will be omitted. Further, there is no intention to limit the specific hardware configuration of the electronic control device to the above configuration, and various known configurations may be applied.

[2. Software configuration]
FIG. 4 is a block diagram for explaining the processing contents of the manufacturing support program 9 executed by the smart glasses 10 and the server 30. The contents of these processes are recorded as an application program in the storage devices 12 and 32 of the smart glasses 10 and the server 30, the recording medium 18, etc., and are expanded on the memory and executed. Functionally classifying the processing contents executed here, the manufacturing support program 9 is provided with an extraction section 3, an acquisition section 4, a recognition section 5, a drawing section 6, a determination section 7, and a highlighting section 8. Furthermore, a first database 1 and a second database 2 are provided as databases that can be referenced by each of these elements.

The first database 1 defines the relationship between the assembly pattern of parts to be assembled and the corresponding identification information. The identification information is information for specifying the content of the work instructed to the worker 51, and is called, for example, a work number, work target vehicle number, instruction target vehicle number, etc. In this embodiment, as shown in FIG. 2, it is assumed that the work number is written in the work instruction sheet 54 presented by the work manager. It is assumed that the work instruction sheet 54 is presented by a work manager before the worker 51 performs the assembly work, or is affixed to the automobile 50 to be assembled.

The contents of the first database 1 are illustrated in FIG. 5(A). The first database 1 defines at least the relationship between a work number for identifying an assembly work and information (assembly position, assembly direction) for specifying the layout (assembly pattern) of parts to be assembled in that work. Ru. If a plurality of types of parts are to be assembled in the work, assembling patterns for the plurality of types of parts are set for one work number. In the example shown in FIG. 5A, assembly patterns for three types of parts (a-01, a-02, a-03) are set for work number "ABCDE01234."

Note that if the model, grade, destination, etc. of the vehicle 50 to be worked on differ, the types of parts and assembly patterns will also differ. Generally, in the assembly work of the automobile 50, the number of assembly patterns that the worker 51 must learn ranges from tens to hundreds, and it is said that it takes several months to several years of experience to memorize these patterns. ing. On the other hand, in this embodiment, all these patterns are defined in the first database 1 and linked to the work number. This frees the operator 51 from memorizing assembly patterns.

The assembly position of a component can be specified using at least three parameters based on a coordinate system fixed to an assembly target (for example, a relay box). Similarly, the direction in which the parts are to be assembled can be specified using at least three parameters based on a coordinate system fixed to the object to be assembled. Therefore, information for specifying the layout of one component can be uniquely specified by using six or more parameters.

The second database 2 defines the relationship between the identification information (work number), the storage location of parts, and the number of parts to be used (the number of parts to be assembled). The storage location of a component is information indicating which tray the component is stored in, for example, among a large number of component trays stored in the material storage shelf 53 shown in FIG. 1. If the storage location of parts used in a certain work is always fixed, the location (storage location) of the parts can be specified using at least three parameters. On the other hand, if there is a possibility that the storage location of the parts may change depending on the work situation, it is preferable to add information for specifying the storage location to each record in the second database 2.

In the example shown in FIG. 5(B), the number of parts "a-01" used in the work number "ABCDE01234" is three, and its storage location is the location code "aA11bB22cC33dD44." At the location where the part is actually stored, a string of characters, symbols, barcodes, two-dimensional codes, etc. representing the location code ``aA11bB22cC33dD44'' will be posted. By assigning a different location code to each part, the storage location of each part can be uniquely identified.

The extraction unit 3 extracts identification information (work number) included in a work instruction sheet 54 in which the contents of the assembly work are described, from images taken by the photographing device 16. It is assumed that the work instruction sheet 54 is photographed before the worker 51 performs the part assembly work. The extraction unit 3 of this embodiment has a function of extracting a work number by reading the barcode of the work instruction sheet 54 shown in FIG. The information on the work number extracted here is transmitted to the acquisition unit 4.

The acquisition unit 4 has two functions. The first function is information acquisition based on the first database 1, and the second function is information acquisition based on the second database 2. The information acquired here is transmitted to the drawing section 6.
The former function is to obtain, based on the first database 1, the assembly pattern of the parts corresponding to the identification information (work number) extracted by the extraction unit 3. For example, if the work number extracted by the extraction unit 3 is "ABCDE01234", three types of parts (a-01, a-02, a -03), the assembly pattern (information on the assembly position and assembly direction) is acquired.
The latter function is to obtain, based on the second database 2, information on the storage location and number of parts used corresponding to the identification information (work number). For example, based on the second database 2 shown in FIG. 5(B), information on the location code and the number of used parts is acquired for each of the three types of parts (a-01, a-02, a-03).

The recognition unit 5 recognizes a specific object or mark present in an image taken by the photographing device 16. The recognition unit 5 has a function of recognizing at least "a target for assembling parts". For example, in the relay assembly work [work number "ABCDE01234" in FIG. 5A], it is determined whether the relay box 55 to which the relay is to be assembled is present in the photographed image. In this determination, the three-dimensional CAD data of the relay box 55 is referred to. As shown in FIG. 6(A), when the relay box 55 is detected in the captured image, the captured image is The position and orientation of the relay box 55 are estimated based on the position (the position of the photographing device 16). This specifies how the relay box 55 looks within the field of view of the worker 51. Information on the position and orientation of the assembly target identified here is transmitted to the drawing section 6 and referred to when drawing the parts.

The recognition unit 5 also has a function of recognizing "part storage location" from the photographed image. For example, in the relay assembly work [work number "ABCDE01234" in FIG. 5(B)], a character string, symbol, two-dimensional code, etc. representing the location code "aA11bB22cC33dD44" is recognized. As shown in FIG. 7(A), when a marker 58 representing a location code is detected in a photographed image, the position and orientation of the marker 58 with respect to the photographing position are estimated based on the position and shape of the marker 58. be done. Information on the storage location of the part recognized here is also transmitted to the drawing unit 6.

The drawing section 6 controls the display contents of the display device 13 when parts are assembled, and has three main functions. The first function is to draw a virtual image of the part in its completed position and orientation when the part is assembled into the object to be assembled. The "virtual image" herein refers to the state (completed drawing) when the parts have been assembled, and serves as a "model" for the assembly work. By performing the assembly work while referring to this, even the worker 51 with little work experience is less likely to make assembly mistakes, and work efficiency is improved.

The virtual image of the part is drawn so that the coordinate system matches the assembly target recognized by the recognition unit 5. For example, a virtual image of a component relative to the relay box 55 shown in FIG. FIG. 6(B) shows only the virtual image of the component drawn here. By superimposing the relay box 55 and the virtual image of the component, it can be made to appear as if the component exists above the relay box 55, as shown in FIG. 6(C).

The second function of the drawing unit 6 is to draw information on the number of parts (number of parts used) used in the assembly work near the relay box 55, and the third function is to store information on the number of parts. It is drawn near a location. When displaying the information on the number of pieces near the relay box 55, it is preferable to draw it at a position that does not overlap the parts to be assembled into the relay box 55, as shown in FIG. 6(B). Further, if the surface on which each component is assembled to the relay box 55 is planar, it is preferable to arrange the virtual images of each component on that plane. Similarly, when drawing the number information near the storage location, it is preferable to draw it at a position that does not overlap the marker 58 or the parts tray, as shown in FIG. 7(B). Note that the storage status may be communicated more clearly by displaying not only the number of parts but also the part number and appearance of the parts.

The determination unit 7 automatically determines whether or not the content of the work is appropriate when the work of assembling parts is completed. Here, it is determined whether the assembled state of the parts matches the assembly pattern defined in the first database 1 based on the photographed image and the identification information (work number). For example, comparison with the assembly pattern may be performed by identifying each component in the photographed image and estimating the position and orientation of each component with respect to the assembly target. Alternatively, the photographed image may be compared with a completed image prepared in advance.

The determination unit 7 of this embodiment determines the assembled state of parts using a learned autoencoder. Specifically, a photographed image is input to an autoencoder, an output image thereof is obtained, and a difference between the input image and the output image is calculated. The autoencoder uses thousands of images taken from various positions and angles of correctly assembled parts to be assembled. When a photographed image is input to such an autoencoder, if the assembled state in the photographed image is appropriate, the output image will be almost the same as the input image. On the other hand, if there is an inappropriate location in the assembled state, the image of that location is converted into an "image close to the correct assembled state" and output. In other words, a location where the assembly state is inappropriate can be detected as a difference between an input image and an output image for the autoencoder. Therefore, the determination unit 7 calculates the difference between the input image and the output image, and calculates a location where the difference is greater than or equal to a predetermined threshold as a "highly abnormal location."

The highlighting section 8 is for highlighting the "locations with a high degree of abnormality" determined by the determining section 7. Here, parts where the assembled state of the parts does not match the assembly pattern are highlighted so as to match the coordinate system of the actual scene within the field of view of the operator. The highlighting section 8 of this embodiment has a function of displaying an image on the display device 13 in which "locations with a high degree of abnormality" are highlighted in red. As a result, as shown in FIG. 8, locations where the assembly is inappropriate are highlighted, and the existence of an assembly error is clearly communicated to the operator 51.

[3. flowchart]
FIG. 9 is a flowchart for explaining the flow of the parts assembly process. Flag A in this flow indicates whether the identification information (work number) has been read or not, and is set to A=0 in the initial state where the identification information has not been read. First, a photographed image is photographed by the photographing device 16, and its contents are recognized (step A1). Further, the value of flag A is determined (step A2), and when A=0, it is determined whether or not the work instruction sheet 54 is recognized in the photographed image (step A3).

Here, when the work instruction sheet 54 is recognized, the extraction unit 3 extracts the work number written in the work instruction sheet 54 (step A4). In response to this, the acquisition unit 4 refers to the first database 1 and acquires the assembly pattern of the parts corresponding to the work number (step A5). The acquisition unit 4 also refers to the second database 2 and acquires information on the storage location and number of parts used for the parts corresponding to the work number (step A6). Thereafter, the value of flag A is set to A=1 (step A7). In the next and subsequent calculation cycles, control proceeds from step A1 to step A8.

In step A8, it is determined whether the marker 58 representing the location code is recognized in the photographed image. If the marker 58 is detected here, as shown in FIG. 7(B), information on the storage location and number of parts used is displayed on the display device 13 of the smart glasses 10. The worker 51 can take out the parts from the material storage shelf 53 while referring to this and prepare for assembly work. Furthermore, in step A10, it is determined whether or not the object to which the parts are to be assembled has been recognized. If the assembly target is recognized here, a virtual image of the component is displayed on the display device 13, as shown in FIG. 6(C). The operator 51 can assemble the parts while referring to this.

FIG. 10 is a flowchart for explaining the flow of the inspection process when the assembly of parts is completed. The inspection process may be started by the operator 51 performing a predetermined input operation (for example, a predetermined voice instruction, button operation, etc.) when the assembly work is completed. Alternatively, the inspection process may be started by automatically detecting through image analysis that the assembly work of the parts has been completed.

In the former case, various information regarding input operations by the worker 51 is input in step B1. Here, when a predetermined test start condition (for example, a predetermined voice instruction is given) is satisfied (step B2), the image taken by the imaging device 16 is inputted as an input image to the autoencoder, and the output image is acquired. (Step B3). Further, the difference between the input image and the output image is calculated, and the portions where the difference is equal to or greater than the threshold value are highlighted. The operator 51 can correct an inappropriate assembly state while referring to this.

[4. Action/Effect]
(1) In the manufacturing support system, method, and program described above, the identification information (work number) included in the work instruction sheet 54 is extracted, and a virtual image of the part is drawn at the position and orientation corresponding to this. Further, the relationship between the identification information (work number) and the component assembly pattern is defined in the first database 1 in advance. As a result, even if the types, numbers, and layouts of parts involved in assembly work differ for each work instruction 54, it is possible to show a model for assembling parts in the correct position and orientation, and can support the work.

Furthermore, the worker 51 does not need to memorize assembly patterns for different parts for each work instruction sheet 54, and assembly errors can be reduced. Furthermore, since a virtual image serving as a model is always displayed during the assembly work, there is no fear that the worker 51 will remember incorrect assembly, and the skill level of the worker 51 can be quickly improved. Therefore, work efficiency (productivity) related to parts assembly work can be improved.

(2) The relationship between the identification information (work number) and the storage location and number of parts used is defined in the second database 2 in advance. By using this to draw information on the number of parts used near the storage location of the parts, mistakes in mixing up parts can be prevented. For example, in an assembly work in which a plurality of parts having similar appearances coexist, it is possible to reduce pickup errors (mistake of parts). Therefore, the defective rate in the assembly work can be reduced, and work efficiency can be further improved.

(3) As shown in FIG. 7(A), by specifying the storage location of parts using the marker 58, it is possible to flexibly respond to changes in component specifications and storage locations, and work Efficiency can be further improved. For example, when the storage location of a part is changed, the marker 58 may be moved together with the part. Note that if the type of component or storage location is not changed, support may be provided based on the appearance of the component or the storage location instead of the marker 58.

(4) As shown in FIG. 8, by highlighting a location with a high degree of abnormality in the assembly state so as to match the coordinate system of the actual scene, it is possible to make the worker 51 more aware of the location. As a result, assembly errors can be corrected immediately, and work efficiency can be further improved. In addition, since any defects in assembly caused by misremembering or misunderstandings by the worker 51 are immediately fed back, the proficiency level of the worker 51 can be quickly improved, making it possible to develop human resources in a short period of time. .

[5. Modified example]
The above-described embodiment is merely an example, and there is no intention to exclude the application of various modifications and techniques not explicitly described in this embodiment. Each structure of this embodiment can be modified and implemented in various ways without departing from the spirit thereof. Further, they can be selected or combined as necessary.

For example, the smart glasses 10 may have a built-in motion sensor (acceleration sensor, orientation sensor, gyro sensor, geomagnetic sensor, etc.). By grasping the postures of the display device 13 and the photographing device 16 with high precision using a motion sensor, it becomes possible to more accurately estimate the position and posture of the real scene and the subject within the field of view of the worker 51, and the virtual The superimposition accuracy can be improved, and the reality and presence of the virtual image can be enhanced.

The minimum configuration of the manufacturing support system, manufacturing support method, and manufacturing support program in this case includes a first database 1, an extraction section 3, an acquisition section 4, a recognition section 5, and a drawing section 6. The device in which each of these elements is stored can be arbitrarily set, and the functions can be distributed and arranged among a plurality of devices. When distributing functions, it is sufficient that at least a plurality of devices are communicably connected to each other by wire or wirelessly.

Regarding the distribution of the above-mentioned functions, as shown in FIG. , extraction section 3, acquisition section 4, recognition section 5, and determination section 7) may be assigned to the assembly work computer 59. Further, the first database 1 and the second database 2 may be stored in the production command server 60. The production command server 60 is a large computer that controls the operating status of the production line and the operation of the production robots, and is connected to the assembly work computer 59 via a wired cable. Further, the assembly work computer 59 is a computer placed near the workplace where the assembly work is performed, and is wirelessly connected to the smart glasses 10 via the relay 36.

With such a device configuration, even if the electronic control device 20 built into the smart glasses 10 has a low arithmetic processing capacity, it is possible to accurately display a virtual image that matches the actual scene on the display device 13. Further, by arranging the databases 1 and 2 in the production command server 60, work support using the database used in the existing production system becomes easy. Note that if the smart glasses 10 are expected to have sufficient arithmetic processing capacity, a system that can operate with the smart glasses 10 alone may be configured as shown in FIG. 11(B). In this case, there is no need for communication with other computers, making it possible to support work in an offline environment.

[6. Additional notes]
Regarding the embodiments including the above-mentioned modifications, the following additional notes are disclosed.
[Additional note 1]
A worker wearing smart glasses on his/her head that includes a photographing device that photographs a real scene within the field of view and a display device that superimposes and displays a virtual image on the real scene is asked to display a virtual image that matches the coordinate system of the real scene on the display device. A manufacturing support method for supporting parts assembly work by displaying the
extracting the identification information included in a work instruction sheet in which the content of the assembly work is described from among the images taken by the photographing device;
obtaining an assembly pattern of the component corresponding to the identification information based on a first database in which a relationship between the assembly pattern of the plurality of components to be assembled and the corresponding identification information is defined;
Recognizing the assembly target from the image taken by the photographing device,
A manufacturing support method, characterized in that a virtual image of the part is drawn in the position and orientation at the time of completion when the part is assembled to the assembly target.

[Additional note 2]
Based on a second database that defines the relationship between the storage location of the parts, the number of parts to be assembled to the assembly target, and the identification information, the information on the storage location of the parts corresponding to the identification information and the information on the parts Obtain information on the number of pieces,
Recognizing the storage location from among images taken by the imaging device;
The manufacturing support method according to appendix 1, characterized in that information on the number of parts is displayed near the storage location.

[Additional note 3]
The manufacturing support method according to appendix 2, characterized in that a marker installed at the storage location in the photographed image is recognized.

[Additional note 4]
Determining whether the assembled state of the parts matches the assembly pattern based on the photographed image of the photographing device and the identification information,
According to any one of Supplementary Notes 1 to 3, the parts where the assembly state of the parts does not match the assembly pattern are highlighted so as to match the coordinate system of the actual scene within the field of view of the worker. Manufacturing support method described.

[Additional note 5]
A worker wearing smart glasses on his/her head that includes a photographing device that photographs a real scene within the field of view and a display device that superimposes and displays a virtual image on the real scene is asked to display a virtual image that matches the coordinate system of the real scene on the display device. A manufacturing support program that supports parts assembly work by displaying
extracting the identification information included in a work instruction sheet in which the content of the assembly work is described from among the images taken by the photographing device;
obtaining an assembly pattern of the component corresponding to the identification information based on a first database in which a relationship between the assembly pattern of the plurality of components to be assembled and the corresponding identification information is defined;
Recognizing the assembly target from the image taken by the photographing device,
A manufacturing support program that causes a computer to execute a process of drawing a virtual image of the part in a completed position and orientation when the part is assembled to the assembly target.

[Additional note 6]
Based on a second database that defines the relationship between the storage location of the parts, the number of parts to be assembled to the assembly target, and the identification information, the information on the storage location of the parts corresponding to the identification information and the information on the parts Obtain information on the number of pieces,
Recognizing the storage location from among images taken by the imaging device;
The manufacturing support program according to appendix 5, which causes a computer to execute a process of displaying information on the number of parts near the storage location.

[Additional note 7]
The manufacturing support program according to appendix 6, which causes a computer to execute a process of recognizing a marker installed at the storage location in the photographed image.

[Additional note 8]
Determining whether the assembled state of the parts matches the assembly pattern based on the photographed image of the photographing device and the identification information,
Any one of Supplementary Notes 5 to 7, wherein the computer executes a process of highlighting a portion where the assembly state of the parts does not match the assembly pattern so as to match the coordinate system of the actual scene within the field of view of the worker. Manufacturing support program as described in .

1 First database 2 Second database 3 Extraction unit 4 Acquisition unit 5 Recognition unit 6 Drawing unit 7 Judgment unit 8 Emphasis display unit 9 Manufacturing support program 10 Smart glasses 13 Display device 16 Photographing device 30 Server 50 Car 51 Worker 52 Helmet 53 Material storage shelf 54 Work instructions 55 Relay box (to be assembled)
56 Relay (parts)

Claims (5)

A manufacturing support system for supporting parts assembly work by workers,
smart glasses that are attached to the head of the worker and include a photographing device that captures a real scene within the field of view of the worker and a display device that displays a virtual image superimposed on the real scene;
a first database that defines a relationship between an assembly pattern that includes information on the assembly position and orientation of the plurality of parts with respect to the assembly target and identification information corresponding thereto;
a second database in which a relationship between a storage location of a plurality of the parts to be assembled to the assembly object, a number of used parts (the number of each of the parts), a part number of each of the parts, and the identification information is defined;
a control device for displaying a virtual image of the part on the display device so as to match the coordinate system of the actual scene within the field of view of the worker;
The control device,
an extraction unit that extracts the identification information included in a work instruction in which the content of the assembly work is described from among images taken by the imaging device;
an acquisition unit that acquires an assembly pattern including information on the assembly position and assembly orientation of the component corresponding to the identification information, based on the first database;
a recognition unit that recognizes the assembly target from among images taken by the photographing device;
a drawing unit that draws virtual images of the plurality of parts in the completed positions and orientations when the plurality of parts are assembled to the assembly target,
The acquisition unit acquires, based on the second database, information on the storage location of the part and information on the number of parts that correspond to the identification information, and also acquires information on the part number and the usage that correspond to the identification information. obtain information on the number of
the recognition unit recognizes the storage location from among images taken by the imaging device;
The drawing unit displays information on the number of parts near the storage location, and draws information on the part number and the number of parts used near the assembly target.
A manufacturing support system characterized by: 2. The manufacturing support system according to claim 1 , wherein the recognition unit recognizes a marker placed at the storage location in the photographed image. The control device includes:
a determination unit that determines whether the assembled state of the parts matches the assembly pattern based on the image taken by the photographing device and the identification information;
2. The method according to claim 1, further comprising a highlighting section that highlights a portion where the assembly state of the parts does not match the assembly pattern so as to match the coordinate system of the actual scene within the field of view of the worker. 2. The manufacturing support system described in 2 . A worker wearing smart glasses on his/her head that includes a photographing device that photographs a real scene within the field of view and a display device that superimposes and displays a virtual image on the real scene is asked to display a virtual image that matches the coordinate system of the real scene on the display device. A manufacturing support method for supporting parts assembly work by displaying the
extracting the identification information included in a work instruction sheet in which the content of the assembly work is described from among the images taken by the photographing device;
Based on a first database in which a relationship between an assembly pattern including information on the assembly positions and orientations of the plurality of parts to be assembled and the corresponding identification information is defined, the assembly position of the part corresponding to the identification information is determined. and obtain an assembly pattern including information on the assembly orientation,
Recognizing the assembly target from the image taken by the photographing device,
drawing virtual images of the plurality of parts in the completed positions and orientations when the plurality of parts are assembled to the assembly target ;
Based on a second database in which the relationship between the storage location of the plurality of parts to be assembled to the assembly object, the number of used parts (the number of each part), the part number of each part, and the identification information is defined, Obtaining information on the storage location of the part and information on the number of parts corresponding to the identification information, and obtaining information on the part number and the number of parts used corresponding to the identification information,
Recognizing the storage location from among images taken by the imaging device;
Information on the number of parts is displayed near the storage location, and information on the part number and the number of parts used is drawn near the assembly target.
A manufacturing support method characterized by: A worker wearing smart glasses on his/her head that includes a photographing device that photographs a real scene within the field of view and a display device that superimposes and displays a virtual image on the real scene is asked to display a virtual image that matches the coordinate system of the real scene on the display device. A manufacturing support program that supports parts assembly work by displaying
extracting the identification information included in a work instruction sheet in which the content of the assembly work is described from among the images taken by the photographing device;
Based on a first database in which a relationship between an assembly pattern including information on the assembly positions and orientations of the plurality of parts to be assembled and the corresponding identification information is defined, the assembly position of the part corresponding to the identification information is determined. and obtain an assembly pattern including information on the assembly orientation,
Recognizing the assembly target from the image taken by the photographing device,
drawing virtual images of the plurality of parts in the completed positions and orientations when the plurality of parts are assembled to the assembly target ;
Based on a second database in which the relationship between the storage location of the plurality of parts to be assembled to the assembly object, the number of used parts (the number of each part), the part number of each part, and the identification information is defined, Obtaining information on the storage location of the part and information on the number of parts corresponding to the identification information, and obtaining information on the part number and the number of parts used corresponding to the identification information,
Recognizing the storage location from among images taken by the imaging device;
Information on the number of parts is displayed near the storage location, and information on the part number and the number of parts used is drawn near the assembly target.
A manufacturing support program that allows a computer to perform processing.

Images