JP7468722B2 - Manufacturing support system, method, and program - Google Patents

Description

The present invention relates to a manufacturing support system, method, and program for supporting workers in assembling parts.

Conventionally, systems have been known in manufacturing production sites that provide work support by having workers wear a light-transmitting head-mounted display and superimposing (overlapping) information such as the work content and work procedure instructions on a real scene. For example, a technology has been proposed to support inspection work during product inspection by displaying a pass image (an image of an inspection object that has previously been judged to pass) near the actual inspection object (real image). It has also been proposed to improve workability in part assembly work by encircling specific locations with a circle to highlight them, or by displaying the wiring route. By providing such support, the efficiency of visual inspection and assembly work can be improved (see Patent Documents 1 and 2).

International Publication No. 2017/033561 JP 2018-014218 A

Meanwhile, at assembly work sites, work instructions are sometimes given using technical terms or difficult-to-decipher symbols (combinations of multiple symbols to convey information), making it difficult for beginners to accurately grasp the work content. In addition, in work in which multiple parts are assembled into an assembly target, the layout of the parts may look different depending on the posture and orientation of the assembly target as seen by the worker. Therefore, even if the assembled state of the parts looks different from the acceptable image, it may actually be in an appropriate state. Furthermore, since multiple parts are checked at the same time, inappropriate assembly states may be overlooked in visual inspection. Thus, with conventional work support systems, there is a problem that assembly errors are likely to occur in work in which multiple parts are assembled, making it difficult to improve work efficiency.

One of the objectives of the present invention was devised in light of the above-mentioned problems, and is to provide a manufacturing support system, method, and program that can improve the work efficiency of part assembly work. In addition to this objective, another objective of the present invention is to achieve effects that cannot be obtained with conventional technology, which are derived from the configurations shown in the "Mode for carrying out the invention" described below.

(1) The disclosed manufacturing support system is a manufacturing support system for supporting a worker in assembling parts. The system includes smart glasses to be worn on the head of the worker, which incorporate an imaging device that captures an image of a real scene within the field of vision of the worker and a display device that displays a virtual image superimposed on the real scene. The system also includes a first database that defines the relationship between an assembly pattern, including information on the assembly positions and assembly orientations of a plurality of parts relative to an assembly target, and corresponding identification information. The system further includes a control device that causes the display device to display a virtual image of the part so as to conform to the coordinate system of the real scene within the field of vision of the worker.

The control device is provided with an extraction unit, an acquisition unit, a recognition unit, and a drawing unit. The extraction unit extracts the identification information included in a work instruction sheet describing the contents of the assembly work from the image captured by the imaging device. The acquisition unit acquires an assembly pattern including information on the assembly position and assembly orientation of the part corresponding to the identification information based on the first database. The recognition unit recognizes the assembly target from the image captured by the imaging device. The drawing unit draws virtual images of the multiple parts in the positions and orientations at which the multiple parts will be assembled on the assembly target when completed.

(2) It is preferable to have a second database in which the relationship between the storage location of the part, the number of the part (number of parts used) to be assembled to the assembly target, and the identification information is defined. In this case, the acquisition unit acquires information on the storage location of the part corresponding to the identification information and information on the number of the parts based on the second database. In addition, the recognition unit recognizes the storage location from within the image captured by the imaging device. Furthermore, the drawing unit displays information on the number of the parts near the storage location.

(3) It is preferable that the recognition unit recognizes a marker placed at the storage location in the captured image.
(4) It is preferable that the control device has a determination unit and a highlighting unit. In this case, the determination unit determines whether or not the assembly state of the part matches the assembly pattern based on the captured image by the photographing device and the identification information. Also, the highlighting unit highlights a portion of the part whose assembly state does not match the assembly pattern so as to fit into a coordinate system of an actual scene within the field of vision of the operator.

By extracting the identification information contained in the work instructions and drawing a virtual image of the part in the corresponding position and orientation, even if the type, number, or layout of the parts to be assembled differs for each work instruction, an example of assembly can be shown in the correct position and orientation, and assembly work can be supported. In addition, workers no longer need to memorize the assembly state of parts, which differs for each work instruction, reducing assembly errors and improving productivity (work efficiency).

FIG. 1 is a diagram showing an application target of a manufacturing support system. FIG. 1 is a diagram for explaining a configuration of a manufacturing support system. FIG. 2 is a block diagram showing a hardware configuration of the manufacturing support system. FIG. 4 is a block diagram showing a software configuration of a manufacturing support program. (A) and (B) are examples of database configurations. 13A to 13C are diagrams for explaining examples of displaying virtual images of parts. 13A and 13B are diagrams for explaining examples of displaying the number of parts. FIG. 13 is a diagram for explaining an example of highlighting of test results. 11 is a flowchart for explaining a manufacturing support technique in an assembly process. 11 is a flowchart for explaining a manufacturing support method in an inspection process. 13A and 13B are diagrams for explaining a modified example of the manufacturing support system.

The manufacturing support system, method, and program as embodiments will be described below with reference to the drawings. The embodiments described below are merely examples, and are not intended to exclude the application of various modifications or techniques not explicitly stated in the following embodiments. Each configuration of this embodiment can be modified in various ways without departing from the spirit of the embodiment. In addition, they can be selected as needed, or combined as appropriate.

The manufacturing support system of this embodiment is a system that uses a wearable gadget to support the part assembly work by a worker 51, and is applied to the manufacturing process of an automobile 50 as shown in FIG. 1. Here, support for the work of a worker 51 taking parts from a material storage shelf 53 and assembling them in an automobile 50 will be described. Specific examples of parts to be assembled include electrical components such as relays, switches, and lamps, as well as glass, bumpers, power trains, seats, and tires. Specific examples of objects to which parts are to be assembled (devices to which parts are to be assembled) include relay boxes, switch boxes, instrument panels, doors, and vehicle bodies. There are a wide variety of objects to be assembled and types of parts in part assembly work, and the manufacturing support system, method, and program of this embodiment can be applied to the assembly work of any parts.

The work support is the provision of visual information using smart glasses 10 worn on the head of the worker 51. As shown in FIG. 2, the smart glasses 10 are fixed to the helmet 52 of the worker 51. The smart glasses 10 are equipped with a display device 13 and a photographing device 16. The photographing device 16 is a camera that photographs the real scene within the field of view of the worker 51 and the work instructions 54. The display device 13 is a device that displays a virtual image superimposed on the real scene within the field of view of the worker 51. The display method may be a method of projecting a virtual image onto the retina of the worker 51 (retinal projection type), a method of projecting a virtual image onto the surface of a half mirror (transmissive type), or a method of displaying an image obtained by combining the real scene and the virtual image photographed by the photographing device 16 (non-transmissive type).

In either method, various virtual images are displayed so as to fit the coordinate system of the real scene as seen from the viewpoint of the worker 51. The virtual images are perspective projection images (or deformed versions of those perspective projection images) that fit the coordinate system of the real scene, and are rendered, for example, as simple polygon models based on three-dimensional CAD (Computer-Aided Design) data of parts. In existing display methods such as those in Patent Documents 1 and 2, the drawn objects are two-dimensional objects. In contrast, in the display method of this case, the drawn objects are three-dimensional objects, and each drawn object is virtually positioned within the coordinate system of the real scene.

The content of the work support is controlled by an arbitrary control device (electronic calculator, computer). The "electronic control device" referred to here may be one built into the smart glasses 10, or one prepared separately from the smart glasses 10. For example, a system may be used in which a program is installed in a server 30 or a workstation on the network 19, and the display content calculated there is displayed on the smart glasses 10. In this case, as shown in FIG. 2, a server 30 connected to the network 19 via a repeater 36 may be used. Alternatively, a program that operates even when not connected to the network 19 may be installed in the smart glasses 10.

[1. Hardware Configuration]
3 is a block diagram showing the hardware configuration of a manufacturing support system realized by cooperation between the smart glasses 10 and the server 30. The smart glasses 10 are provided with a recording medium drive 11 for reading a program recorded on a recording medium 18 (removable media), a storage device 12, a display device 13, an input device 14, a communication device 15, a photographing device 16 (camera), a microphone 17, and an electronic control device 20. Similarly, the server 30 also has a recording medium drive 31, a storage device 32, a display device 33, an input device 34, a communication device 35, and an electronic control device 40 built in. The functions of each element are almost the same as those of the elements of the same name built in the smart glasses 10, so the description will be omitted.

The recording medium 18 is, for example, a flash memory card, a semiconductor memory device conforming to the Universal Serial Bus standard, an optical disk, etc. The recording medium drive 11 is a reading device that reads information (programs and data) recorded and stored on the recording medium 18. The storage device 12 is a memory device that stores data and programs that are to be retained for a long period of time, and includes, for example, non-volatile memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), and a solid-state drive.

The display device 13 is one of the devices for providing visual information to the worker 51. In a retinal projection type device, the retina of the worker 51 serves as the image forming surface. On the other hand, in a transmissive or non-transmissive type device, the surface of a half mirror or a display (such as a Liquid Crystal Display (LCD) or an Organic Electro-Luminescence Display (OLED)) placed within the field of view of the worker 51 serves as the image forming surface.

The input device 14 is one of the pointing devices for inputting an input signal to the electronic control devices 20, 40. Specific examples of the input device 14 include devices such as a wireless keyboard and a wireless mouse that can be connected to the smart glasses 10. In addition, the temples (handle, side parts) of the smart glasses 10 are provided with input devices 14 such as push buttons and physical keys. In addition, the microphone 17 is a device for inputting the voice of the worker 51 as an input signal.

The communication device 15 is a device that controls communication with the outside of the smart glasses 10, and is, for example, a device for performing wireless communication with the server 30 via the network 19. The repeater 36 shown in FIG. 2 can be considered to be included in the communication device 15. When input/output devices that comply with a short-range wireless communication standard are connected to the smart glasses 10, those input/output devices are connected via the communication device 15.

The image capturing device 16 is a digital video camera with a built-in image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The image capturing device 16 is attached to the rim (periphery of the lens) or bridge (connection between the left and right lenses) of the smart glasses 10, and functions to capture the field of view of the worker 51 wearing the smart glasses 10. Information on the captured image captured by the image capturing device 16 is transmitted to the electronic control device 20 and the server 30. Note that the "captured image" referred to here includes not only still images but also one frame of an image included in a video.

The position and shooting direction of the image capturing device 16 are fixed with respect to the smart glasses 10, and the position of the display device 13 is also fixed with respect to the smart glasses 10. Therefore, as long as the relationship between the smart glasses 10 and the line of sight of the worker 51 is kept constant, a virtual image (i.e., a three-dimensional object whose coordinate system matches the real scene) that appears to exist in the same environment as the real scene captured by the image capturing device 16 can be rendered by the display device 13. Note that a publicly known AR (Augmented Reality) development library can be used to calculate the position and orientation in three-dimensional space of the subject in the image captured by the image capturing device 16. Also, a publicly known three-dimensional renderer or AR rendering engine can be used to render a three-dimensional object that matches the real scene.

The electronic control device 20 incorporates a processor 21 (central processing unit), memory 22 (main memory, primary storage device), auxiliary storage device 23, interface device 24, etc., which are communicatively connected to each other via a bus 25. The processor 21 is a central processing unit incorporating a control unit (control circuit), an arithmetic unit (arithmetic circuit), cache memory (register group), etc. The memory 22 is a storage device in which programs and data being worked on are stored, and includes, for example, ROM (Read Only Memory) and RAM (Random Access Memory).

The auxiliary storage device 23 is a memory device that stores data and firmware that are retained for a longer period of time than the memory 22, and includes non-volatile memories such as flash memory and EEPROM. The interface device 24 is responsible for input and output (I/O) between the electronic control device 20 and the outside. The electronic control device 20 is connected to the recording medium drive 11, storage device 12, display device 13, input device 14, communication device 15, etc. via the interface device 24.

The electronic control device 40 of the server 30 also includes a processor 41 (central processing unit), memory 42 (main memory, primary storage device), auxiliary storage device 43, interface device 44, etc., which are connected to each other via a bus 45 so that they can communicate with each other. The functions of each element are almost the same as the elements of the same name built into the electronic control device 20, so a description thereof will be omitted. Furthermore, there is no intention to limit the specific hardware configuration of the electronic control device to the above configuration, and various known configurations may be applied.

2. Software Configuration
4 is a block diagram for explaining the processing contents of the manufacturing support program 9 executed by the smart glasses 10 and the server 30. These processing contents are recorded as application programs in the storage devices 12, 32 and the recording medium 18 of the smart glasses 10 and the server 30, and are deployed on the memory and executed. If the processing contents executed here are classified functionally, the manufacturing support program 9 is provided with an extraction unit 3, an acquisition unit 4, a recognition unit 5, a drawing unit 6, a determination unit 7, and a highlighting unit 8. In addition, a first database 1 and a second database 2 are provided as databases to which each of these elements can refer.

The first database 1 specifies the relationship between the assembly pattern of parts for the assembly target and the corresponding identification information. The identification information is information for specifying the work content instructed to the worker 51, and is called, for example, the work number, the work target vehicle number, the instruction target vehicle number, etc. In this embodiment, as shown in FIG. 2, the work number is described in the work instruction sheet 54 presented by the work manager. The work instruction sheet 54 is presented by the work manager before the worker 51 performs the assembly work, or is affixed to the automobile 50 that is the object of assembly.

The contents of the first database 1 are shown in FIG. 5(A). The first database 1 defines the relationship between at least a job number for identifying an assembly job and information (assembly position, assembly direction) for specifying the layout (assembly pattern) of the parts to be assembled in that job. If there are multiple types of parts to be assembled in that job, assembly patterns for multiple types of parts are set for one job number. In the example shown in FIG. 5(A), assembly patterns for three types of parts (a-01, a-02, a-03) are set for job number "ABCDE01234".

If the model, grade, destination, etc. of the automobile 50 being worked on differs, the types of parts and assembly patterns will also differ. In general, when assembling an automobile 50, the worker 51 must learn dozens to hundreds of assembly patterns, and it is said that it takes several months to several years of experience to memorize these patterns. On the other hand, in this embodiment, all of these patterns are defined in the first database 1 and linked to the work number. This relieves the worker 51 from the task of memorizing the assembly patterns.

The installation position of a part can be specified by at least three parameters, with a coordinate system fixed to the object (e.g., a relay box) as the reference. Similarly, the installation orientation of a part can be specified by at least three parameters, with a coordinate system fixed to the object as the reference. Therefore, information for specifying the layout of a single part can be uniquely specified by using six or more parameters.

The second database 2 defines the relationship between the above-mentioned identification information (job number) and the storage location and number of uses (number of parts to be assembled) of the part. The storage location of a part is information that indicates which of the many parts trays stored in the material storage shelf 53 shown in FIG. 1 the part is in. If the storage location of a part used in a certain job is always fixed, the position (storage location) of the part can be identified using at least three parameters. On the other hand, if there is a possibility that the storage location of the part may change depending on the work situation, it is preferable to add information for identifying the storage location to each record in the second database 2.

In the example shown in FIG. 5(B), the part "a-01" used in operation number "ABCDE01234" has a usage count of 3, and its storage location is location code "aA11bB22cC33dD44". A character string, symbol, barcode, or two-dimensional code representing the location code "aA11bB22cC33dD44" is posted at the location where the part is actually stored. By assigning a different location code to each part, the storage location of each part can be uniquely identified.

The extraction unit 3 extracts, from the image captured by the image capture device 16, identification information (job number) included in the work instruction sheet 54 that describes the details of the assembly work. The work instruction sheet 54 is assumed to be photographed before the worker 51 performs the part assembly work. The extraction unit 3 of this embodiment has a function of extracting the job number by reading the barcode of the work instruction sheet 54 shown in FIG. 2. The job number information extracted here is transmitted to the acquisition unit 4.

The acquisition unit 4 has two functions. The first function is to acquire information based on the first database 1, and the second function is to acquire information based on the second database 2. The information acquired here is transmitted to the drawing unit 6.
The former function is to obtain an assembly pattern of a part corresponding to the identification information (operation number) extracted by the extraction unit 3, based on the first database 1. For example, if the operation number extracted by the extraction unit 3 is "ABCDE01234", an assembly pattern (information on the assembly position and assembly direction) is obtained for each of the three types of parts (a-01, a-02, a-03) corresponding thereto in the first database 1 shown in Fig. 5(A).
The latter function is to obtain information on the storage location and the number of parts used that correspond to the identification information (job number) based on the second database 2. For example, based on the second database 2 shown in Fig. 5(B), information on the location code and the number of parts used is obtained for each of three types of parts (a-01, a-02, a-03).

The recognition unit 5 recognizes specific objects and marks present in the image captured by the image capture device 16. The recognition unit 5 has at least the function of recognizing the "part installation target." For example, in a relay installation task [task number "ABCDE01234" in FIG. 5A], it is determined whether the relay box 55 to which the relay is to be installed is present in the captured image. In this determination, the three-dimensional CAD data of the relay box 55 is referenced. As shown in FIG. 6A, when the relay box 55 is detected in the captured image, the position and orientation of the relay box 55 based on the shooting position (position of the image capture device 16) are estimated based on the distribution of feature points (points and vertices included in the edge of the relay box 55, etc.) in the captured image. This identifies how the relay box 55 appears within the field of view of the worker 51. The information on the position and orientation of the installation target identified here is transmitted to the drawing unit 6 and is referenced when drawing the part.

The recognition unit 5 also has the function of recognizing the "storage location of parts" from the captured image. For example, in a relay assembly task [task number "ABCDE01234" in FIG. 5(B)], a character string, symbol, or two-dimensional code representing the location code "aA11bB22cC33dD44" is recognized. As shown in FIG. 7(A), when a marker 58 representing the location code is detected in the captured image, the position and orientation of the marker 58 relative to the capture position are estimated based on the position and shape of the marker 58. Information on the storage location of parts recognized here is also transmitted to the drawing unit 6.

The drawing unit 6 controls the display content of the display device 13 when parts are being assembled, and has three main functions. The first function is to draw a virtual image of the part in the position and orientation it will be in when assembled to the object to be assembled. The "virtual image" here indicates the state (finished drawing) when assembly of the part is complete, and serves as a "model" for the assembly work. By performing the assembly work while referring to this, even an inexperienced worker 51 is less likely to make an assembly mistake, improving work efficiency.

The virtual image of the part is drawn so that the coordinate system matches the assembly target recognized by the recognition unit 5. For example, the virtual image of the part for the relay box 55 shown in FIG. 6(A) is drawn in a position and orientation corresponding to the state in which the part is assembled to the relay box 55. FIG. 6(B) shows only the virtual image of the part drawn here. By superimposing the relay box 55 and the virtual image of the part, it is possible to make it appear as if the part is present on top of the relay box 55, as shown in FIG. 6(C).

The second function of the drawing unit 6 is to draw information on the number of parts (usage number) used in the assembly work near the relay box 55, and the third function is to draw the information on the number near the storage location. When displaying the information on the number near the relay box 55, it is preferable to draw it at a position that does not overlap with the parts to be assembled to the relay box 55, as shown in FIG. 6(B). Also, if the surface on which the parts are assembled to the relay box 55 is flat, it is preferable to arrange the virtual images of each part on that plane. Similarly, when drawing the information on the number near the storage location, it is preferable to draw it at a position that does not overlap with the marker 58 or the part tray, as shown in FIG. 7(B). It is also possible to convey the storage status more clearly by displaying not only the number of parts but also the part numbers and the appearances of the parts.

The judgment unit 7 automatically judges whether the work content is appropriate when the part assembly work is completed. Here, based on the captured image and identification information (work number), it is judged whether the assembly state of the parts matches an assembly pattern defined in the first database 1. For example, each part in the captured image may be identified, and the position and orientation of each part relative to the assembly target may be estimated, and a comparison with the assembly pattern may be performed. Alternatively, the captured image may be compared with a previously prepared image of the completed state.

In this embodiment, the judgment unit 7 judges the assembly state of the parts using a trained autoencoder. Specifically, a captured image is input to the autoencoder, and the output image is obtained, and the difference between the input image and the output image is calculated. As the learning data for the autoencoder, several thousand images of the assembly target in which the parts are correctly assembled are taken from various positions and angles. When a captured image is input to such an autoencoder, if the assembly state in the captured image is appropriate, the output image will be almost identical to the input image. On the other hand, if there is an inappropriate part in the assembly state, the image of that part is converted to an "image close to the correct assembly state" and output. In other words, a part in an inappropriate assembly state can be detected as a difference between the input image and the output image for the autoencoder. Therefore, the judgment unit 7 calculates the difference between the input image and the output image, and calculates the part where the difference is equal to or greater than a predetermined threshold as a "part with a high degree of abnormality".

The highlighting unit 8 highlights the "highly abnormal areas" determined by the determining unit 7. Here, areas where the assembly state of the parts does not match the assembly pattern are highlighted so as to fit the coordinate system of the actual scene within the worker's field of vision. The highlighting unit 8 in this embodiment has the function of displaying an image on the display device 13 in which the "highly abnormal areas" are highlighted in red. This highlights the areas where the assembly state is inappropriate, as shown in FIG. 8, and the existence of an assembly error is clearly communicated to the worker 51.

3. Flowchart
9 is a flow chart for explaining the flow of the part assembly process. The flag A in this flow indicates whether or not the identification information (job number) has been read, and is set to A=0 in the initial state where the identification information has not been read. First, an image is taken by the photographing device 16, and its contents are recognized (step A1). The value of the flag A is then determined (step A2), and if A=0, it is determined whether or not the work instruction sheet 54 has been recognized in the photographed image (step A3).

When the work instruction 54 is recognized, the extraction unit 3 extracts the work number written on the work instruction 54 (step A4). In response, the acquisition unit 4 refers to the first database 1 and acquires the assembly pattern of the part corresponding to the work number (step A5). The acquisition unit 4 also refers to the second database 2 and acquires information on the storage location and number of parts used corresponding to the work number (step A6). After that, the value of flag A is set to A=1 (step A7). In the next and subsequent calculation cycles, control proceeds from step A1 to step A8.

In step A8, it is determined whether a marker 58 indicating a location code has been recognized in the captured image. If a marker 58 has been detected, information on the storage location of the part and the number of parts used is displayed on the display device 13 of the smart glasses 10, as shown in FIG. 7(B). The worker 51 can refer to this information to take the part out of the material storage shelf 53 and prepare for the assembly work. In addition, in step A10, it is determined whether an object to which the part is to be assembled has been recognized. If an object to which the part is to be assembled has been recognized, a virtual image of the part is displayed on the display device 13, as shown in FIG. 6(C). The worker 51 can refer to this information to assemble the part.

Figure 10 is a flow chart for explaining the flow of the inspection process when the assembly of parts is completed. The inspection process may be started by the worker 51 performing a predetermined input operation (e.g., a predetermined voice instruction or button operation) when the assembly work is completed. Alternatively, the inspection process may be started by automatically detecting the completion of the assembly work of parts through image analysis.

In the former case, in step B1, various information related to input operations by the worker 51 is input. Here, when a predetermined inspection start condition (for example, a predetermined voice instruction is given) is met (step B2), the image captured by the imaging device 16 is input as an input image to the autoencoder, and the output image is obtained (step B3). In addition, the difference between the input image and the output image is calculated, and the parts where the difference is equal to or greater than a threshold value are highlighted. The worker 51 can refer to this and correct any improper assembly state.

[4. Actions and Effects]
(1) In the above-described manufacturing support system, method, and program, identification information (job number) included in the work instruction sheet 54 is extracted, and a virtual image of the part is drawn in a position and orientation corresponding to the extracted information. In addition, the relationship between the identification information (job number) and the part assembly pattern is defined in advance in the first database 1. As a result, even if the type, number of parts used, and layout of parts involved in the assembly work differ depending on each work instruction sheet 54, it is possible to show an example of how to assemble the parts in the correct position and in the correct orientation, thereby supporting the assembly work.

In addition, the worker 51 is no longer required to memorize different part assembly patterns for each work instruction sheet 54, which reduces assembly errors. Furthermore, because a model virtual image is always displayed during the assembly work, there is no concern that the worker 51 will remember incorrect assembly, and the worker 51's proficiency can be improved quickly. Therefore, the work efficiency (productivity) related to the part assembly work can be improved.

(2) The relationship between the identification information (operation number) and the storage location and number of uses of the part is predefined in the second database 2. Using this, information on the number of uses can be drawn near the storage location of the part, thereby preventing mistakes in picking up parts. For example, in an assembly operation in which multiple parts with similar appearances are mixed together, pick-up mistakes (mistakes in picking up parts) can be reduced. This makes it possible to reduce the defect rate in the assembly operation and further improve work efficiency.

(3) As shown in FIG. 7(A), by using marker 58 to identify the storage location of a part, it becomes possible to flexibly respond to changes in part specifications and storage locations, thereby further improving work efficiency. For example, when the storage location of a part is changed, marker 58 can be moved together with the part. Note that, when the type of part or storage location is not changed, assistance may be provided based on the appearance of the part or the appearance of the storage location instead of marker 58.

(4) As shown in FIG. 8, by highlighting the parts of the assembly state where the degree of abnormality is high so as to conform to the coordinate system of the actual scene, it is possible to make the worker 51 more aware of those parts. This allows assembly errors to be corrected immediately, and the work efficiency can be further improved. In addition, since assembly problems resulting from the worker's 51 misremembering or misunderstanding are immediately fed back, the proficiency of the worker 51 can be rapidly improved, and human resource development can be achieved in a short period of time.

5. Modifications
The above embodiment is merely an example, and is not intended to exclude various modifications or application of techniques not explicitly stated in the present embodiment. Each configuration of the present embodiment can be modified in various ways without departing from the spirit of the present embodiment. In addition, the configurations can be selected or combined as necessary.

For example, the smart glasses 10 may be equipped with a built-in motion sensor (such as an acceleration sensor, a direction sensor, a gyro sensor, or a geomagnetic sensor). By using the motion sensor to grasp the orientation of the display device 13 and the image capture device 16 with high accuracy, it becomes possible to more accurately estimate the position and orientation of the real scene and the subject within the field of view of the worker 51, thereby improving the accuracy of superimposing the virtual image on the real scene and enhancing the realism and presence of the virtual image.

The minimum configuration of the manufacturing support system, manufacturing support method, and manufacturing support program in this case includes a first database 1, an extraction unit 3, an acquisition unit 4, a recognition unit 5, and a drawing unit 6. The device in which each of these elements is stored can be set arbitrarily, and it is also possible to distribute and arrange functions on multiple devices. When distributing functions, it is sufficient that at least multiple devices are connected to each other so that they can communicate with each other via wire or wirelessly.

Regarding the distribution of the above functions, as shown in FIG. 11(A), the elements related to drawing on the display device 13 (e.g., the drawing unit 6, the highlighting unit 8) may be assigned to the smart glasses 10, and the other elements (e.g., the extraction unit 3, the acquisition unit 4, the recognition unit 5, the judgment unit 7) may be assigned to the assembly work computer 59. In addition, the first database 1 and the second database 2 may be stored in the production command server 60. The production command server 60 is a large computer that controls the operating status of the production line and the operation of the production robots, and is connected to the assembly work computer 59 by a wired cable. In addition, the assembly work computer 59 is a computer placed near the work area where the assembly work is performed, and is wirelessly connected to the smart glasses 10 via a repeater 36.

With this device configuration, even if the computing power of the electronic control device 20 built into the smart glasses 10 is low, it is possible to accurately display a virtual image that matches the actual scene on the display device 13. In addition, by arranging the databases 1 and 2 in the production command server 60, it becomes easy to provide work support using databases used in existing production systems. Note that if the smart glasses 10 are expected to have sufficient computing power, a system that can operate with the smart glasses 10 alone may be configured, as shown in FIG. 11 (B). In this case, communication with other computers is not required, making it possible to provide work support in an offline environment.

[6. Notes]
The following supplementary notes are disclosed regarding the embodiments including the above-mentioned modifications.
[Appendix 1]
A manufacturing support method for supporting a part assembly operation by displaying a virtual image conforming to a coordinate system of a real scene on a display device that displays a virtual image that is superimposed on a real scene, the virtual image being displayed on the display device to a worker wearing smart glasses on the worker's head, the virtual image being conforming to a coordinate system of the real scene,
extracting the identification information included in a work instruction sheet describing the details of the assembly work from the image captured by the imaging device;
acquiring an assembly pattern of the part corresponding to the identification information based on a first database in which a relationship between an assembly pattern of the part for an assembly target and the corresponding identification information is defined;
Recognizing the assembly target from the captured image of the imaging device;
A manufacturing support method comprising: drawing a virtual image of the part in a position and orientation that will result when the part is completely assembled onto the object to be assembled.

[Appendix 2]
acquiring information on the storage location of the part and information on the number of the parts corresponding to the identification information based on a second database in which a relationship between the storage location of the part, the number of the parts to be assembled to the assembly target, and the identification information is defined;
Recognizing the storage location from the image captured by the imaging device;
2. The manufacturing support method according to claim 1, further comprising displaying information on the number of parts in the vicinity of the storage location.

[Appendix 3]
The manufacturing support method according to claim 2, further comprising recognizing a marker placed at the storage location in the captured image.

[Appendix 4]
determining whether or not an assembly state of the part matches the assembly pattern based on the captured image of the imaging device and the identification information;
The manufacturing support method according to any one of appendices 1 to 3, characterized in that a portion where the assembly state of the part does not match the assembly pattern is highlighted so as to fit into a coordinate system of an actual scene within the field of vision of the worker.

[Appendix 5]
A manufacturing support program that supports a part assembly operation by displaying a virtual image that matches a coordinate system of a real scene on the display device, the virtual image being displayed on the display device to a worker wearing smart glasses that incorporate an imaging device that captures a real scene within a field of vision and a display device that displays a virtual image superimposed on the real scene,
extracting the identification information included in a work instruction sheet describing the details of the assembly work from the image captured by the imaging device;
acquiring an assembly pattern of the part corresponding to the identification information based on a first database in which a relationship between an assembly pattern of the part for an assembly target and the corresponding identification information is defined;
Recognizing the assembly target from the captured image of the imaging device;
A manufacturing support program that causes a computer to execute a process of drawing a virtual image of the part in a position and orientation when the part is completely assembled onto the assembly target.

[Appendix 6]
acquiring information on the storage location of the part and information on the number of the parts corresponding to the identification information based on a second database in which a relationship between the storage location of the part, the number of the parts to be assembled to the assembly target, and the identification information is defined;
Recognizing the storage location from the image captured by the imaging device;
6. The manufacturing support program according to claim 5, which causes a computer to execute a process of displaying information on the number of parts near the storage location.

[Appendix 7]
The manufacturing support program according to claim 6, which causes a computer to execute a process for recognizing a marker placed at the storage location in the captured image.

[Appendix 8]
determining whether or not an assembly state of the part matches the assembly pattern based on the captured image of the imaging device and the identification information;
The manufacturing support program according to any one of appendices 5 to 7, further comprising a computer executing a process of highlighting a portion where the assembly state of the part does not match the assembly pattern so as to conform to a coordinate system of an actual scene within the field of vision of the worker.

Reference Signs List 1 First database 2 Second database 3 Extraction unit 4 Acquisition unit 5 Recognition unit 6 Drawing unit 7 Determination unit 8 Highlighting unit 9 Manufacturing support program 10 Smart glasses 13 Display device 16 Photography device 30 Server 50 Automobile 51 Worker 52 Helmet 53 Material storage shelf 54 Work instructions 55 Relay box (assembly target)
56 Relay (parts)

Claims (6)

A manufacturing support system for supporting a part assembly operation by a worker, comprising:
A smart glass that is mounted on the head of the worker and includes a photographing device that photographs a real scene within the worker's field of vision and a display device that superimposes a virtual image on the real scene;
a first database that defines a relationship between an assembly pattern, including information on the assembly positions and assembly orientations of the plurality of parts with respect to an assembly target, and identification information corresponding to the assembly pattern;
a second database in which the relationship between the part numbers and the number of uses of the plurality of parts and the identification information is defined;
a control device that causes the display device to display a virtual image of the part so as to conform to a coordinate system of a real scene within the field of vision of the operator;
The control device,
an extraction unit that extracts the identification information included in a work instruction sheet in which details of the assembly work are described from the image captured by the imaging device;
an acquisition unit that acquires an assembly pattern including information on an assembly position and an assembly orientation of the part corresponding to the identification information based on the first database;
a recognition unit that recognizes the assembly target from within an image captured by the imaging device;
a drawing unit that draws virtual images of the plurality of parts in positions and orientations when the plurality of parts are assembled to the assembly target,
the acquiring unit acquires information on the part number and the usage quantity corresponding to the identification information based on the second database,
The manufacturing support system according to claim 1, wherein the drawing unit draws the part number and the information on the number of parts used near the object to be assembled. 2. The manufacturing support system according to claim 1, wherein the drawing unit draws an image showing the appearance of the part together with the information on the part number and the number of parts used, in the vicinity of the object to be assembled. 3. The manufacturing support system according to claim 2, wherein the recognition unit recognizes a marker placed at a storage location of the part in the captured image. The control device,
a determination unit that determines whether or not an assembly state of the part matches the assembly pattern based on the captured image of the imaging device and the identification information;
and a highlighting unit that highlights a portion where the assembly state of the part does not match the assembly pattern so as to conform to a coordinate system of an actual scene within the field of vision of the worker. A manufacturing support method for supporting a part assembly operation by displaying a virtual image conforming to a coordinate system of a real scene on a display device that displays a virtual image that is superimposed on a real scene, the virtual image being displayed on the display device to a worker wearing smart glasses on the worker's head, the virtual image being conforming to a coordinate system of the real scene,
extracting identification information included in a work instruction sheet describing the details of the assembly work from the image captured by the imaging device;
acquiring an assembly pattern including information on assembly positions and assembly orientations of the parts corresponding to the identification information based on a first database in which a relationship between an assembly pattern including information on assembly positions and assembly orientations of the parts with respect to an assembly target and the identification information corresponding to the assembly pattern is defined;
acquiring information on the part numbers and the usage numbers corresponding to the identification information based on a second database in which the relationship between the part numbers and the usage numbers of the multiple parts and the identification information is defined;
Recognizing the assembly target from the captured image of the imaging device;
A manufacturing support method comprising: drawing virtual images of a plurality of parts in positions and orientations when the plurality of parts are assembled to the object to be assembled, and drawing information on the part numbers and the number of parts used near the object to be assembled. A manufacturing support program that supports a part assembly operation by displaying a virtual image that matches a coordinate system of a real scene on the display device, the virtual image being displayed on the display device to a worker wearing smart glasses that incorporate an imaging device that captures a real scene within a field of vision and a display device that displays a virtual image superimposed on the real scene,
extracting identification information included in a work instruction sheet describing the details of the assembly work from the image captured by the imaging device;
acquiring an assembly pattern including information on assembly positions and assembly orientations of the parts corresponding to the identification information based on a first database in which a relationship between an assembly pattern including information on assembly positions and assembly orientations of the parts with respect to an assembly target and the identification information corresponding to the assembly pattern is defined;
acquiring information on the part numbers and the usage numbers corresponding to the identification information based on a second database in which the relationship between the part numbers and the usage numbers of the multiple parts and the identification information is defined;
Recognizing the assembly target from the captured image of the imaging device;
A manufacturing support program that causes a computer to execute a process of drawing virtual images of the multiple parts in the positions and orientations at which the multiple parts will be assembled to the assembly object, and drawing information on the part numbers and the number of parts used near the assembly object.

Images