WO2020049999A1

WO2020049999A1 - Manufacturing assistance system, method, and program

Info

Publication number: WO2020049999A1
Application number: PCT/JP2019/032359
Authority: WO
Inventors: 航粟津
Original assignee: 三菱自動車工業株式会社
Priority date: 2018-09-03
Filing date: 2019-08-20
Publication date: 2020-03-12
Also published as: JP7468722B2; JP2023065371A; JP7374908B2; JPWO2020049999A1

Abstract

In the present invention, the head of a worker is fitted with smart glasses having a built-in photographing device for photographing the actual view in the field of view of the worker and a display device for superimposing and displaying a virtual image on the actual view. A relationship between an assembly pattern of a plurality of components with respect to an object to be assembled and identification information corresponding to the pattern is defined in a first database (1). A control device causes the display device to display a virtual image of the components so as to match a coordinate system of the actual view in the field of view of the worker. The control device is provided with: an extraction unit (3) for extracting identification information contained in work instructions; an acquisition unit (4) for acquiring the assembly pattern of the components corresponding to the identification information; a recognition unit (5) for recognizing the object to be assembled from the photographed image of the photographing device; and a drawing unit (6) for drawing a virtual image of the components at the positions and orientations at the time of completion.

Description

Manufacturing support system, method, program

The present invention relates to a manufacturing support system, a method, and a program for assisting a worker in assembling parts.

Conventionally, at the production site of the manufacturing industry, workers are provided with a light transmission type head mounted display, and work support is provided by superimposing and displaying information such as work contents and work procedure instructions on the actual scene (superimposed display) There is a known system. For example, in product inspection, a technology has been proposed that supports an inspection operation by displaying a passing image (an image of an inspection object determined to be acceptable in the past) near an actual inspection object (real image). . It has also been proposed to improve the operability by assembling a part with a circle so as to emphasize a specific place, or by displaying a wiring route of a wiring member in a part assembling operation. By performing these supports, the efficiency of visual inspection and assembly work can be improved (see Patent Documents 1 and 2).

International Publication No. 2017/033561 JP 2018-014218A

By the way, at the site of the assembling work, the work content may be indicated using technical terms or codes that are difficult to decipher (combination of a plurality of symbols for information transmission). There is also the aspect that it is difficult to grasp even exactly. In a work in which a plurality of components are assembled to an assembly target, the layout of the components may look different depending on the posture and orientation of the assembly target as viewed from an operator. Therefore, even if the assembled state of the parts looks different from the acceptable image, it may actually be an appropriate state. Further, since a plurality of parts are checked at the same time, an improper assembly state may be overlooked by visual inspection. As described above, in the conventional work support system, there is a problem that an assembling error easily occurs in a work in which a plurality of parts are assembled, and it is difficult to improve work efficiency.

One of the objects of the present invention is to provide a manufacturing support system, a method, and a program, which were created in view of the above-mentioned problems, and which can improve the work efficiency in assembling parts. It is to be noted that the present invention is not limited to this purpose, and is an operation and effect derived from each configuration shown in “Embodiments for Carrying Out the Invention” described later. Can be positioned as an objective.

(1) The disclosed manufacturing support system is a manufacturing support system for assisting a worker in assembling parts. The present system has a built-in photographing device for photographing a real scene in the field of view of the worker and a display device for superimposing and displaying a virtual image on the real scene, and includes smart glasses mounted on the head of the worker. In addition, there is provided a first database that defines a relationship between an assembling pattern of the plurality of components with respect to an assembling target and identification information corresponding thereto. Furthermore, a control device is provided for displaying a virtual image of the component on the display device so as to match a coordinate system of a real scene in the field of view of the worker.

制御 The control device includes an extraction unit, an acquisition unit, a recognition unit, and a drawing unit. The extraction unit extracts the identification information included in a work instruction document in which the contents of the assembling work are described, from a photographed image of the photographing device. The acquisition unit acquires an assembly pattern of the component corresponding to the identification information based on the first database. The recognition unit recognizes the assembly target from a captured image of the imaging device. The drawing unit draws a virtual image of the component in a completed position and orientation in which the component is assembled to the assembly target.

(2) It is preferable that a second database is provided in which a relationship between a storage location of the component, the number (use number) of the component to be assembled to the assembly target, and the identification information is defined. In this case, the acquisition unit acquires, based on the second database, information on the storage location of the component corresponding to the identification information and information on the number of the components. Further, the recognition unit recognizes the storage location from a captured image of the imaging device. Further, the drawing unit displays information on the number of the components near the storage location.

(3) It is preferable that the recognition unit recognizes a marker installed in the storage location in the captured image.
(4) Preferably, the control device includes a determination unit and a highlighting unit. In this case, the determination unit determines whether or not the assembly state of the component matches the assembly pattern based on the captured image of the imaging device and the identification information. The emphasis display unit highlights a part in which the assembly state of the component does not match the assembly pattern so as to conform to a coordinate system of a real scene in the field of view of the worker.

By extracting the identification information included in the work instruction and drawing the virtual image of the part at the corresponding position and orientation, the type, the number, and the layout of the parts to be assembled are different for each work instruction. However, it is possible to set an example of assembling in a correct position and orientation, and to assist the assembling work. Further, it is not necessary for the operator to memorize the assembly state of the different parts for each work instruction document, so that assembly errors can be reduced, and productivity (work efficiency) can be improved.

It is a figure showing the application object of a manufacturing support system. It is a figure for explaining composition of a manufacturing support system. FIG. 2 is a block diagram illustrating a hardware configuration of the manufacturing support system. FIG. 3 is a block diagram illustrating a software configuration of a manufacturing support program. (A) and (B) are configuration examples of a database. (A)-(C) is a figure for demonstrating the example of a display of the virtual image of a component. (A), (B) is a figure for demonstrating the example of a display of the number of components. It is a figure for explaining an example of highlighting of an inspection result. It is a flowchart for demonstrating the manufacturing support technique in an assembly process. It is a flowchart for explaining the manufacturing support method in the inspection process. (A), (B) is a figure for demonstrating the modification of a manufacturing support system.

Hereinafter, a manufacturing support system, method, and program as an embodiment will be described with reference to the drawings. The embodiment described below is merely an example, and there is no intention to exclude various modifications and application of technology not explicitly described in the following embodiment. Each configuration of the present embodiment can be variously modified and implemented without departing from the spirit thereof. Further, they can be selected as needed, or can be appropriately combined.

The manufacturing support system according to the present embodiment is a system using a wearable gadget that assists the worker 51 in assembling parts, and is applied to a manufacturing process of the automobile 50 as shown in FIG. Here, a description will be given of support for the worker 51 to take out parts from the material storage shelf 53 and assemble them to the automobile 50. Specific examples of parts to be assembled include electrical components such as relays, switches, and lamps, as well as glass, bumpers, power trains, seats, and tires. Specific examples of components to be assembled (devices to which the components are assembled) include a relay box, a switch box, an instrument panel, a door, and a vehicle body. There are a wide variety of objects to be assembled and types of parts in the work of assembling parts, and the manufacturing support system, method, and program of the present invention can be applied to work of assembling all parts.

The content of the work support is the provision of visual information using the smart glass 10 attached to the head of the worker 51. As shown in FIG. 2, the smart glass 10 is fixed to a helmet 52 of an operator 51. The smart glass 10 incorporates a display device 13 and a photographing device 16. The photographing device 16 is a camera that photographs the actual scene in the field of view of the worker 51 and the work instruction sheet 54. The display device 13 is a device that superimposes and displays a virtual image on a real scene in the field of view of the worker 51. The display method may be a method of projecting a virtual image on the retina of the worker 51 (retinal projection type), a method of projecting a virtual image on the surface of a half mirror (transmission type), or an image captured by the imaging device 16. A method (non-transmissive type) of displaying an image in which a real scene and a virtual image are combined may be displayed.

も In either method, various virtual images are displayed so as to be compatible with the coordinate system of the real scene viewed from the viewpoint of the worker 51. This virtual image is a perspective projection image (or a deformed version of the perspective projection image) that conforms to the coordinate system of the real scene, and is rendered, for example, as a simple polygon model based on three-dimensional CAD (Computer-Aided Design) data of the part Is done. In the existing display methods as disclosed in

Patent Documents

1 and 2, the drawing object is a two-dimensional object. On the other hand, in the present display method, the drawing objects are three-dimensional objects, and each drawing object is virtually arranged in the coordinate system of the real scene.

The content of work support is controlled by any control device (computer, computer). The “electronic control device” here may be built in the smart glass 10 or may be prepared separately from the smart glass 10. For example, the program may be installed in a server 30 or a workstation on the network 19, and the display content calculated there may be displayed on the smart glass 10. In this case, as shown in FIG. 2, the server 30 connected to the network 19 via the repeater 36 may be used. Alternatively, a program that operates even when not connected to the network 19 may be installed in the smart glass 10.

[1. Hardware configuration]
FIG. 3 is a block diagram showing a hardware configuration of the manufacturing support system realized by cooperation between the smart glass 10 and the server 30. The smart glass 10 includes a recording medium drive 11, a storage device 12, a display device 13, an input device 14, a communication device 15, a photographing device 16 (camera) for reading a program recorded on a recording medium 18 (removable medium). A microphone 17 and an electronic control unit 20 are provided. Similarly, the server 30 includes a recording medium drive 31, a storage device 32, a display device 33, an input device 34, a communication device 35, and an electronic control device 40. The function of each element is almost the same as that of the element with the same name built in the smart glass 10, and therefore the description is omitted.

The recording medium 18 is, for example, a flash memory card, a semiconductor memory device conforming to the universal serial bus standard, an optical disk, or the like. The recording medium drive 11 is a reading device that reads information (programs and data) recorded and stored on the recording medium 18. The storage device 12 is a memory device for storing data and programs stored for a long period of time, and includes, for example, a non-volatile memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), and a solid state drive. It is.

The display device 13 is one of devices for providing visual information to the worker 51. In the retinal projection type device, the retina of the worker 51 is an image plane. On the other hand, in a transmission type or non-transmission type device, a half mirror or a display (for example, a liquid crystal display (Liquid Crystal Display, LCD) or an organic EL display (Organic Electro-Luminescence Display, OELD)) is arranged within the field of view of the worker 51. ] Is the imaging plane.

The input device 14 is one of pointing devices for inputting input signals to the

electronic control devices

20 and 40. As a specific example of the input device 14, a device such as a wireless keyboard and a wireless mouse can be connected to the smart glass 10. An input device 14 such as a push button or a physical key is provided on a temple (a handle, a side portion) of the smart glass 10. The microphone 17 is a device for inputting the voice of the worker 51 as an input signal.

The communication device 15 is a device that manages communication with the outside of the smart glass 10, and is, for example, a device that performs wireless communication with the server 30 via the network 19. The repeater 36 shown in FIG. 2 can be considered to be included in the communication device 15. When input / output devices conforming to the short-range wireless communication standard are connected to the smart glass 10, those input / output devices are connected via the communication device 15.

The photographing device 16 is a digital video camera having a built-in image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The photographing device 16 is attached to a rim (periphery of a lens) or a bridge (a connection portion between left and right lenses) of the smart glass 10 and functions to photograph a field of view of an operator 51 wearing the smart glass 10. Information on the image captured by the image capturing device 16 is transmitted to the electronic control device 20 and the server 30. Note that the “captured image” here includes not only a still image but also an image of one frame (one frame) included in a moving image.

位置 The position and the photographing direction of the photographing device 16 are fixed to the smart glass 10, and the position of the display device 13 is also fixed to the smart glass 10. Therefore, as long as the relationship between the smart glass 10 and the line of sight of the worker 51 is kept constant, a virtual image (that is, a coordinate system whose appearance is the same as that of the real scene photographed by the photographing device 16) appears to exist in the same environment. A three-dimensional object that matches the actual scene) can be drawn on the display device 13. Note that a known AR (Augmented Reality, Augmented Reality) development library can be used to calculate the position and orientation in a three-dimensional space of a subject in an image captured by the imaging device 16. In addition, a known three-dimensional renderer or an AR drawing engine can be used for drawing a three-dimensional object that matches a real scene.

The electronic control unit 20 includes a processor 21 (central processing unit), a memory 22 (main memory, main storage device), an auxiliary storage device 23, an interface device 24, and the like, and is communicably connected to each other via a bus 25. You. The processor 21 is a central processing unit including a control unit (control circuit), an operation unit (operation circuit), a cache memory (register group), and the like. The memory 22 is a storage device that stores programs and data during work, and includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory).

The auxiliary storage device 23 is a memory device that stores data and firmware that are held for a longer time than the memory 22, and includes, for example, a nonvolatile memory such as a flash memory or an EEPROM. The interface device 24 controls input / output (I / O) between the electronic control device 20 and the outside. The electronic control device 20 is connected to the recording medium drive 11, the storage device 12, the display device 13, the input device 14, the communication device 15, and the like via the interface device 24.

The electronic control unit 40 of the server 30 also includes a processor 41 (central processing unit), a memory 42 (main memory, main storage device), an auxiliary storage device 43, an interface device 44, and the like. Connected communicable. The function of each element is almost the same as that of the element having the same name built in the electronic control unit 20, and therefore the description is omitted. Further, there is no intention to limit the specific hardware configuration of the electronic control device to the above configuration, and various known configurations may be applied.

[2. Software configuration]
FIG. 4 is a block diagram for explaining the processing contents of the manufacturing support program 9 executed by the smart glasses 10 and the server 30. These processing contents are recorded as application programs in the

storage devices

12 and 32 of the smart glass 10 and the server 30, the recording medium 18, and the like, and are developed and executed on a memory. When the processing contents executed here are functionally classified, the manufacturing support program 9 includes an extraction unit 3, an acquisition unit 4, a recognition unit 5, a drawing unit 6, a determination unit 7, and a highlighting unit 8. Further, a first database 1 and a second database 2 are provided as databases to which these elements can be referred.

(1) The first database 1 defines a relationship between an assembly pattern of a component with respect to an assembly target and identification information corresponding thereto. The identification information is information for specifying the content of the work instructed by the worker 51, and is called, for example, a work number, a work target vehicle number, an instruction target vehicle number, and the like. In the present embodiment, as shown in FIG. 2, it is assumed that the work number is described in the work instruction 54 presented by the work manager. It is assumed that the work instruction 54 is presented by the work manager before the worker 51 performs the assembling work, or is attached to the automobile 50 to be assembled.

(5) The contents of the first database 1 are illustrated in FIG. The first database 1 defines at least a relationship between an operation number for identifying an assembling operation and information (an assembling position and an assembling direction) for specifying a layout (an assembling pattern) of components to be assembled in the operation. You. If there are a plurality of types of parts to be assembled in the operation, an assembly pattern of a plurality of types of parts is set for one operation number. In the example shown in FIG. 5A, an assembling pattern of three types of parts (a-01, a-02, a-03) is set for the work number "ABCDE01234".

(4) If the type, grade, destination, etc. of the automobile 50 to be worked on are different, the types of parts and the assembling patterns will also be different. Generally, in assembling the automobile 50, the assembling patterns to be acquired by the worker 51 range from tens to hundreds of patterns, and it is said that memorizing these patterns requires months to years of experience. ing. On the other hand, in the present embodiment, all of these patterns are defined in the first database 1 and are linked to the operation numbers. Thereby, the worker 51 is released from the memorization work of the assembly pattern.

The mounting position of the component can be specified by at least three parameters based on a coordinate system fixed to the mounting target (for example, a relay box). Similarly, the assembling direction of the component can be specified by at least three parameters based on a coordinate system fixed to the assembling target. Therefore, information for specifying the layout of one component can be uniquely specified by using six or more parameters.

The second database 2 defines the relationship between the identification information (work number), the storage location and the number of used parts (the number of assembled parts). The component storage location is, for example, information indicating which tray is in which of the many component trays stored in the material storage shelf 53 shown in FIG. If the storage location of a component used in a certain operation is always fixed, the position (storage location) of the component can be specified by at least three parameters. On the other hand, when there is a possibility that the storage location of the component is changed depending on the work situation, it is preferable to add information for specifying the storage location to each record of the second database 2.

In the example shown in FIG. 5B, the number of parts “a-01” used in the operation number “ABCDE01234” is 3, and the storage location is the location code “aA11bB22cC33dD44”. A character string, a symbol, a bar code, a two-dimensional code, or the like representing the place code “aA11bB22cC33dD44” is posted at a place where the part is actually stored. By assigning a different location code to each component, the storage location of each component can be uniquely specified.

The extracting unit 3 extracts identification information (work number) included in the work instruction sheet 54 in which the contents of the assembling work are described, from the image captured by the imaging device 16. It is assumed that the work instruction 54 is photographed before the worker 51 performs the work of assembling parts. The extraction unit 3 of the present embodiment has a function of extracting a work number by reading a barcode of the work instruction sheet 54 shown in FIG. The information of the work number extracted here is transmitted to the acquisition unit 4.

The acquisition unit 4 has two functions. The first function is information acquisition based on the first database 1, and the second function is information acquisition based on the second database 2. The information obtained here is transmitted to the drawing unit 6.
The former function is to acquire a component assembling pattern corresponding to the identification information (work number) extracted by the extraction unit 3 based on the first database 1. For example, when the work number extracted by the extraction unit 3 is “ABCDE01234”, three types of parts (a-01, a-02, aa) corresponding to this in the first database 1 shown in FIG. -03), an assembling pattern (information of an assembling position and an assembling direction) is acquired.
The latter function is to acquire, based on the second database 2, information on the storage location and the number of used parts corresponding to the identification information (work number). For example, based on the second database 2 shown in FIG. 5 (B), information on the location code and the number of uses is obtained for each of the three types of components (a-01, a-02, a-03).

The recognition unit 5 recognizes a specific object or mark present in an image captured by the image capturing device 16. The recognizing unit 5 has a function of recognizing at least “part assembly target”. For example, in a relay assembling operation (operation number “ABCDE01234” in FIG. 5A), it is determined whether or not the relay box 55 to which the relay is to be installed is in the captured image. In this determination, the three-dimensional CAD data of the relay box 55 is referred to. As shown in FIG. 6A, when the relay box 55 is detected in the photographed image, photographing is performed based on the distribution of characteristic points (points and vertices included in the edge of the relay box 55) in the photographed image. The position and orientation of the relay box 55 based on the position (the position of the imaging device 16) are estimated. Thereby, the appearance of the relay box 55 within the field of view of the worker 51 is specified. The information on the position and orientation of the assembly target specified here is transmitted to the drawing unit 6 and is referred to when drawing a part.

(4) The recognizing unit 5 also has a function of recognizing “part storage location” from the captured image. For example, in the relay assembling work (work number “ABCDE01234” in FIG. 5B), a character string, a symbol, a two-dimensional code, or the like representing the location code “aA11bB22cC33dD44” is recognized. As shown in FIG. 7A, when a marker 58 representing a location code is detected in a captured image, the position and orientation of the marker 58 based on the capturing position are estimated based on the position and shape of the marker 58. Is done. Information on the storage location of the part recognized here is also transmitted to the drawing unit 6.

The drawing unit 6 controls the display content of the display device 13 when assembling components, and has three main functions. The first function is to draw a virtual image of a component at the position and orientation at the time of completion when the component is assembled to the assembly target. Here, the “virtual image” indicates a state (completion drawing) when the assembling of the parts is completed, and serves as a “model” of the assembling work. By performing the assembling work while referring to this, even if the worker 51 has little working experience, it is difficult to make an assembling error, and the working efficiency is improved.

虚 The virtual image of the component is drawn such that the coordinate system matches the assembly target recognized by the recognition unit 5. For example, the virtual image of the component with respect to the relay box 55 shown in FIG. 6A is drawn at a position and orientation corresponding to a state where the component is assembled to the relay box 55. FIG. 6B shows only the virtual image of the part drawn here. By superimposing the virtual image of the component on the relay box 55, it is possible to make it appear as if the component exists on the relay box 55 as shown in FIG.

The second function of the drawing unit 6 is a function of drawing information on the number (use number) of parts used in the assembling work near the relay box 55, and the third function is storing the information on the number. It is drawn near the place. When displaying the number information in the vicinity of the relay box 55, it is preferable to draw the information at a position that does not overlap with the components assembled in the relay box 55, as shown in FIG. Further, if the surface on which each component is assembled to relay box 55 is planar, it is preferable to arrange the virtual images of each component on the plane. Similarly, when drawing the number information near the storage location, it is preferable to draw the information at a position that does not overlap with the marker 58 or the component tray, as shown in FIG. 7B. In addition, not only the number of parts but also the part number and the appearance of the parts may be displayed together, so that the storage state may be transmitted in a more easily understood manner.

The determination unit 7 automatically determines whether or not the content of the work is appropriate when the work of assembling the parts is completed. Here, based on the captured image and the identification information (work number), it is determined whether or not the assembly state of the component matches the assembly pattern defined in the first database 1. For example, each component in the captured image may be specified, and the position and orientation of each component with respect to the assembly target may be estimated to compare with the assembly pattern. Alternatively, an image in a completed state prepared in advance and a captured image may be compared.

判定 The determining unit 7 of the present embodiment determines the assembly state of the parts by using the learned auto encoder. Specifically, the captured image is input to the auto encoder, the output image is obtained, and the difference between the input image and the output image is calculated. As the learning data of the auto-encoder, thousands of images obtained by photographing an assembly target in which components are correctly assembled from various positions and angles are used. When a captured image is input to such an auto encoder, if the assembled state in the captured image is appropriate, the output image is almost the same as the input image. On the other hand, if there is an inappropriate part in the assembled state, the image at that part is converted into an “image close to the correct assembled state” and output. In other words, an improperly assembled portion can be detected as a difference between an input image and an output image for the auto encoder. Therefore, the determination unit 7 calculates a difference between the input image and the output image, and calculates a location where the difference is equal to or more than a predetermined threshold value as a “location having a high degree of abnormality”.

The highlighting section 8 highlights the “place with a high degree of abnormality” determined by the determining section 7. Here, a part where the assembly state of the parts does not match the assembly pattern is highlighted so as to conform to the coordinate system of the actual scene in the field of view of the worker. The highlighter 8 according to the present embodiment has a function of displaying an image in which “a portion having a high degree of abnormality” is highlighted in red on the display device 13. As a result, as shown in FIG. 8, a portion having an improper assembling state is emphasized, and the presence of the assembling error is transmitted to the worker 51 in an easily understandable manner.

[3. flowchart]
FIG. 9 is a flowchart for explaining the flow of the component assembling process. The flag A in this flow indicates whether or not the identification information (work number) has been read, and is set to A = 0 in an initial state where the identification information has not been read. First, a photographed image is photographed by the photographing device 16 and its contents are recognized (step A1). Further, the value of the flag A is determined (step A2), and when A = 0, it is determined whether or not the work instruction 54 has been recognized in the captured image (step A3).

Here, when the work instruction 54 is recognized, the extraction unit 3 extracts the work number described in the work instruction 54 (step A4). In response to this, the acquisition unit 4 refers to the first database 1 and acquires an assembly pattern of the part corresponding to the work number (step A5). Further, the acquiring unit 4 refers to the second database 2 to acquire information of the storage location and the number of used parts corresponding to the work number (step A6). Thereafter, the value of the flag A is set to A = 1 (step A7). In the next and subsequent calculation cycles, control proceeds from step A1 to step A8.

In step A8, it is determined whether or not the marker 58 representing the location code has been recognized in the captured image. Here, when the marker 58 is detected, as shown in FIG. 7B, information on the storage location and the number of used parts is displayed on the display device 13 of the smart glass 10. The worker 51 can take out the parts from the material storage shelf 53 while referring to this, and prepare for the assembling work. Also, in step A10, it is determined whether or not the component installation target has been recognized. Here, when the assembly target is recognized, a virtual image of the component is displayed on the display device 13 as shown in FIG. The worker 51 can assemble the parts while referring to this.

FIG. 10 is a flowchart for explaining the flow of the inspection process at the time of completing the assembly of the components. The inspection process may be started when the worker 51 performs a predetermined input operation (for example, a predetermined voice instruction or a button operation) when the assembling operation is completed. Alternatively, the completion of the assembling work of the components may be automatically detected by image analysis, and the inspection process may be started.

In the former case, in step B1, various information related to the input operation by the worker 51 is input. Here, when a predetermined inspection start condition (for example, a predetermined voice instruction is given) is satisfied (step B2), the image captured by the image capturing device 16 is input to the auto encoder as an input image, and the output image is obtained. (Step B3). In addition, a difference between the input image and the output image is calculated, and a portion where the difference is equal to or larger than the threshold is highlighted. The worker 51 can correct the improper assembling state while referring to this.

[4. Action / Effect]
(1) In the above-described manufacturing support system, method, and program, identification information (work number) included in the work instruction sheet 54 is extracted, and a virtual image of the part is drawn at a position and an orientation corresponding to the identification information. The relationship between the identification information (work number) and the component assembling pattern is defined in the first database 1 in advance. Thereby, even when the type, the number of use, and the layout of the parts related to the assembling work are different for each work instruction sheet 54, a model of the assembling of the parts can be shown in a correct position and a correct direction. We can support the work.

{Circle around (4)} The operator 51 does not need to memorize the assembling patterns of the different parts for each work instruction 54, thereby reducing assembling errors. Further, since a virtual image serving as a model is always displayed during the assembling work, there is no fear that the worker 51 will remember the wrong assembling, and the skill of the worker 51 can be quickly increased. Therefore, work efficiency (productivity) related to the work of assembling parts can be improved.

(2) The relationship between the identification information (work number) and the storage location and the number of parts used is prescribed in the second database 2 in advance. By using this information and drawing the information on the number of used parts near the storage location of the parts, it is possible to prevent mistakes in the parts being mixed up. For example, in an assembling operation in which a plurality of parts having similar appearances are mixed, it is possible to reduce pickup errors (errors in taking out parts). Therefore, the failure rate in the assembling work can be reduced, and the working efficiency can be further improved.

(3) As shown in FIG. 7A, by specifying the storage location of the component using the marker 58, it becomes possible to flexibly respond to a change in the specification of the component or a change in the storage location. Efficiency can be further improved. For example, when the storage location of the part is changed, the marker 58 may be moved together with the part. If the type or storage location of the component is not changed, support based on the appearance of the component or the storage location may be performed instead of the marker 58.

(4) As shown in FIG. 8, by highlighting a portion where the degree of abnormality in the assembled state is high so as to conform to the coordinate system of the actual scene, it is possible to make the worker 51 easily notice that portion. Thereby, the assembling mistake can be corrected immediately, and the working efficiency can be further improved. In addition, since an assembling defect resulting from a mistake in memory or misunderstanding of the worker 51 is immediately fed back, the proficiency of the worker 51 can be quickly increased, and human resources can be cultivated in a short period of time. .

[5. Modification]
The above embodiment is merely an example, and there is no intention to exclude various modifications and application of technology that are not explicitly described in the present embodiment. Each configuration of the present embodiment can be variously modified and implemented without departing from the spirit thereof. Further, they can be selected as needed, or can be appropriately combined.

For example, a motion sensor (acceleration sensor, direction sensor, gyro sensor, geomagnetic sensor, etc.) may be built in the smart glass 10. By grasping the postures of the display device 13 and the photographing device 16 with high accuracy using the motion sensor, it is possible to more accurately estimate the actual scene and the position and orientation of the subject within the field of view of the worker 51, and to realize a virtual image of the actual scene. The superimposition accuracy can be improved, and the realism and realism of the virtual image can be enhanced.

The minimum configuration of the manufacturing support system, the manufacturing support method, and the manufacturing support program in this case includes the first database 1, the extraction unit 3, the acquisition unit 4, the recognition unit 5, and the drawing unit 6. The device in which these elements are stored can be set arbitrarily, and the functions can be distributed to a plurality of devices. When distributing functions, at least a plurality of devices need only be connected to be able to communicate with each other by wire or wirelessly.

Regarding the distribution of the above functions, as shown in FIG. 11A, the smart glass 10 is in charge of elements related to drawing on the display device 13 (for example, the drawing unit 6 and the highlighting unit 8), and other elements (for example, , The extracting unit 3, the acquiring unit 4, the recognizing unit 5, and the determining unit 7) may be assigned to the assembling work computer 59. Further, the first database 1 and the second database 2 may be stored in the production command server 60. The production command server 60 is a large computer that controls the operation state of the production line and the operation of the production robot, and is connected to the assembly work computer 59 by a wired cable. The assembling work computer 59 is a computer disposed near a work place where the assembling work is performed, and is wirelessly connected to the smart glass 10 via the repeater 36.

With such a device configuration, it is possible to accurately display a virtual image suitable for a real scene on the display device 13 even when the electronic control device 20 built in the smart glass 10 has a low arithmetic processing capability. In addition, by arranging the

databases

1 and 2 in the production command server 60, work support using the database used in the existing production system becomes easy. Note that if sufficient computation processing capability can be expected from the smart glass 10, a system that can operate by itself as shown in FIG. 11B may be configured. In this case, since communication with another computer is not required, work support in an offline environment is possible.

[6. Appendix]
The following supplementary notes are disclosed with respect to the embodiment including the above-described modification.
[Appendix 1]
For a worker wearing a smart glass equipped with a built-in imaging device for photographing a real scene in a field of view and a display device for superimposing and displaying a virtual image on the real scene, the display device displays a virtual image conforming to the coordinate system of the real scene. A manufacturing support method for supporting the assembly work of parts by displaying
From the photographed image of the photographing device, extract the identification information included in the work instruction sheet in which the contents of the assembling work are described,
Based on a first database in which a relationship between a plurality of assembling patterns of the component to the assembling target and identification information corresponding thereto is obtained, an assembling pattern of the component corresponding to the identification information is obtained,
From the photographed image of the photographing device, recognize the assembly target,
A manufacturing support method, wherein a virtual image of the component is drawn at a position and an orientation at the time of completion when the component is assembled to the assembly target.

[Appendix 2]
Based on a second database in which the relationship between the storage location of the component and the number of components to be assembled to the assembly target and the identification information is defined, information on the storage location of the component corresponding to the identification information and the Get the number information and
Recognizing the storage location from the photographed image of the photographing device,
The manufacturing support method according to claim 1, wherein information on the number of the parts is displayed near the storage location.

[Appendix 3]
3. The manufacturing support method according to claim 2, further comprising recognizing a marker installed at the storage location in the captured image.

[Appendix 4]
Based on the captured image of the imaging device and the identification information, determine whether the assembly state of the component matches the assembly pattern,
The part according to any one of supplementary notes 1 to 3, wherein a part in which the assembly state of the component does not match the assembly pattern is highlighted so as to conform to a coordinate system of a real scene in the field of view of the worker. Manufacturing support method as described.

[Appendix 5]
For a worker wearing a smart glass equipped with a built-in imaging device for photographing a real scene in a field of view and a display device for superimposing and displaying a virtual image on the real scene, the display device displays a virtual image conforming to the coordinate system of the real scene. Is a manufacturing support program that supports the assembly work of parts by displaying
From the photographed image of the photographing device, extract the identification information included in the work instruction sheet in which the contents of the assembling work are described,
Based on a first database in which a relationship between a plurality of assembling patterns of the component to the assembling target and identification information corresponding thereto is obtained, an assembling pattern of the component corresponding to the identification information is obtained,
From the photographed image of the photographing device, recognize the assembly target,
A manufacturing support program for causing a computer to execute a process of drawing a virtual image of the component at a completed position and orientation in which the component is assembled to the assembly target.

[Appendix 6]
Based on a second database in which the relationship between the storage location of the component and the number of components to be assembled to the assembly target and the identification information is defined, information on the storage location of the component corresponding to the identification information and the Get the number information and
Recognizing the storage location from the photographed image of the photographing device,
The manufacturing support program according to claim 5, wherein the program causes a computer to execute a process of displaying the information on the number of the parts near the storage location.

[Appendix 7]
The manufacturing support program according to claim 6, wherein the program causes a computer to execute a process of recognizing a marker installed in the storage location in the captured image.

[Appendix 8]
Based on the captured image of the imaging device and the identification information, determine whether the assembly state of the component matches the assembly pattern,
8. The computer according to any one of claims 5 to 7, further comprising: causing a computer to execute a process of highlighting a part in which the assembly state of the component does not match the assembly pattern so as to match a coordinate system of a real scene in the field of view of the worker. Manufacturing support program described in 1.

DESCRIPTION OF SYMBOLS 1 1st database 2 2nd database 3 Extraction part 4 Acquisition part 5 Recognition part 6 Drawing part 7 Judgment part 8 Highlight display part 9 Manufacturing support program 10 Smart glasses 13 Display device 16 Imaging device 30 Server 50 Car 51 Worker 52 Helmet 53 Material storage shelf 54 Work instructions 55 Relay box (for assembly)
56 relays (parts)

Claims

A manufacturing support system for assisting a worker in assembling parts,
Incorporating a photographing device for photographing a real scene in the field of view of the worker and a display device for superimposing and displaying a virtual image on the real scene, a smart glass mounted on the head of the worker,
A first database that defines a relationship between an assembling pattern of the plurality of components to the assembling target and identification information corresponding thereto,
A control device for displaying a virtual image of the component on the display device so as to conform to a coordinate system of a real scene in the field of view of the worker,
The control device,
From an image captured by the image capturing device, an extraction unit configured to extract the identification information included in a work instruction sheet in which the content of the assembly operation is described,
An acquisition unit that acquires an assembly pattern of the component corresponding to the identification information based on the first database,
From a photographed image of the photographing device, a recognition unit that recognizes the assembly target,
A drawing unit that draws a virtual image of the component at a position and orientation at the time of completion when the component is assembled to the assembly target.
The storage location of the parts and a second database in which the relationship between the number of the parts and the identification information to be assembled to the assembly target is defined,
The acquisition unit, based on the second database, acquires information on the storage location of the component corresponding to the identification information and information on the number of the component,
The recognition unit recognizes the storage location from the captured image of the imaging device,
The manufacturing support system according to claim 1, wherein the drawing unit displays information on the number of the components near the storage location.
The manufacturing support system according to claim 2, wherein the recognition unit recognizes a marker installed in the storage location in the captured image.
The control device,
A determination unit configured to determine whether an assembly state of the component matches the assembly pattern based on a captured image of the imaging device and the identification information;
The image display device according to claim 1, further comprising: a highlighting unit that highlights a part in which the assembly state of the component does not match the assembly pattern so as to conform to a coordinate system of a real scene in the field of view of the worker. 4. The manufacturing support system according to any one of items 3.
For a worker wearing a smart glass equipped with a built-in imaging device for photographing a real scene in a field of view and a display device for superimposing and displaying a virtual image on the real scene, the display device displays a virtual image conforming to the coordinate system of the real scene. A manufacturing support method for supporting the assembly work of parts by displaying
From the photographed image of the photographing device, extract the identification information included in the work instruction sheet in which the contents of the assembling work are described,
Based on a first database in which a relationship between a plurality of assembling patterns of the component to the assembling target and identification information corresponding thereto is obtained, an assembling pattern of the component corresponding to the identification information is obtained,
From the photographed image of the photographing device, recognize the assembly target,
A manufacturing support method, wherein a virtual image of the component is drawn at a position and an orientation at the time of completion when the component is assembled to the assembly target.
For a worker wearing a smart glass equipped with a built-in imaging device for photographing a real scene in a field of view and a display device for superimposing and displaying a virtual image on the real scene, the display device displays a virtual image conforming to the coordinate system of the real scene. Is a manufacturing support program that supports the assembly work of parts by displaying
From the photographed image of the photographing device, extract the identification information included in the work instruction sheet in which the contents of the assembling work are described,
Based on a first database in which a relationship between a plurality of assembling patterns of the component to the assembling target and identification information corresponding thereto is obtained, an assembling pattern of the component corresponding to the identification information is obtained,
From the photographed image of the photographing device, recognize the assembly target,
A manufacturing support program for causing a computer to execute a process of drawing a virtual image of the component at a completed position and orientation in which the component is assembled to the assembly target.