WO2020213074A1 - 整備サポート拡張現実出力システム、コンピュータ、端末、整備サポート拡張現実出力方法及びプログラム - Google Patents

整備サポート拡張現実出力システム、コンピュータ、端末、整備サポート拡張現実出力方法及びプログラム Download PDF

Info

Publication number
WO2020213074A1
WO2020213074A1 PCT/JP2019/016383 JP2019016383W WO2020213074A1 WO 2020213074 A1 WO2020213074 A1 WO 2020213074A1 JP 2019016383 W JP2019016383 W JP 2019016383W WO 2020213074 A1 WO2020213074 A1 WO 2020213074A1
Authority
WO
WIPO (PCT)
Prior art keywords
maintenance
vehicle
augmented reality
support information
maintenance worker
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/016383
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
茂樹 中澤
秀明 南雲
崇治 櫻田
大河 浦瀬
憲之 根本
将仁 谷口
浩 秋葉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Uhuru Corp
Logisteed Ltd
Original Assignee
Hitachi Transport System Ltd
Uhuru Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hitachi Transport System Ltd, Uhuru Corp filed Critical Hitachi Transport System Ltd
Priority to PCT/JP2019/016383 priority Critical patent/WO2020213074A1/ja
Priority to JP2021514702A priority patent/JP7189332B2/ja
Publication of WO2020213074A1 publication Critical patent/WO2020213074A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management

Definitions

  • the present invention relates to a maintenance support augmented reality output system, a computer, a terminal, a maintenance support augmented reality output method, and a program that output support information according to the proficiency level of a maintenance worker who maintains a vehicle as augmented reality.
  • the present invention provides a maintenance support augmented reality output system, a computer, a terminal, a maintenance support augmented reality output method and a program capable of outputting support information according to the proficiency level of a maintenance worker as augmented reality.
  • the purpose a maintenance support augmented reality output system, a computer, a terminal, a maintenance support augmented reality output method and a program capable of outputting support information according to the proficiency level of a maintenance worker as augmented reality.
  • the present invention provides the following solutions.
  • the present invention is a maintenance support augmented reality output system that outputs support information according to the proficiency level of a maintenance worker who maintains a vehicle as augmented reality.
  • the first acquisition means for acquiring the first maintenance work image obtained by photographing the scenery of the maintenance work by the maintenance worker, and
  • the first analysis means for analyzing the acquired first maintenance work image and Specific means to identify the vehicle to be maintained and
  • a first determination means for determining the proficiency level of the maintenance worker based on the analysis result of the first maintenance work image, and
  • a determination means for the maintenance worker to determine support information for servicing the vehicle from the identified vehicle and the determined proficiency level.
  • An output means for outputting the determined support information as augmented reality so as to overlap the vehicle on the display unit of the terminal used by the maintenance worker.
  • the maintenance support augmented reality output system that outputs support information according to the proficiency level of the maintenance worker who maintains the vehicle as augmented reality is the first image of the scenery of the maintenance work by the maintenance worker.
  • the maintenance work image is acquired, the acquired first maintenance work image is analyzed, the vehicle to be maintained is specified, and the proficiency level of the maintenance worker is determined based on the analysis result of the first maintenance work image.
  • the support information for the maintenance worker to maintain the vehicle is determined from the specified vehicle and the determined proficiency level, and the determined display unit of the terminal used by the maintenance worker is used.
  • the support information is output as augmented reality so as to overlap the vehicle.
  • the present invention is in the category of systems, but in other categories such as methods and programs, the same actions and effects are exhibited according to the categories.
  • the present invention is a computer that outputs support information according to the proficiency level of a maintenance worker who maintains a vehicle as augmented reality.
  • the first acquisition means for acquiring the first maintenance work image obtained by photographing the scenery of the maintenance work by the maintenance worker, and
  • the first analysis means for analyzing the acquired first maintenance work image and Specific means to identify the vehicle to be maintained and
  • a first determination means for determining the proficiency level of the maintenance worker based on the analysis result of the first maintenance work image, and
  • a determination means for the maintenance worker to determine support information for servicing the vehicle from the identified vehicle and the determined proficiency level.
  • An output means for outputting the determined support information as augmented reality so as to overlap the vehicle on the display unit of the terminal used by the maintenance worker.
  • the present invention is a terminal that outputs support information according to the proficiency level of a maintenance worker who maintains a vehicle as augmented reality.
  • the first output means for outputting the first maintenance work image obtained by photographing the scenery of the maintenance work by the maintenance worker, and Support for the maintenance worker to maintain the vehicle determined from the specified vehicle to be maintained and the proficiency level of the maintenance worker determined based on the analysis result of the first maintenance work image. How to get information and how to get it
  • a second output means that outputs the acquired support information as augmented reality so as to overlap the vehicle on its own display unit.
  • a maintenance support augmented reality output system a computer, a terminal, a maintenance support augmented reality output method and a program capable of outputting support information according to the proficiency level of a maintenance worker as augmented reality are provided. It becomes possible.
  • FIG. 1 is a diagram showing an outline of the maintenance support augmented reality output system 1.
  • FIG. 2 is an overall configuration diagram of the maintenance support augmented reality output system 1.
  • FIG. 3 is a diagram showing a flowchart of maintenance support augmented reality output processing executed by the computer 10.
  • FIG. 4 is a diagram showing a flowchart of the proficiency evaluation process executed by the computer 10.
  • FIG. 5 is a diagram showing a flowchart of a report automatic creation process executed by the computer 10.
  • FIG. 6 is a diagram schematically showing an example in which the computer 10 outputs the support information to the maintenance worker terminal as augmented reality.
  • FIG. 1 is a diagram for explaining an outline of the maintenance support augmented reality output system 1 which is a preferred embodiment of the present invention.
  • the maintenance support augmented reality output system 1 is a computer system composed of a computer 10 and outputs support information according to the proficiency level of a maintenance worker who maintains a vehicle as augmented reality.
  • the maintenance support augmented reality output system 1 is a maintenance worker terminal (for example, a mobile terminal such as a smartphone or tablet terminal or a wearable terminal such as a head-mounted display such as a smart glass) possessed by a maintenance worker who maintains a vehicle. Other terminals and devices such as mobile imaging devices arranged at predetermined positions may be included. Further, the maintenance support augmented reality output system 1 may be realized by one computer such as a computer 10, or may be realized by a plurality of computers such as a cloud computer.
  • the maintenance support augmented reality output system 1 may be composed of a computer 10 and other terminals and devices. In this case, the maintenance support augmented reality output system 1 executes each process described later by any or a combination of the computer 10 and other terminals and devices.
  • the computer 10 is connected to other terminals and devices such as a maintenance worker terminal and a photographing device so as to be capable of data communication via a public network or the like, and executes necessary data transmission / reception.
  • the computer 10 acquires the first maintenance work image obtained by capturing the scenery of the maintenance work by the maintenance worker with a moving image, a still image, or the like.
  • the computer 10 acquires, for example, a first maintenance work image taken by a maintenance worker terminal or a photographing device.
  • the maintenance worker terminal or the photographing device captures the scenery of the maintenance work of the maintenance worker as a first maintenance work image such as a moving image or a still image, and transmits the photographed first maintenance work image to the computer 10.
  • the computer 10 acquires the first maintenance work image from the terminal or the like.
  • the computer 10 analyzes the acquired first maintenance work image.
  • the computer 10 extracts, for example, feature points (for example, shape, contour, hue) and feature amounts (for example, statistical numerical values such as average, variance, and histogram of pixel values) of the first maintenance work image. , Perform the analysis.
  • the computer 10 analyzes the vehicle shown in the first maintenance work image and the movement of the maintenance worker (for example, work content, work speed, work process, how to handle tools). It will be.
  • the computer 10 identifies the vehicle to be maintained.
  • the computer 10 identifies the vehicle to be maintained, for example, based on the analysis result of the first maintenance work image.
  • the computer 10 records, for example, a vehicle database in which a vehicle feature point or feature amount is previously recorded in association with a vehicle identifier (for example, vehicle serial number, vehicle number, model, reference number, control number, name). This vehicle is identified by reference.
  • the computer 10 compares the extracted feature points and feature quantities with the feature points and feature quantities recorded in the vehicle database, and corresponds to the feature points and feature quantities in the vehicle database that match the extracted feature points and feature quantities. Identify the attached vehicle identifier. As a result, the computer 10 identifies the vehicle to be maintained as shown in the first maintenance work image.
  • the computer 10 receives input of an identifier of a vehicle to be maintained from, for example, a maintenance worker terminal or other terminals or devices, and identifies the vehicle to be maintained based on the received identifier.
  • the maintenance worker terminal and other terminals and devices accept the input of the identifier of the vehicle to be maintained, and transmit the accepted identifier to the computer 10.
  • the computer 10 acquires the identifier of the vehicle to be maintained. As a result, the computer 10 identifies the vehicle to be maintained.
  • the computer 10 determines the proficiency level of the maintenance worker based on the analysis result of the first maintenance work image.
  • the computer 10 refers to, for example, a proficiency level database in which the operation of the maintenance worker and the proficiency level of the maintenance work of the maintenance worker who performs this operation are recorded in association with each other. To judge.
  • the computer 10 identifies the operation of the maintenance worker based on the extracted feature points and feature quantities.
  • the computer 10 compares this identified action with the action recorded in the proficiency database, and identifies the proficiency associated with the action in the proficiency database that matches the identified action.
  • the computer 10 determines the specified proficiency level as the proficiency level of the maintenance worker. As a result, the computer 10 determines the proficiency level of the maintenance worker shown in the first maintenance work image.
  • the computer 10 Based on the identified vehicle and the determined proficiency level, the computer 10 provides support information for the maintenance worker to maintain the vehicle (for example, instructions on how to use tools, instructions on the position of the body, and work contents. Information on the operation required for the maintenance worker to perform the maintenance work, such as instructions) is determined.
  • the computer 10 determines this support information by, for example, referring to a support information database in which the identifier of the vehicle, the proficiency level of the maintenance worker, and the support information are recorded in association with each other.
  • the computer 10 compares the identified vehicle identifier and the determined proficiency level with this support information database, and obtains support information associated with the identifier and proficiency level in the support information database that matches the vehicle identifier and proficiency level. Identify.
  • the computer 10 determines this identified support information as support information for the maintenance worker to maintain the vehicle.
  • the computer 10 causes the display unit of the maintenance worker terminal to output the determined support information as augmented reality so as to overlap the vehicle.
  • the computer 10 outputs the determined support information to the maintenance worker terminal.
  • the computer 10 transmits this support information to the maintenance worker terminal.
  • the maintenance worker terminal receives this support information.
  • the maintenance worker terminal displays the received support information on its own display unit.
  • the maintenance worker terminal overlaps the vehicle to be photographed by itself (for example, overlaps a part or all of this vehicle, or overlaps a part or all of the part to be maintained of this vehicle). Display information as augmented reality.
  • the computer 10 causes the maintenance worker terminal to output this support information.
  • the maintenance worker terminal will output this support information to its own display unit. Therefore, the maintenance support augmented reality output system 1 outputs the determined support information to the display unit of the maintenance worker terminal as augmented reality so as to overlap the vehicle.
  • the maintenance support augmented reality output system 1 that outputs support information according to the proficiency level of the maintenance worker who maintains the vehicle as augmented reality is a photograph of the scenery of the maintenance work by the maintenance worker. 1
  • the maintenance work image is acquired, the acquired first maintenance work image is analyzed, the vehicle to be maintained is specified based on the analysis result of the first maintenance work image, and the analysis of the first maintenance work image is performed.
  • the proficiency level of the maintenance worker is determined, and the support information for the maintenance worker to maintain the vehicle is determined from the specified vehicle and the determined proficiency level, and the maintenance is performed.
  • the determined support information is output as augmented reality so as to overlap the vehicle on the display unit of the terminal used by the operator.
  • the computer 10 can also be configured to acquire a second maintenance work image obtained by capturing the scenery of the maintenance work of the maintenance worker with a moving image, a still image, or the like after outputting the support information.
  • the computer 10 acquires, for example, a second maintenance work image taken by the maintenance worker terminal or the photographing device.
  • the maintenance worker terminal or the photographing device captures the scenery of the maintenance work of the maintenance worker after outputting the support information as a second maintenance work image such as a moving image or a still image, and captures the second maintenance work image.
  • a second maintenance work image such as a moving image or a still image
  • the computer 10 acquires the second maintenance work image from the terminal or the like.
  • the computer 10 analyzes the acquired second maintenance work image.
  • the computer 10 executes the analysis, for example, by extracting the feature points and feature quantities of the second maintenance work image. As a result, the computer 10 executes the analysis of the operation of the maintenance worker shown in the second maintenance work image.
  • the computer 10 evaluates the proficiency level of the maintenance worker based on this analysis result.
  • This proficiency evaluation is based on, for example, the number of times the maintenance worker has achieved an operation or work according to the support information. When the number of times exceeds the specified number of times, the computer 10 raises the evaluation of the proficiency level of the maintenance worker.
  • the evaluation information is, for example, that when the evaluation of the proficiency level of this maintenance worker in the past is 1, the evaluation of the proficiency level of this maintenance worker is set to 2 by achieving this number of times this time. ..
  • the computer 10 adds the evaluation of the proficiency level of the maintenance worker to the analysis result when determining the proficiency level of the maintenance worker based on the analysis result of the first maintenance work image from the next time onward. Determine the degree.
  • the computer 10 determines, for example, the proficiency level of the maintenance worker this time according to the evaluation of the proficiency level of the maintenance worker.
  • the computer 10 determines the proficiency level of the maintenance worker by using a predetermined coefficient corresponding to the numerical value of the evaluation of the proficiency level of the maintenance worker.
  • the computer 10 can be configured to determine whether or not the operation of the maintenance worker is an operation in accordance with the support information based on the analysis result of the acquired second maintenance work image. In this case, the computer 10 determines, for example, whether or not the operation of the maintenance worker based on the analysis result of the second maintenance work image is an operation in accordance with the determined support information. The computer 10 automatically creates a report based on the determination result.
  • the computer 10 determines that the operation of the maintenance worker based on the analysis result is the operation according to the support information
  • the computer 10 issues a report stating that the work content is appropriate based on the predetermined format. Create automatically. At this time, the computer 10 can also state in this report that the operation of the maintenance worker was appropriate.
  • the computer 10 determines that the operation of the maintenance worker based on the analysis result does not conform to the support information, the computer 10 issues a report stating that the work content is not appropriate based on the predetermined format. Create automatically. At this time, the computer 10 can further describe the operation to be originally performed, the operation actually performed, the improvement portion, and the like in this report.
  • the computer 10 acquires the above-mentioned first maintenance work image (step S01).
  • the maintenance worker terminal and the photographing device take the above-mentioned first maintenance work image, and transmit the first maintenance work image to the computer 10.
  • the computer 10 acquires the first maintenance work image from the terminal or the like.
  • the computer 10 analyzes the acquired first maintenance work image (step S02).
  • the computer 10 analyzes the first maintenance work image by extracting the feature points and feature amounts of the first maintenance work image.
  • the computer 10 will analyze the movements of the vehicle and the maintenance worker in the first maintenance work image.
  • the computer 10 identifies the vehicle to be maintained (step S03).
  • the computer 10 identifies the vehicle to be maintained based on the analysis result of the first maintenance work image.
  • the computer 10 identifies the vehicle by comparing the feature points and feature quantities extracted this time with the vehicle database described above.
  • the computer 10 identifies the identifier of the vehicle corresponding to the feature points and the feature quantities extracted this time based on the vehicle database. By specifying the identifier of this vehicle, the computer 10 will specify the vehicle to be maintained as shown in the first maintenance work image. Further, the computer 10 accepts input of an identifier of the vehicle to be maintained from the maintenance worker terminal, other terminals, devices, etc., and identifies the vehicle to be maintained based on the received identifier.
  • the maintenance worker terminal and other terminals and devices accept the input of the identifier of the vehicle to be maintained, and transmit the accepted identifier to the computer 10. By receiving this identifier, the computer 10 acquires the identifier of the vehicle to be maintained. The computer 10 identifies the vehicle to be maintained by acquiring the identifier of the vehicle to be maintained.
  • the computer 10 determines the proficiency level of the maintenance worker based on the analysis result (step S04).
  • the computer 10 determines the proficiency level of the maintenance worker by comparing the operation of the maintenance worker based on the feature points and feature quantities extracted this time with the proficiency level database described above.
  • the computer 10 identifies the proficiency level of the maintenance worker corresponding to the operation of the maintenance worker based on the feature points and feature quantities extracted this time based on the proficiency level database.
  • the computer 10 determines the proficiency level of the specified maintenance worker as the proficiency level of the maintenance worker. By determining the proficiency level of the maintenance worker, the computer 10 determines the proficiency level of the maintenance worker shown in the first maintenance work image.
  • the computer 10 determines the support information for the maintenance worker to maintain this vehicle from the specified vehicle and the determined proficiency level (step S05).
  • the computer 10 determines the support information by comparing the vehicle and the proficiency level with the support information database described above.
  • the computer 10 specifies the identifier of the specified vehicle and the support information corresponding to the determined proficiency level of the maintenance worker based on the support information database.
  • the computer 10 determines this identified support information as support information for the maintenance worker to maintain the vehicle. By determining this support information, the computer 10 determines the support information for the maintenance worker to maintain the vehicle.
  • the computer 10 causes the display unit of the maintenance worker terminal to output the determined support information as augmented reality so as to overlap the vehicle (step S06).
  • the computer 10 transmits this support information to the maintenance worker terminal.
  • the maintenance worker receives this support information and displays it as augmented reality on his / her display unit.
  • the maintenance worker terminal displays this support information as augmented reality so as to overlap the vehicle to be photographed by itself.
  • the computer 10 will output this support information to the maintenance worker terminal, and the maintenance worker terminal will output this support information to its own display unit. Therefore, the maintenance support augmented reality output system 1 outputs the determined support information to the display unit of the maintenance worker terminal as augmented reality so as to overlap the vehicle.
  • FIG. 2 is a diagram showing a system configuration of the maintenance support augmented reality output system 1 which is a preferred embodiment of the present invention.
  • the maintenance support augmented reality output system 1 is a computer system composed of a computer 10 and outputs support information according to the proficiency level of a maintenance worker who maintains a vehicle as augmented reality.
  • the computer 10 is connected to other terminals and devices such as a maintenance worker terminal and a photographing device so as to be capable of data communication via a public network or the like, and executes necessary data transmission / reception.
  • the maintenance support augmented reality output system 1 may include other terminals and devices such as maintenance worker terminals and photographing devices (not shown). Further, the maintenance support augmented reality output system 1 may be realized by one computer such as a computer 10, or may be realized by a plurality of computers such as a cloud computer.
  • the maintenance support augmented reality output system 1 may be composed of a computer 10 and other terminals and devices. In this case, the maintenance support augmented reality output system 1 executes each process described later by any or a combination of the computer 10 and other terminals and devices.
  • the computer 10 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and enables communication with other terminals and devices as a communication unit.
  • a device for this purpose for example, a Wi-Fi (Wireless-Fidelity) compatible device compliant with IEEE802.11 and the like.
  • the computer 10 includes a data storage unit such as a hard disk, a semiconductor memory, a recording medium, and a memory card as a recording unit.
  • the computer 10 includes various devices and the like that execute various processes as a processing unit.
  • the control unit reads a predetermined program to realize the first maintenance work image acquisition module 20, the output module 21, and the second maintenance work image acquisition module 22 in cooperation with the communication unit. Further, in the computer 10, the control unit reads a predetermined program to realize the recording module 30 in cooperation with the recording unit. Further, in the computer 10, when the control unit reads a predetermined program, the analysis module 40, the vehicle identification module 41, the proficiency level determination module 42, the support information determination module 43, and the proficiency level evaluation module cooperate with the processing unit. 44, maintenance content determination module 45, and report automatic creation module 46 are realized.
  • FIG. 3 is a diagram showing a flowchart of maintenance support augmented reality output processing executed by the computer 10. The process executed by each of the above-mentioned modules will be described together with this process.
  • the first maintenance work image acquisition module 20 acquires the first maintenance work image obtained by capturing the scenery of the maintenance work by the maintenance worker with a moving image, a still image, or the like (step S10). In step S10, the first maintenance work acquisition module 20 acquires the first maintenance work image taken by the maintenance worker terminal or the photographing device.
  • the maintenance worker terminal and the photographing device accept the input operation for instructing the maintenance worker, the manager, and others to take a picture, and the input operation by voice, and take the first maintenance work image.
  • the maintenance worker terminal and the photographing device transmit the photographed first maintenance work image to the computer 10.
  • the first maintenance work image acquisition module 20 acquires the first maintenance work image from the terminal or the like.
  • the analysis module 40 analyzes the acquired first maintenance work image (step S11).
  • the analysis module 40 performs image analysis of the first maintenance work image and extracts feature points and feature quantities in the first maintenance work image.
  • the feature points are, for example, the shape, contour, and hue as described above.
  • the feature amount is, for example, a statistical value such as the average, variance, or histogram of pixel values as described above.
  • the analysis module 40 analyzes the movements of the vehicle and the maintenance worker shown in the first maintenance work image by extracting the feature points and feature quantities.
  • the operation of the maintenance worker is, for example, the work content performed by the maintenance worker, the work speed of the maintenance worker, the work process, and the handling of tools as described above.
  • the vehicle identification module 41 identifies the vehicle to be maintained (step S12). In step S12, the vehicle identification module 41 is based on either or both of the image analysis result of the first maintenance work image or the identifier of the vehicle to be maintained obtained from the maintenance worker terminal or other terminals or devices. , Identify the vehicle to be serviced.
  • the vehicle identification module 41 identifies this vehicle by referring to a vehicle database in which the feature points and amounts of the vehicle are previously recorded in association with the identifier of the vehicle.
  • the vehicle identifier is, for example, the vehicle serial number, vehicle number, model, reference number, control number, or name of the vehicle as described above.
  • This vehicle database is recorded in the recording module 30.
  • the vehicle identification module 41 compares the feature points and feature quantities extracted from the first maintenance work image with the feature points and feature quantities recorded in the vehicle database, and the vehicle database that matches the extracted feature points and feature quantities. Identify the feature points and features recorded in.
  • the vehicle identification module 41 specifies a vehicle identifier associated with the specified feature point or feature amount in the vehicle database.
  • the vehicle identification module 41 identifies the vehicle to be maintained by specifying the identifier of this vehicle.
  • the vehicle identification module 41 acquires the identifier of the vehicle to be maintained for which the maintenance worker terminal, other terminals, devices, etc. have received the input.
  • the maintenance worker terminal, other terminals, devices, and the like accept the input of the identifier of the vehicle to be maintained.
  • the maintenance worker terminal, other terminals, devices, and the like transmit this received identifier to the computer 10.
  • the computer 10 acquires the identifier of the vehicle to be maintained.
  • the vehicle identification module 41 identifies the acquired identifier as an identifier of the vehicle to be maintained.
  • the vehicle identification module 41 identifies the vehicle to be maintained by acquiring the identifier of this vehicle.
  • the proficiency level determination module 42 determines the proficiency level of the maintenance worker based on the analysis result of the first maintenance work image (step S13).
  • step S13 the operation of the maintenance worker shown in the first maintenance work image is specified based on the feature points and feature quantities extracted from the first maintenance work image.
  • the proficiency level determination module 42 identifies the operation of the maintenance worker based on the changes in the feature points and the feature amounts.
  • the proficiency level determination module 42 is based on, for example, the movement of the body of the maintenance worker, the movement of the tools used by the maintenance worker, and the position between the vehicle and the maintenance worker based on the changes in the feature points and the feature amounts. Identify the relationship as an action.
  • the proficiency level determination module 42 refers to a proficiency level database that records in advance the movement of the maintenance worker and the proficiency level of the maintenance work of the maintenance worker who performs this movement. Identify proficiency. This proficiency database is recorded in the recording module 30.
  • This proficiency level is a numerical value of the degree of skill of maintenance workers in maintenance work. This proficiency level is, for example, divided into a predetermined number of stages, and the higher the skill level of the maintenance work, the higher the value, and the lower the level, the lower the value. Specifically, it is divided into stages from 1 to 10, and the higher the degree of skill, the closer to 10.
  • the proficiency level determination module 42 compares the specified operation with the operation recorded in the proficiency level database, and identifies the operation recorded in the proficiency level database that matches the specified operation.
  • the proficiency level determination module 42 specifies the proficiency level associated with this specified operation in the proficiency level database.
  • the proficiency level determination module 42 determines the specified proficiency level as the proficiency level of the maintenance worker.
  • the support information determination module 43 determines the support information for the maintenance worker to maintain the vehicle based on the specified vehicle and the determined proficiency level (step S14).
  • the support information includes information on operations necessary for the maintenance worker to perform maintenance work, such as instructions on how to use tools, instructions on body position, and instructions on work contents, as described above. Is.
  • the support information determination module 43 determines the support information by referring to the support information database recorded in advance by associating the vehicle identifier and the proficiency level of the maintenance worker with the support information. This support information database is recorded in the recording module 30. The support information determination module 43 compares the identified vehicle identifier and determined proficiency level with the vehicle identifier and proficiency level recorded in the support information database, and matches the identified vehicle identifier and determined proficiency level. Identify the vehicle identifier and proficiency recorded in the support information database. The support information determination module 43 identifies the support information associated with the identified vehicle identifier and proficiency level. The support information determination module 43 determines the specified support information as support information for the maintenance worker to maintain the vehicle.
  • the support information determination module 43 supports support information associated with the vehicle identifier “0001” and the proficiency level “3” in the support information database. To identify. The support information determination module 43 determines the specified support information as support information for the maintenance worker to maintain the vehicle.
  • the output module 21 causes the display unit of the maintenance worker to output the determined support information as augmented reality so as to overlap the vehicle (step S15).
  • the output module 21 causes the maintenance worker terminal to output the determined support information.
  • the output module 21 transmits this support information to the maintenance worker terminal.
  • the maintenance worker terminal receives this support information.
  • the maintenance worker terminal displays the received support information on its own display as augmented reality. At this time, the maintenance worker terminal displays this support information as augmented reality so as to overlap the vehicle displayed on its own display unit. The maintenance worker terminal displays this augmented reality so as to overlap a part or all of the vehicle or a part or all of the part to be maintained.
  • the computer 10 outputs this support information to the maintenance worker terminal.
  • the maintenance worker terminal will output this support information to its own display unit. Therefore, the maintenance support augmented reality output system 1 outputs the determined support information to the display unit of the maintenance worker terminal as augmented reality so as to overlap the vehicle.
  • FIG. 6 is a diagram schematically showing an example in which the output module 21 outputs the support information to the maintenance worker terminal as augmented reality.
  • the maintenance worker terminal displays the vehicle 110 on its own display unit 100.
  • the maintenance worker terminal displays the support information 120 as augmented reality at a position overlapping the vehicle 110.
  • the maintenance worker terminal displays the support information 120 as augmented reality by superimposing it on the vehicle 110, particularly the portion to be maintained.
  • the output module 21 causes the support information 120 to be displayed at a position overlapping the vehicle 110 displayed by the display unit of the maintenance worker terminal.
  • the above is the maintenance support augmented reality output processing.
  • FIG. 4 is a diagram showing a flowchart of the proficiency evaluation process executed by the computer 10. The process executed by each of the above-mentioned modules will be described together with this process. The detailed description of the same processing as the above-mentioned processing will be omitted.
  • the second maintenance work image acquisition module 22 acquires the second maintenance work image obtained by photographing the scenery of the maintenance work of the maintenance worker after the output module 21 outputs the support information (step S20).
  • the second maintenance work image acquired by the second maintenance work image acquisition module 22 shows the scenery of the maintenance work of the maintenance worker in the state after the output module 21 outputs the support information to the maintenance worker terminal. It was taken.
  • Substantial processing in step S20 is the same as the processing in step S10 described above.
  • the analysis module 40 analyzes the acquired second maintenance work image (step S21).
  • the process of step S21 is the same as the process of step S11 described above. However, in this process, the analysis module 40 analyzes only maintenance workers.
  • the proficiency level evaluation module 44 evaluates the proficiency level of this maintenance worker based on the analysis result (step S22).
  • the evaluation of the proficiency level of the maintenance worker is a quantification of the proficiency level of the maintenance worker based on the number of times the maintenance worker has achieved an operation or work according to the support information.
  • the proficiency level evaluation module 44 identifies the operation of the maintenance worker reflected in the second maintenance worker based on the feature points and feature quantities extracted from the second maintenance work image.
  • the proficiency level evaluation module 44 identifies the operation of the maintenance worker based on the changes in the feature points and feature quantities. For example, the proficiency evaluation module 44 has the movement of the body of the maintenance worker, the movement of the tools used by the maintenance worker, and the position between the vehicle and the maintenance worker based on the changes in the feature points and the feature amounts. Identify the relationship as an action.
  • the proficiency evaluation module 44 counts the number of times the operation or work is achieved according to the support information determined this time. When the number of times exceeds the specified number of times, the proficiency evaluation module 44 records in advance an evaluation database in which the identifier of the maintenance worker and the past evaluation of the proficiency level of the maintenance worker are recorded in association with each other. Update. This evaluation database is recorded in the recording module 30.
  • the identifier of the maintenance worker is, for example, the identifier of the maintenance worker terminal, the name of the maintenance worker, the management number, and the face photograph.
  • This proficiency evaluation is, for example, divided into a predetermined number of stages, and is a numerical value when the number of times a maintenance worker has achieved an operation or work according to support information exceeds the specified number of times. is there. The higher the number of times the specified number of times is achieved, the higher the evaluation of the proficiency level, and the lower the number, the lower the value. Specifically, it is divided into stages from 1 to 10, and the more times the specified number of times is achieved, the closer to 10.
  • the proficiency evaluation module 44 uses the identifier of this maintenance worker in the evaluation database. Identify the proficiency assessment of this associated maintenance worker. The proficiency evaluation module 44 raises this identified proficiency evaluation by one level.
  • the proficiency evaluation module 44 evaluates the specified proficiency level when the number of times the operation or work according to the support information determined this time is achieved based on the specified operation does not exceed the specified number of times. Maintain without raising.
  • the proficiency level evaluation module 44 updates the evaluation database based on the evaluation raised by one level (step S23).
  • step S23 the proficiency level evaluation module 44 raises the proficiency level evaluation associated with the identifier of the maintenance worker to be targeted this time by one level in this evaluation database.
  • the proficiency evaluation module 44 records the increased proficiency evaluation in the evaluation database. As a result, the proficiency evaluation module 44 updates the evaluation database.
  • the support information determination module 43 determines the support information in consideration of the evaluation of the proficiency level. This case will be described.
  • the support information determination module 43 determines the support information for the maintenance worker to maintain this vehicle based on the specified vehicle, the determined proficiency level, and the evaluation of the proficiency level of the maintenance worker.
  • the support information determination module 43 determines support information based on the identified vehicle, the determined proficiency level, and the evaluation of the proficiency level of the maintenance worker.
  • the support information determination module 43 determines the proficiency level of the maintenance worker used for determining the support information based on the evaluation of the proficiency level of the maintenance worker and the determined proficiency level of the maintenance worker. At this time, for example, the support information determination module 43 uses the lower value or the higher value of the evaluation of the proficiency level of the maintenance worker and the determined proficiency level of the maintenance worker as the support information. Determined as the proficiency level of the maintenance worker used for the determination. Further, the support information determination module 43 performs predetermined processing such as addition and subtraction of a predetermined coefficient according to the evaluation of the proficiency level of the maintenance worker to the determined proficiency level of the maintenance worker to obtain the support information. Determined as the proficiency level of the maintenance worker used for the determination.
  • the support information determination module 43 compares the identified vehicle identifier and the proficiency level of the maintenance worker determined this time with the vehicle identifier and proficiency level recorded in the support information database, and compares the identified vehicle identifier and determination. Identify the vehicle identifier and proficiency recorded in the support information database that matches the proficiency level. The support information determination module 43 identifies the support information associated with the identified vehicle identifier and proficiency level. The support information determination module 43 determines the specified support information as support information for the maintenance worker to maintain the vehicle.
  • the support information determination module 43 has this maintenance.
  • the proficiency level of the worker is determined based on the determined proficiency level 3, which is a low value.
  • the support information determination module 43 identifies the support information associated with the vehicle identifier “0002” and the proficiency level “3” in the support information database.
  • the support information determination module 43 determines the specified support information as support information for the maintenance worker to maintain the vehicle.
  • the method of determining the support information executed by the support information determination module 43 is not limited to the above-mentioned example, and can be changed as appropriate.
  • FIG. 5 is a diagram showing a flowchart of a report automatic creation process executed by the computer 10. The process executed by each of the above-mentioned modules will be described together with this process. The detailed description of the same processing as the above-mentioned processing will be omitted.
  • the second maintenance work image acquisition module 22 acquires the second maintenance work image obtained by photographing the scenery of the maintenance work of the maintenance worker after the output module 21 outputs the support information (step S30).
  • the process of step S30 is the same as the process of step S20 described above.
  • the analysis module 40 analyzes the acquired second maintenance work image (step S31).
  • the process of step S31 is the same as the process of step S21 described above.
  • the maintenance content determination module 45 determines whether or not the operation of the maintenance worker is an operation in accordance with the support information based on the analysis result (step S32). In step S32, the maintenance content determination module 45 identifies the operation of the maintenance worker reflected in the second maintenance worker based on the feature points and the feature amounts extracted from the second maintenance work image. The maintenance content determination module 45 identifies the operation of the maintenance worker based on the changes in the feature points and the feature amounts. For example, the maintenance content determination module 45 determines the movement of the body of the maintenance worker, the movement of the tools used by the maintenance worker, and the position between the vehicle and the maintenance worker based on the changes in the feature points and the feature amounts. Identify the relationship as an action. The maintenance content determination module 45 determines whether or not the operation of the specified maintenance worker matches the determined support information.
  • step S32 when the maintenance content determination module 45 determines that the operation of the maintenance worker is an operation in accordance with the support information (step S32 YES), the report automatic creation module 46 is suitable for internal use. A report stating that this was the case is automatically created (step S33).
  • step S33 the report automatic creation module 46 automatically indicates as a report that the matters described in the general report and the work contents of the maintenance worker are appropriate based on the predetermined format. create.
  • the report automatic creation module 46 automatically creates a report by, for example, describing the date, address, title, and text (for example, the content indicating that the work content of the maintenance work was appropriate) in a predetermined format. ..
  • step S32 when the maintenance content determination module 45 determines that the operation of the maintenance worker does not conform to the support information (step S32 NO), the report automatic creation module 46 is for internal use. Automatically creates a report stating that was not appropriate (step S34).
  • step S34 the report automatic creation module 46 automatically indicates as a report that the matters described in the general report and the work contents of the maintenance worker are not appropriate based on the predetermined format. create.
  • the report automatic creation module 46 has, for example, the date, address, title, and text (for example, the content indicating that the maintenance work content was not appropriate, the maintenance work content that should be performed, the actual operation performed, and the maintenance work.
  • a report is automatically created by describing the points to be improved in the work) in a predetermined format.
  • output module 21 can also output the report created by the processes of steps S34 and S35 described above to the maintenance worker terminal and other terminals and devices.
  • the above-mentioned means and functions are realized by a computer (including a CPU, an information processing device, and various terminals) reading and executing a predetermined program.
  • the program is provided, for example, in the form of being provided from a computer via a network (Software as a Service). Further, the program is provided in a form recorded on a computer-readable recording medium such as a flexible disk, a CD (CD-ROM or the like), or a DVD (DVD-ROM, DVD-RAM or the like).
  • the computer reads the program from the recording medium, transfers it to an internal recording device or an external recording device, records the program, and executes the program. Further, the program may be recorded in advance on a recording device (recording medium) such as a magnetic disk, an optical disk, or a magneto-optical disk, and provided to the computer from the recording device via a communication line.

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Economics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Development Economics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
PCT/JP2019/016383 2019-04-16 2019-04-16 整備サポート拡張現実出力システム、コンピュータ、端末、整備サポート拡張現実出力方法及びプログラム Ceased WO2020213074A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2019/016383 WO2020213074A1 (ja) 2019-04-16 2019-04-16 整備サポート拡張現実出力システム、コンピュータ、端末、整備サポート拡張現実出力方法及びプログラム
JP2021514702A JP7189332B2 (ja) 2019-04-16 2019-04-16 整備サポート拡張現実出力システム、コンピュータ、端末、整備サポート拡張現実出力方法及びプログラム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/016383 WO2020213074A1 (ja) 2019-04-16 2019-04-16 整備サポート拡張現実出力システム、コンピュータ、端末、整備サポート拡張現実出力方法及びプログラム

Publications (1)

Publication Number Publication Date
WO2020213074A1 true WO2020213074A1 (ja) 2020-10-22

Family

ID=72838164

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/016383 Ceased WO2020213074A1 (ja) 2019-04-16 2019-04-16 整備サポート拡張現実出力システム、コンピュータ、端末、整備サポート拡張現実出力方法及びプログラム

Country Status (2)

Country Link
JP (1) JP7189332B2 (https=)
WO (1) WO2020213074A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7145553B1 (ja) 2022-06-23 2022-10-03 英明 原田 遠隔指導管理装置及び遠隔指導システム
JP2023086618A (ja) * 2021-12-10 2023-06-22 トヨタ自動車株式会社 情報処理装置、プログラム及び情報処理方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016056449A1 (ja) * 2014-10-10 2016-04-14 富士通株式会社 スキル判定プログラム、スキル判定方法、スキル判定装置およびサーバ
WO2017119127A1 (ja) * 2016-01-08 2017-07-13 三菱電機株式会社 作業支援装置、作業学習装置及び作業支援システム
JP2017220109A (ja) * 2016-06-09 2017-12-14 大和ハウス工業株式会社 情報提供システム及び情報提供方法

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002287815A (ja) * 2001-03-26 2002-10-04 Mitsubishi Heavy Ind Ltd 現場支援システム
JP6917844B2 (ja) * 2017-09-20 2021-08-11 株式会社東芝 作業支援システム、作業支援方法及び作業支援プログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016056449A1 (ja) * 2014-10-10 2016-04-14 富士通株式会社 スキル判定プログラム、スキル判定方法、スキル判定装置およびサーバ
WO2017119127A1 (ja) * 2016-01-08 2017-07-13 三菱電機株式会社 作業支援装置、作業学習装置及び作業支援システム
JP2017220109A (ja) * 2016-06-09 2017-12-14 大和ハウス工業株式会社 情報提供システム及び情報提供方法

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023086618A (ja) * 2021-12-10 2023-06-22 トヨタ自動車株式会社 情報処理装置、プログラム及び情報処理方法
JP7700659B2 (ja) 2021-12-10 2025-07-01 トヨタ自動車株式会社 情報処理装置、プログラム及び情報処理方法
JP7145553B1 (ja) 2022-06-23 2022-10-03 英明 原田 遠隔指導管理装置及び遠隔指導システム
JP2024002129A (ja) * 2022-06-23 2024-01-11 英明 原田 遠隔指導管理装置及び遠隔指導システム

Also Published As

Publication number Publication date
JPWO2020213074A1 (https=) 2020-10-22
JP7189332B2 (ja) 2022-12-13

Similar Documents

Publication Publication Date Title
KR102270499B1 (ko) 이미지 기반의 차량 손상 판정 방법과 장치, 및 전자기기
KR102322773B1 (ko) 전극편의 버를 검출하기 위한 방법 및 장치
US20190331483A1 (en) Analysis apparatus and analysis program
US20160098636A1 (en) Data processing apparatus, data processing method, and recording medium that stores computer program
CN109285079A (zh) 终端屏幕保险的数据处理方法、装置、客户端及服务器
EP3255603A1 (en) Image processing device, image processing system, and image processing method
CN105302413B (zh) 控件的ui测试方法及系统
JP6922399B2 (ja) 画像処理装置、画像処理方法及び画像処理プログラム
JP7694679B2 (ja) 画像処理装置、画像処理方法および画像処理プログラム
WO2020213074A1 (ja) 整備サポート拡張現実出力システム、コンピュータ、端末、整備サポート拡張現実出力方法及びプログラム
CN112734706A (zh) 电子装置及焊点检测方法
CN109271982A (zh) 多个识别区域识别方法、识别终端及可读存储介质
US20170078633A1 (en) User terminal and system and method for correcting color
CN114972500A (zh) 查验方法、标注方法、系统、装置、终端、设备及介质
CN114529232A (zh) 一种数字中心资产管理方法和装置
JP2021177435A (ja) 読取システム、読取方法、プログラム、記憶媒体、及び移動体
CN112686102A (zh) 一种适应于地铁站点的快速排障方法
CN114896148A (zh) 一种基于图像识别能力的终端应用自动化测试装置、系统、方法、设备以及存储介质
US11205258B2 (en) Image processing apparatus, image processing method, and storage medium
WO2021152819A1 (ja) コンピュータシステム、情報コード読取方法及びプログラム
CN113507571B (zh) 视频防剪辑方法、装置、设备、可读存储介质
US20180241905A1 (en) Image processing apparatus and non-transitory computer readable medium
JP2008009938A (ja) 動画像データ処理装置、動画像データ処理方法、動画像データ処理プログラム、およびこれを記録した記録媒体
CN114723829B (zh) 进展判定系统、进展判定方法及存储介质
CN110633976B (zh) 虚拟资源转移方法与装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19924943

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021514702

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19924943

Country of ref document: EP

Kind code of ref document: A1