JP2018067243A - Work support method, work support program and work support device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To avoid change of color information of a marker caused by environment change, and easily and efficiently determine whether a work target position matches a work target position recognized by a worker.SOLUTION: The work support method performs a step of grasping a position of an operated target 32-2 in an image of the operated target 32-2 and superimposing and displaying an outline image 35 on a moving image based on the position of the operated target 32-2, a step of taking a still image so that the operated target 32-2 and a maker 42 appear and displaying it on a display 6, when a position PM in the still image displayed on the display 6 is specified, a step of acquiring color information of the specified position PM, searching an element having the same color information in the still image, and recognizing a gravity center coordinate PC of the element having the detected same color information, and a step of determining whether both of the specified position PM and the gravity center coordinate PC are in the outline image 35.SELECTED DRAWING: Figure 5

Description

  The present disclosure relates to a work support method, a work support program, and a work support apparatus.

  In operation and maintenance work for telecommunications equipment, it is necessary to accurately determine the position of the equipment to be worked and work without error. If work is performed on different facilities, an unexpected line disconnection is caused, which causes problems such as generation of maintenance costs and reduction of corporate reliability due to the restoration work.

  In order to avoid such erroneous work, work is performed carefully and accurately by confirming the operation target by visual and pointing based on the work instruction, mutual confirmation of the operation target by multiple persons, etc. Is called. However, there are concerns about erroneous work caused by human error due to the increase in the number of single-person work in the field due to a lack of attention and prejudice when moving to work after visual confirmation, and a shortage of workers and maintenance costs.

  Therefore, there is a method of providing the work target device with an LED or the like to make the operation target emit light and clearly indicating it, but it is necessary to add an LED or the like to all the work target devices, and many work target devices are installed in the field. If this is the case, there will be problems such as the operation of the mounting operation taking place and the price of the device rising.

  Non-patent document 1 shows an example of a technical solution for solving such an existing problem. This is a figure showing the contour of an operation target based on a work order on a camera moving image photographed with a tablet using a technique called augmented reality (AR) (for example, see Non-Patent Document 2). Is displayed by the system, and the operator can visually confirm the operation target on the tablet screen without mistake. In addition, the position and shape of the display figure changes appropriately following the movement of the camera, and the contour of the operation target is continuously specified. Thereby, it becomes possible to immediately specify the operation target by visual understanding.

  In addition, after correctly specifying the operation target, the operator attaches a marker such as a seal as a mark to the operation target. The actual removal work is carried out using this marker as a mark, but there is room for erroneous removal such as separation of the marker when attaching the marker or incorrect marker attachment due to assumptions.

  In order to eliminate such factors and perform the work reliably on the operation target, the system operates the operation target that has been ordered and the operation target recognized by the worker, that is, the target to which the marker is attached. There is a function to make a correct / incorrect judgment by collating the above.

  This is realized by combining AR technology and color recognition technology. The color information held by the marker is registered in advance in the system and becomes a reference value for color recognition. Based on this reference value, the system detects the same color location in the image and assumes that the barycentric coordinate is the marker position, and the position of the operation target based on this marker position and work order (this can be recognized by AR technology). ) Is checked to see if they match.

  However, in this method, there is a problem that the detection rate is remarkably lowered when the color information of the marker changes depending on the brightness of the illumination and the illumination type, and the marker is detected with a reference value for color recognition registered in advance. Therefore, in order to avoid the influence of fluctuations in the environment typified by the lighting state, the color information is not stored in the system in advance, but the marker attached to the operation target is photographed with the tablet camera. Thus, the color information of the marker is measured in the field, and the result is reflected as the reference value of the color information registered in the system.

  However, even with this method, the positional relationship between the lighting, the marker, and the tablet changes due to the slight movement of the tablet and the time difference that occurs between the time when the marker is captured by the camera and the color information is captured and the correctness is determined. Therefore, the color information of the marker may change, and it may be difficult to detect the marker attached to the operation target.

  Further, as described in Non-Patent Document 3, in order to capture color information, it is necessary to perform color measurement in a state where the marker is placed in a frame displayed on the moving image of the camera, and the marker is placed in the frame. There is a problem that it is a process that requires a certain level of proficiency for efficient use, such as taking time and maintaining a long working time to maintain the stored state.

  Also, to determine the center of gravity coordinates as the marker position, if the operator accidentally puts markers at multiple locations, or if there is a part with the same color information as the marker color at another point, combine them. The system recognizes the center of gravity coordinates as the marker position. Here, as shown in FIG. 1, when the barycentric coordinates of the part having the same color information as the marker 42 accidentally falls within the operation target 32-2, which is the operation target, the pasting position of the marker 42 is Even though the operation target 32-1, which is not the target, is erroneous, the determination is OK, which may lead to erroneous work.

Furoshikiya et al., "In-house transmission device maintenance work support system using AR technology", IEICE Technical Report, OFT 115, 29-34, October 2015. Information Processing Society of Japan CVIM Study Group [Image Recognition / Understanding Symposium (MIRU2011)]: Shimamura, Morimoto, Koike “Registration to Object Coordinate System Using Partial Template Detection and vSLAM for AR for Facility Maintenance Support”, IS4 -13, July 2011 Enomoto et al., "Evaluation of workability of in-house transmission equipment maintenance work support system using AR", IEICE Society Conference, B-10-9, September 2016

  Therefore, the present disclosure aims to avoid a change in marker color information due to an environmental change, and to easily and efficiently determine whether a work target location and a work target location recognized by an operator match. And

Specifically, the work support method of the present disclosure is:
Executed by a work support device for determining whether a target to be operated based on a work order matches a target to be operated recognized by a worker in a work target having a group of operation targets in which a plurality of operation targets are arranged. A work support method for
The image of the operation target is captured using a camera, the position of the operation target in the image is grasped, and the figure representing the contour of the operation target based on the grasped position of the operation target A contour image display step to be superimposed on the image;
A still image capturing step of capturing a still image using a camera so that a marker attached to one of the operation objects is captured, and displaying the still image on a display;
When the position in the still image displayed on the display is designated, the color information at the designated position is fetched as the marker color information, and an element having the same color information as the marker color information is captured in the still image. A color information recognition step for searching and recognizing a barycentric coordinate of an element having the same color information detected by the search as a marker pasting position;
If the specified position and the barycentric coordinates are both present in the contour, and if both the specified position and the barycentric coordinates exist in the contour, the work order and the work A work position matching step for determining that the operation target recognized by the person matches,
It is characterized by performing.

  In the work support method of the present disclosure, in the color information recognition step, adjacent elements among the elements having the same color information are regarded as one set, and an outline for each set may be displayed on the display. Good.

  In the work support method of the present disclosure, in the color information recognition step, when the area of the set is equal to or less than a predetermined threshold value, it may be excluded from the elements having the same color information detected by the search. .

  In the work support method according to the present disclosure, in the color information recognition step, when there is a set having a predetermined threshold value or more in the set, the specified position and the barycentric coordinates are set as the operation target. Even if both exist in the contour, it may be determined that the operation order and the operation target recognized by the worker do not match.

  The display may be a touch panel display, and in the color information recognition step, a position in the still image displayed on the display may be designated by acquiring a position touched on the display.

  The work support program according to the present disclosure is a program for causing a computer to execute each step provided in the work support method according to the present disclosure, and for realizing the computer as each functional unit provided in the work support device according to the present disclosure. It is a program. The work support program according to the present disclosure may be recorded on a computer-readable recording medium.

Specifically, the work support device of the present disclosure is:
In a work target having an operation target group in which a plurality of operation targets are arranged, a work support device that determines whether or not the operation target based on the work order matches the operation target recognized by the worker,
The image of the operation target is captured using a camera, the position of the operation target in the image is grasped, and the figure representing the contour of the operation target based on the grasped position of the operation target A contour image display section to be superimposed on the image;
A still image capturing unit that captures a still image using a camera so that a marker attached to one of the operation objects is captured, and displays the image on a display;
When the position in the still image displayed on the display is designated, the color information at the designated position is fetched as the marker color information, and an element having the same color information as the marker color information is captured in the still image. A color information recognition unit that searches and recognizes the barycentric coordinates of the elements having the same color information detected by the search as marker pasting positions;
If the specified position and the barycentric coordinates are both present in the contour, and if both the specified position and the barycentric coordinates exist in the contour, the work order and the work A work position verification unit that determines that the operation target recognized by the person matches,
Is provided.

  The above disclosures can be combined as much as possible.

  According to the present disclosure, it is possible to avoid a change in the color information of the marker due to an environmental change, and to perform a correct / incorrect determination of whether the work target location and the work target location recognized by the worker coincide with each other easily and efficiently.

It is explanatory drawing for demonstrating the subject in this indication. It is an example of the schematic diagram of the work assistance apparatus and work object in this embodiment. It is an example of the work target and marker in this embodiment. It is an example of a structure of the work assistance apparatus in this embodiment. It is an example of the flowchart of the insertion / extraction operation | work using the operation | work assistance system in this embodiment. It is an example of the display of the outline figure by the work assistance apparatus and AR in this embodiment. It is another example of the display of the outline figure by the work assistance apparatus and AR in this embodiment. It is an example of the display of the image pattern by the work assistance apparatus in this embodiment.

  Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In addition, this indication is not limited to embodiment shown below. These embodiments are merely examples, and the present disclosure can be implemented in various modifications and improvements based on the knowledge of those skilled in the art. In the present specification and drawings, the same reference numerals denote the same components.

  Hereinafter, a work support apparatus and system according to an embodiment of the present disclosure will be described. FIG. 2 shows an example of a work support apparatus according to the embodiment. The work support device 9 according to the embodiment is a device for supporting work on the work target 30, and includes an input unit 1, an information processing unit 2, a camera 5, a display 6, and a storage unit 7. The work support device 9 is a portable computer such as a tablet having a camera and a touch panel display. In the case of a touch panel display, the display 6 functions as the input unit 1.

  FIG. 3 shows an example of work objects and markers in the present embodiment. The work target 30 is equipment to be worked, for example, telecommunication equipment such as a router and a transmission device. The operation target group 31 is a group of operation targets 32 for performing an insertion / extraction operation such as a mounted package, a connected connector, and a cable. The operation target is one operation target in the operation target group 31 instructed by the work order to execute the insertion / extraction operation. In the present embodiment, an example in which the operation target is the operation target 32-2 will be described as an example.

  The marker 42 is placed on an operation target to be inserted / extracted by the worker, and serves as a mark indicating the operation target recognized by the worker. The marker 42 has a specific color and is attached to the operation target 32 by an adhesive member such as a seal or a tape or a magnetic force. Further, the size of the marker 42 may be smaller than the package size or connector size that is mounted without overlapping the other packages or connectors. Also, the color of the marker 42 is good because the color close to the operation target group 31 is not used, and the uniform inside of the marker 42 can be taken widely when obtaining the color region.

  FIG. 4 is a diagram illustrating an example of the configuration of the information processing unit 2. The information processing unit 2 includes a contour image display unit 22, a still image imaging unit 23, a color information recognition unit 25, a barycentric coordinate calculation unit 26, and a work position collation unit 28.

  FIG. 5 shows an example of an operation flowchart of the insertion / extraction work using the work support method according to the present embodiment. In the work support method according to the present embodiment, the work support device 9 sequentially executes a contour image display step, a still image capturing step, a color information recognition step, and a work position collation step. Steps S101 to S103 are executed in the contour image display step, steps S104 to S105 are executed in the still image capturing step, steps S106 to S108 are executed in the color information recognition step, and steps S109 to S110 are executed in the work position matching step. To do.

Step S101: Work order information is input to the input unit 1 of the work support apparatus 9. The work order information includes the position information of the work target 30 and the operation target for selection, and the size information of the marker 42.
Step S <b> 102: The contour image display unit 22 shoots the work target 30 with the camera 5 when acquiring the work order information. Shooting may be either moving image shooting or still image shooting.

  Step S103: The contour image display unit 22 uses the camera 5 to display the contour image 35 indicating the contour of the operation target 32-2 in accordance with the position of the operation target 32-2 instructed by the work order information by the AR function. It is displayed superimposed on the captured image.

  As shown in FIG. 6, the work object 30 is photographed by the camera 5, and a moving image or a still image is displayed on the display 6. By incorporating the AR function, the contour image display unit 22 can grasp the position of the contour of the operation target 32-2 in the moving image or still image, and can display the contour image 35 superimposed on the moving image or still image. .

  Step S104: The worker refers to the position of the contour image 35 displayed in a superimposed manner, and pastes the marker 42 on the operation target 32-2 recognized by the worker. The marker 42 is an arbitrary color, shape, pattern, or a combination thereof that can be distinguished from characters and figures written on the work object 30 and is, for example, a square-shaped red seal.

  Step S105: Thereafter, the input unit 1 shifts the work support apparatus 9 to the color information correction mode. In the still image mode, the still image capturing unit 23 causes the camera 5 to capture a still image so that the pasted marker 42 is included. FIG. 7 shows an example of a still image. The still image includes an operation object 32-2 and a marker 42.

  Step S106: When the designated position PM on the still image displayed on the display 6 is designated from the input unit 1, the color information recognition unit 25 measures the color information of the touched element by the designated position PM. Remember. For example, when the display 6 is a touch panel display, the designated position PM is, for example, a position tapped on the touch panel.

  The designation method of the designated position PM may be realized by moving the cursor displayed on the display with a mouse or the like and clicking on the marker position. The designated position PM may be designated by an operator wearing the marker 42. Further, by sharing a still image with a worker at a remote location via a network and executing it at the remote location side, a cross check may be performed to further increase safety.

  Step S107: The color information obtained in step S106 is displayed on the display 6 as a color sample, and the operator can confirm that the obtained color information and the marker 42 are the same color. For this reason, when an incorrect position is designated, the operator can recognize that the wrong position has been designated and redo step S106.

  Here, the color information recognition unit 25 preferably treats adjacent elements among elements having the same color information as one set. In this case, it is preferable to display the outline for each set on the display 6.

  Step S108: Next, the barycentric coordinate calculation unit 26 calculates the barycentric coordinates PC of the pixels in the still image that match the obtained color information and the set thereof. The display 6 displays the calculated barycentric coordinates PC on a still image. The image pattern is an image having an arbitrary shape. FIG. 8 shows an example of the display of the barycentric coordinates PC. The barycentric coordinates PC are preferably displayed in a predetermined image pattern, such as an “x” shape shown in FIG.

  When the barycentric coordinate calculation unit 26 calculates, the size of the marker 42 included in the work order information may be taken into consideration. For example, when the area of the set of pixels matching the obtained color information is small compared to the size of the marker 42 included in the work order information, for example, when it is equal to or less than a threshold value, it is regarded as noise and is calculated from the barycentric coordinates PC. May be excluded.

  Here, if the area of the set of pixels is captured in a state of facing the marker 42, the number of pixels included in the set of pixels and the captured image and real space possessed by the AR function of the barycentric coordinate calculation unit 26 will be described. It can be determined from the scale information to associate the length of. However, in the case where the marker 42 is not directly facing, if the area is obtained only from the number of pixels, the area of the marker 42 is not correct. Therefore, the center-of-gravity coordinate calculation unit 26 uses the shooting angle information of the camera that is grasped by the AR function to convert and correct the number of pixels to the number of pixels when facing the front, and then uses the scale information to calculate the area. After conversion, the area may be compared with the size information of the marker 42 in the work order information. For the correction calculation, the marker 42 is preferably pasted on the same plane as the front surface of the operation target 32-2.

  The size of the marker 42 is preferably large, but if it is too large, it may overlap other packages and connectors. Therefore, it is preferable that the size of the marker 42 is slightly smaller than the package size or connector size. Therefore, the size range of the marker 42 can be determined.

  Step S109: The input unit 1 shifts the work support device 9 to the work position collation mode. In the work position collation mode, the work position collation unit 28 determines whether or not both the designated position PM and the barycentric coordinate PC exist in the contour image 35 of the operation target 32-2 indicated by the work order information. To do. Thereby, it can be determined whether the work order and the operation target 32-2 recognized by the worker match.

  Only when both the designated position PM and the barycentric coordinate PC exist in the contour image 35, the determination result is OK. When either the designated position PM or the barycentric coordinates PC exists outside the contour image 35, the determination result is NG. If the determination result is NG, the process is repeated from step S104 or step S105.

  At this time, if the area of the set of pixels matching the color information obtained in step S106 is larger than the size of the marker 42 included in the work order information, the marker 42 is present even if both exist. The determination result in step S109 may be NG assuming that there is a possibility that a part other than is specified by touch. At this time, the outline of the color region used for the calculation of the barycentric coordinate PC may be displayed on the screen of the display 6. If the size of the marker 42 is too large or too small, the operator can confirm the reason for NG, such as a marker 42 of a similar color in another region.

  Step S110: When the determination result is OK, the determination result is confirmed, and the determination result and the image are stored in the storage unit 7. The storage unit 7 is, for example, a hard disk or an external storage medium provided in the work support device 9.

  Step S111: Finally, the operation target 32-2 to which the marker 42 is pasted is inserted and removed, and the operation is finished.

  By realizing this disclosure, the color information of the marker 42 is captured from the still image obtained by the camera 5 and the collation determination is performed using the still image. Therefore, the color information of the marker 42 is obtained between the acquisition of the color information and the collation determination. The marker 42 can be detected with high accuracy.

  In addition, since the color information of the marker 42 can be acquired by touching the display 6, it is not necessary to move the work support device 9 well to match the color measurement frame on the camera image with the marker 42, and the work time can be shortened. Therefore, it is possible to perform simple and efficient work without learning the operation.

  Further, the present disclosure also discloses a case in which the position of the center of gravity accidentally falls within the operation target due to the presence of the same color part different from the marker 42 even though the marker pasting position is wrong, and the determination is erroneously determined as OK. Is used to check whether the position of the center of gravity is within the contour image 35 of the operation target 32-2, as well as whether the touch position is within the contour image 35 of the operation target 32-2. For this reason, the determination result of whether the operation order and the operation target recognized by the worker are the same can be realized by correct determination.

  In addition, even if a color similar to the marker 42 (for example, LED) is used on the work object 32, if the area is equal to or less than the specified threshold value, it is excluded from the calculation of the barycentric coordinates PC. The position of the marker can be correctly recognized without affecting the recognition of the position of the center of gravity of the marker as noise.

  Note that the work support method of the present disclosure may be realized by causing a computer to execute each step. In this case, each structure is implement | achieved when the information processing part 2 with which the work assistance apparatus 9 is provided executes the computer program memorize | stored in the information storage part (not shown). Here, the computer for realizing the work support method may further include an arbitrary device controlled by the computer. In addition, a program for realizing the work support device 9 can be recorded on a recording medium or provided through a network.

  The present disclosure can be applied to the telecommunications industry.

1: input unit 2: information processing unit 22: contour image display unit 23: still image imaging unit 25: color information recognition unit 26: barycentric coordinate calculation unit 28: work position collation unit 30: work target 31: operation target group 32- 1, 32-2, 32-3: Operation target 35: Outline image 42: Marker 5: Camera 6: Display 7: Storage unit 9: Work support device

Claims (7)

Executed by a work support device for determining whether a target to be operated based on a work order matches a target to be operated recognized by a worker in a work target having a group of operation targets in which a plurality of operation targets are arranged. A work support method for
The image of the operation target is captured using a camera, the position of the operation target in the image is grasped, and the figure representing the contour of the operation target based on the grasped position of the operation target A contour image display step to be superimposed on the image;
A still image capturing step of capturing a still image using a camera so that a marker attached to one of the operation objects is captured, and displaying the still image on a display;
When the position in the still image displayed on the display is designated, the color information at the designated position is fetched as the marker color information, and an element having the same color information as the marker color information is captured in the still image. A color information recognition step for searching and recognizing a barycentric coordinate of an element having the same color information detected by the search as a marker pasting position;
If the specified position and the barycentric coordinates are both present in the contour, and if both the specified position and the barycentric coordinates exist in the contour, the work order and the work A work position matching step for determining that the operation target recognized by the person matches,
A work support method characterized by executing In the color information recognition step, adjacent elements among the elements having the same color information are regarded as one set, and an outline for each set is displayed on the display.
The work support method according to claim 1, wherein: In the color information recognition step, when the area of the set is equal to or less than a predetermined threshold value, it is excluded from the elements having the same color information detected by the search.
The work support method according to claim 2, wherein: In the work position matching step, when there is a set having an area larger than the marker in the set, the designated position and the barycentric coordinates are both present in the contour of the operation target. It is determined that the operation target and the operation target recognized by the worker do not match.
The work support method according to any one of claims 1 to 3, wherein the work support method is provided. The display is a touch panel display,
In the color information recognition step, a position in the still image displayed on the display is designated by acquiring a position touched on the display.
The work support method according to claim 1, wherein:   A work support program for causing a computer to execute each step according to claim 1. In a work target having an operation target group in which a plurality of operation targets are arranged, a work support device that determines whether or not the operation target based on the work order matches the operation target recognized by the worker,
The image of the operation target is captured using a camera, the position of the operation target in the image is grasped, and the figure representing the contour of the operation target based on the grasped position of the operation target A contour image display section to be superimposed on the image;
A still image capturing unit that captures a still image using a camera so that a marker attached to one of the operation objects is captured, and displays the image on a display;
When the position in the still image displayed on the display is designated, the color information at the designated position is fetched as the marker color information, and an element having the same color information as the marker color information is captured in the still image. A color information recognition unit that searches and recognizes the barycentric coordinates of the elements having the same color information detected by the search as marker pasting positions;
If the specified position and the barycentric coordinates are both present in the contour, and if both the specified position and the barycentric coordinates exist in the contour, the work order and the work A work position verification unit that determines that the operation target recognized by the person matches,
A work support apparatus comprising:

Images