JP2018207271A - Terminal, image processing system, image processing program, and method for processing image - Google Patents

Abstract

To allow an appropriate specification of an image region to which predetermined processing is to be performed.SOLUTION: A storage unit 1a stores input images P1 and P2 taken by an imaging device 2. A detection unit 1b detects a marker 9 and a move of the marker 9 from the input images P1 and P2 stored in the storage unit 1a. A calculation unit 1c specifies an image region according to a move of the marker 9 from the input image P2. An image processing unit 1d performs first processing to the specified image region and performs second processing, which is different from the first processing, to an image region which is not the specified image region of the input image P2.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a terminal, an image processing system, an image processing program, and an image processing method.

  A technique for performing image processing on an image captured by an imaging apparatus is used. For example, there is a proposal for protecting privacy without impairing the monitoring function of the monitoring camera. In this proposal, the field of view taken by a surveillance camera that can rotate in one direction is divided into a vertical direction and a horizontal direction, and the overlapping state of the pre-registered specific field area and the field of view is determined using the rotation angle of the camera. Determine and mask the polymerized portion.

  In addition, the positional relationship between the marker coordinates and the image processing range is defined in advance by definition data, and when a marker is identified in the image captured by the camera, it is indicated by the marker based on the definition data. There is also a proposal of a video processing system that cancels the mask of the video range to be displayed.

JP-A-6-181539 JP 2010-11017 A

  If the marker can be attached to or detached from an arbitrary portion of the subject, the relative positional relationship between the marker and the area to be subjected to the predetermined processing can be changed every time it is attached. For this reason, there is a problem that it is not easy to previously define the positional relationship between the marker coordinates and the range to be subjected to the predetermined processing as described above.

  In one aspect, an object of the present invention is to make it possible to appropriately specify an image area to be subjected to predetermined processing.

  In one aspect, a terminal is provided. The terminal includes a detection unit, a calculation unit, and an image processing unit. The detection unit detects the marker and the movement of the marker from the first image and the second image. The calculation unit specifies a first image area corresponding to the movement of the marker from the second image. The image processing unit performs a first process on the first image area, and performs a second process different from the first process on the second image area of the second image.

  In one aspect, it is possible to appropriately specify an image area to be subjected to predetermined processing.

It is a figure which shows the terminal of 1st Embodiment. It is a figure which shows the example of the information processing system of 2nd Embodiment. It is a figure which shows the example of a ring marker. It is a figure which shows the hardware example of a smart terminal. It is a figure which shows the function example of a smart terminal. It is a figure which shows the example of a frame image. It is a figure which shows the example of a color data table. It is a figure which shows the example of the last frame table. It is a figure which shows the example of the newest frame table. It is a figure which shows the example of the last flame | frame AR detection information. It is a figure which shows the example of a mask information table. It is a flowchart which shows the process example of a smart terminal. It is a flowchart which shows the example of AR mask production | generation processing. It is a flowchart which shows the example of the newest frame table creation process. It is a flowchart which shows the example of a ring marker detection process. It is a flowchart which shows the example of the mask production | generation process of a manual operation image. It is a flowchart which shows the example of the comparison process of color data. It is a figure which shows the example of rotation of a ring marker image. It is a figure which shows the comparative example of color data. It is a figure which shows the example of determination of a manual operation image area | region. It is a figure which shows the example of a display of a frame image. It is a figure which shows the example of a display of a frame image (continuation). It is a figure which shows the example of the production | generation of the mask image with respect to an apparatus image. It is a figure which shows the production | generation example (continuation) of the mask image with respect to an apparatus image. It is a figure which shows the example of a synthesis | combination of AR mask. It is a figure which shows the example of a display of a support terminal.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating a terminal according to the first embodiment. The terminal 1 is connected to the imaging device 2 and the output device 3. The imaging device 2 is a camera that captures an area including the objects 4 and 5. The output device 3 is a display that displays an image output from the terminal 1. The output device 3 may be a network interface that transmits an image output from the terminal 1 to another device. The terminal 1 may incorporate the imaging device 2 and the output device 3.

  The objects 4 and 5 are information devices such as server computers, switches, and storage devices. However, the objects 4 and 5 may be other than information equipment. The objects 4 and 5 are installed adjacent to each other. A marker 6 is attached to the front surface of the object 5 (the surface that can be imaged by the imaging device 2). The marker 6 is a predetermined pattern corresponding to information regarding the object 5. The pattern may be, for example, a combination of two-dimensional figures distinguished by color (white / black etc.). The marker 6 may be an AR marker used in a technique called AR (Augmented Reality), for example. In AR, for example, a part of an image being captured may be displayed with additional information (such as an additional image) according to the AR marker in the image.

  The hand 7 is an operator's hand. The operator moves the hand 7 to perform work such as attachment or removal of a part 8 (an example of an object gripped by the hand 7) with respect to the object 5. A marker 9 is attached to the hand 7. The marker 9 can be attached to or detached from the hand 7. For example, the marker 9 may be attached to the hand 7. Alternatively, the marker 9 may be provided on the surface of a ring (for example, a bracelet or a ring). In this case, the marker 9 is attached to the hand 7 by fitting the ring to the hand 7. For example, the marker 9 is a predetermined pattern (pattern) represented by a combination of two-dimensional figures having a predetermined color. The pattern of the marker 6 and the pattern of the marker 9 are different from each other. The marker 9 may be an AR marker.

  The imaging device 2 images the hand 7, the part 8, and the marker 9 that perform the work together with the objects 4 and 5 to sequentially generate input images P <b> 1 and P <b> 2, and inputs them to the terminal 1. The terminal 1 specifies an image area on which predetermined processing (for example, masking) is performed based on the images of the markers 6 and 9 included in the input images P1 and P2. For example, the terminal 1 makes it possible to visually recognize the area in front of the object 5 and the area of the hand 7 and the part 8 on the display image by the output device 3 and make other areas other than these areas invisible on the display image. It is conceivable to conceal the other area on the image.

  Here, the marker 6 is attached to a predetermined position of the object 5. The position of the marker 6 is not normally changed. In this case, it is possible to preset in the terminal 1 an area for performing predetermined processing (for example, an area in front of the object 5 or an area other than the area in front of the object 5) with respect to the position of the marker 6 in the image. .

  On the other hand, the attachment position of the marker 9 with respect to the hand 7 may be different every time work is performed. In addition, an object (for example, the part 8) held by the hand 7 for work or the like may be different for each work. For this reason, it is possible to preset in the terminal 1 an area for performing predetermined processing (for example, the area of the hand 7 and the part 8 held by the hand 7 or an area other than the area) with respect to the position of the marker 9 in the image. difficult. Therefore, the terminal 1 provides a function that allows the marker 9 to appropriately specify an area for performing a predetermined process.

  The terminal 1 includes a storage unit 1a, a detection unit 1b, a calculation unit 1c, and an image processing unit 1d. The storage unit 1a may be a volatile storage device such as a RAM (Random Access Memory) or a nonvolatile storage device such as an HDD (Hard Disk Drive) or a flash memory.

  The detection unit 1b, the calculation unit 1c, and the image processing unit 1d may be realized by a processing device (not shown in FIG. 1) included in the terminal 1. The processing device may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like. The processing device may be a processor that executes a program. The “processor” may also include a set of multiple processors (multiprocessor).

  The storage unit 1a stores input images P1 and P2 input by the imaging device 2. For example, the input images P <b> 1 and P <b> 2 are images generated by the imaging device 2 imaging an area including the objects 4 and 5 from a fixed point. The input image P2 is an image captured after the input image P1. Moreover, the memory | storage part 1a memorize | stores the information of the pattern of the markers 6 and 9 previously.

  The input image P1 includes object images P1a and P1b, marker images P1c and P1d, a hand image P1e, and a component image P1f. The object image P1a is an image of the object 4. The object image P1b is an image of the object 5. The marker image P1c is an image of the marker 6. The marker image P1d is an image of the marker 9. The hand image P1e is an image of the hand 7. The component image P1f is an image of the component 8.

  The input image P2 includes object images P2a and P2b, marker images P2c and P2d, a hand image P2e, and a component image P2f. The object image P2a is an image of the object 4. The object image P2b is an image of the object 5. The marker image P2c is an image of the marker 6. The marker image P2d is an image of the marker 9. The hand image P2e is an image of the hand 7. The component image P2f is an image of the component 8.

  The detection unit 1b detects the marker 9 and the movement of the marker 9 from the first image (for example, the input image P1) and the second image (for example, the input image P2). For example, the detection unit 1b detects the marker image P1d from the input image P1. Further, the detection unit 1b detects the marker image P2d from the input image P2. The detection unit 1b detects the marker image P1d corresponding to the marker 9 by comparing the pattern information of the marker 9 stored in the storage unit 1a with the marker image P1d. The detection unit 1b also detects the marker image P2d in the same manner.

  The marker images P1d and P2d correspond to the marker 9, and the marker images P1d and P2d may be simply referred to as “markers”. However, in the description of FIG. 1, in order to distinguish from the marker 9, “markers” in the input images P <b> 1 and P <b> 2 are referred to as marker images P <b> 1 d and P <b> 2 d.

  The detection unit 1b then moves the marker image P2d relative to the marker image P1d (movement of the marker 9) based on a change in the position of the input image P1 on the marker image P1d and the position of the input image P2 on the marker image P2d. Is equivalent).

  The calculation unit 1c specifies the first image area corresponding to the movement of the marker 9 from the second image. In other words, the calculation unit 1c sets an image area linked to the marker 9 in the input image P2 as the first image area. More specifically, the calculation unit 1c moves in the same direction (or the same within a predetermined error) as the movement of the marker image P2d relative to the marker image P1d (movement from the marker image P1d to the marker image P2d) and the amount of movement. The image area to be used is a first image area. The calculation unit 1c calculates the movement amount based on a difference (a difference in coordinates in the image) between a predetermined position in the marker image P1d and a position corresponding to the predetermined position in the marker image P2d. This amount of movement can also be said to be the amount of movement of the marker 9 between the input images P1, P2.

  Note that there may be a case where the imaging device 2 is not fixed. In this case, the calculation unit 1c may obtain the first position of the marker image P1d with respect to the marker image P1c and the second position of the marker image P2d with respect to the marker image P2c. Then, the calculation unit 1c may obtain the movement direction and the movement amount of the marker image P2d with respect to the marker image P1d from the changes in the first position and the second position. The marker images P1c and P2c are images of the marker 6 attached to the object 5. The marker 6 is stationary. For this reason, the change in the position of the marker image P2d relative to the marker image P1d can be appropriately detected by using the marker images P1c and P2c as a reference. Note that the movement of the marker image P2d with respect to the marker image P1d is not limited to parallel movement, but may be rotation or enlargement / reduction (equivalent to movement of the marker 9 in the depth direction). In addition to the parallel movement amount of the movement of the marker image P2d with respect to the marker image P1d, the calculation unit 1c may obtain both or any one of the rotation amount and the enlargement / reduction ratio as the movement amount of the movement.

  For example, the calculation unit 1c converts the position of each pixel of the input image P1 according to the moving direction and moving amount of the marker image P2d with respect to the marker image P1d. The calculation unit 1c compares the color information of the pixel after the position conversion with the color information of the pixel of the input image P2 at the same position as the position of the pixel, and the color information of both the pixels matches (or has a predetermined error). The corresponding pixel is a pixel related to the marker image P2d. The calculation unit 1c sets a set of pixels related to the marker image P2d among the pixels in the input image P2 as a first image region.

  For example, the calculation unit 1c specifies the hand image P2e and the component image P2f as a set of pixels related to the marker image P2d. In this case, the image area corresponding to the hand image P2e and the component image P2f is the first image area.

  The image processing unit 1d performs a first process on the first image area, and performs a second process different from the first process on the second image area of the second image. For example, the first process is a process of outputting the first image area as it is to the output device 3 without masking. For example, the second process is a process of masking a second image area different from the first image area and outputting the masked image to the output device 3.

  For example, the image processing unit 1d outputs, to the output device 3, an output image obtained by masking a second image area other than the first image area corresponding to the hand image P2e and the component image P2f in the input image P2. Also good. In this case, the user who sees the output image can check the hand image P2e and the component image P2f, but cannot check other background portions. Masking may be performed by processing such as mosaicing or blurring covered with a mask image.

  In another example, in addition to the marker 9, the image processing unit 1d outputs an output image P3 in which an image region (for example, part of the object image P2b) determined to be out of the mask target by the marker 6 is also out of the mask target. You may output to the output device 3. In this case, the output image P3 includes a mask image P3a, an object image P3b, marker images P3c and P3d, a hand image P3e, and a component image P3f. The mask image P3a is an image that covers an image area corresponding to the mask image P3a. The object image P3b is an image of the object 5. The marker image P3c is an image of the marker 6. The marker image P3d is an image of the marker 9. The hand image P3e is an image of the hand 7. The component image P3f is an image of the component 8. In this case, the user who has seen the output image P3 can confirm the object image P3b, the hand image P2e, and the component image P2f, but cannot confirm other portions.

  According to the terminal 1, the movement of the marker is detected from the first image and the second image, the image area corresponding to the movement of the marker is specified, and different processing is performed on the area and the other areas. Thereby, even if it does not predetermine processing areas, such as a mask, to a marker position, a processing area can be specified appropriately. The reason is as follows.

  Based on the input images P1 and P2, the terminal 1 detects the first image area moved in the same movement direction and movement amount as the movement of the marker image P2d with respect to the marker image P1d (or within the predetermined error). In this case, the detected first image region is an image region interlocked with the movement of the marker 9 and corresponds to the hand image P2e and the component image P2f. That is, by detecting the first image area, the terminal 1 detects a first image area, and among the input image P2, the component corresponding to the hand image P2e corresponding to the hand 7 with the marker 9 and the component 8 gripped by the hand 7. The part of the image P2f can be specified. Therefore, the terminal 1 can perform a process in which the first image area corresponding to the hand image P2e and the component image P2f and the second image area other than the first image area in the input image P2 are appropriately distinguished. .

  As an example of processing, as described above, the terminal 1 may output an output image obtained by masking an image area other than the hand image P2e and the component image P2f in the input image P2. In this case, the image area other than the hand image and the component image in the output image can be concealed. Therefore, it is possible to check the manual operation of the operator from the output image and make it impossible to check the objects 4 and 5.

  Alternatively, the terminal 1 may output the output image P3. In this case, the image area other than the object image P3b, the hand image P3e, and the component image P3f in the output image P3 can be concealed by the mask image P3a. Therefore, it is possible to check the operator's manual work or the work target object 5 from the output image P3, and make it impossible to check the object 4.

  Such a function is used, for example, when an operator at a site transmits an output image P3 via a network to an information processing apparatus such as a computer at a remote location in order to receive support from a supporter at a remote location. Useful. This is because the non-work object 4 or the like may be concealed from outside parties for security reasons.

  According to the terminal 1, the work status of the hand 7 and the parts 8 can be confirmed, and other areas can be masked, so that it is not necessary to show an extra portion to a remote supporter, and security can be improved. . As illustrated in the output image P <b> 3, the terminal 1 can also allow the supporter to check the work target object 5 in addition to the work situation of the hand 7 and the part 8.

Note that the terminal 1 can also be referred to as an image processing system having the above-described storage device and processing device. The image processing system may include the imaging device 2 and the output device 3.
In the following description, the function of the terminal 1 will be described more specifically by exemplifying the information processing system used for remote support by the supporter for the worker.

[Second Embodiment]
FIG. 2 is a diagram illustrating an example of an information processing system according to the second embodiment. The information processing system according to the second embodiment is a system that supports maintenance of a plurality of information devices (for example, the information device 10) installed in a facility such as a data center.

  The information processing system according to the second embodiment includes a smart terminal 100, a streaming server 200, and a support terminal 300. The streaming server 200 and the support terminal 300 are connected to the network 20. The network 20 may be, for example, a LAN (Local Area Network) laid in an organization or a wide area network such as the Internet. An access point 21 is connected to the network 20. The access point 21 is a wireless communication relay device. The smart terminal 100 is connected to the network 20 via the access point 21.

  The information processing system according to the second embodiment is used by a worker U1 who performs work at a site where the information device 10 is installed, and a supporter U2 who supports the worker U1 from a remote location. The worker U1 can receive instructions regarding work from the supporter U2. It can be said that the worker U1 and the supporter U2 are users of the information processing system according to the second embodiment.

  The information device 10 includes a server computer, a switch, a storage device, and the like. The information device 10 is provided with AR markers 11 and 12. The AR markers 11 and 12 are signs used to give a predetermined additional image to an image obtained by imaging the information device 10. In the second embodiment, as an example, the shape of the AR marker is a square, but other shapes may be used.

  The smart terminal 100 is a client computer used by the worker U1. The smart terminal 100 includes a camera and a display. The area including the AR markers 11 and 12 is imaged by the camera, and the work support image corresponding to the AR markers 11 and 12 (support image) is superimposed on the captured images. ) Can be displayed on the display. The support image is, for example, an image for instructing the work location to the worker U1, a message for instructing the worker U1 about the work content, or the like. Thereby, the smart terminal 100 supports the work by the worker U1. As described above, a technique for displaying an additional image superimposed on an original image in real time and providing additional information with respect to real information is sometimes called augmented reality (AR). The smart terminal 100 can also transmit the captured image to the streaming server 200. The smart terminal 100 is an example of the terminal 1 according to the first embodiment.

The streaming server 200 is a server computer that distributes the image received from the smart terminal 100 to the support terminal 300.
The support terminal 300 is a client computer that receives an image distributed by the streaming server 200 and provides it to the supporter U2. The supporter U2 can share the image captured by the smart terminal 100 with the worker U1 by browsing the image displayed by the support terminal 300. The supporter U2 can transmit a work instruction to the worker U1 by using means such as telephone or chat while confirming the image provided by the support terminal 300.

  Here, when a maintenance company that performs maintenance of information equipment performs work at a customer's facility, the worker U1 may receive logistic support from the supporter U2. In this case, for security reasons, it is conceivable that information devices other than the information device to be worked cannot be visually recognized by the supporter U2. On the other hand, if the supporter U2 can confirm the work status of the manual operation by the worker U1 (for example, the order of the manual operation procedure of the operator with respect to the manual procedure), the appropriate work such as the indication of erroneous operation and the work procedure is appropriate. The instruction can be transmitted to the worker U1. When all the areas other than the information equipment are not visible, if manual work such as operation of the maintenance terminal or cable insertion / extraction by the worker U1 is performed in the area other than the information equipment, the manual work by the worker U1 is also not visible. Thus, the supporter U2 cannot confirm the manual operation of the worker U1. This becomes more prominent as the work target information device is smaller.

  Therefore, the worker U1 performs work by attaching the ring 30 with the predetermined AR marker attached to the hand. In the following description, the AR marker attached to the ring 30 is particularly referred to as a ring marker. Then, the smart terminal 100 identifies the AR markers 11 and 12 attached to the information device 10 and the ring marker attached to the ring 30, and among the captured images, the device to be worked and the image region of manual work. Provides a function to output the data other than masked.

  FIG. 3 is a diagram illustrating an example of a ring marker. The ring 30 is a bracelet attached to the arm (for example, wrist) of the worker U1. The ring 30 has a ring marker 31. The ring marker 31 is an AR marker having a predetermined color and a predetermined graphic pattern (pattern). For example, the graphic pattern may be one or more two-dimensional barcodes expressed in a predetermined color. In the example of FIG. 3, four two-dimensional barcodes are attached to the outer peripheral edge of the ring 30. In this case, the set of the four two-dimensional barcodes can be considered as the ring marker 31. A plurality of ring markers 31 may be provided on the outer peripheral edge of the ring 30 (for example, so as to cover the entire outer periphery of the ring 30).

  FIG. 4 is a diagram illustrating a hardware example of the smart terminal. The smart terminal 100 includes a processor 101, a RAM 102, a flash memory 103, a front camera 104, a rear camera 105, a display 106, a speaker 107, a touch panel 108, a microphone 109, a medium reader 110, and a communication interface 111. Each unit is connected to the bus of the smart terminal 100. The streaming server 200 and the support terminal 300 can also be realized using the same unit as the smart terminal 100.

  The processor 101 controls information processing of the smart terminal 100. The processor 101 may be a multiprocessor. The processor 101 is, for example, a CPU, DSP, ASIC, or FPGA. The processor 101 may be a combination of two or more elements of CPU, DSP, ASIC, FPGA, and the like.

  The RAM 102 is a main storage device of the smart terminal 100. The RAM 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 stores various data used for processing by the processor 101.

  The flash memory 103 is an auxiliary storage device of the smart terminal 100. The flash memory 103 electrically writes data to and reads data from a built-in storage element. The flash memory 103 stores an OS program, application programs, and various data.

The front camera 104 is an imaging device provided on the same surface (front) as the display 106 of the smart terminal 100.
The rear camera 105 is an imaging device provided on the surface (back surface) opposite to the display 106 of the smart terminal 100.

In addition to the front camera 104 and the rear camera 105, the smart terminal 100 may be connected to a camera that captures the work area of the worker U1 from a fixed point.
The display 106 is a display device that displays an image in accordance with a command from the processor 101. The display 106 can display images captured by the front camera 104 and the rear camera 105 in real time. As the display 106, for example, a liquid crystal display can be used.

The speaker 107 acquires audio data from the processor 101, performs predetermined signal processing, and reproduces audio corresponding to the audio data.
The touch panel 108 is provided so as to overlap the display 106 and receives a touch operation by the worker U1. The touch panel 108 outputs information indicating the touched position to the processor 101.

The microphone 109 receives input such as voice uttered by the worker U 1, generates voice data by performing predetermined signal processing, and outputs the voice data to the processor 101.
The medium reader 110 reads data stored in the recording medium 22 and stores data in the recording medium 22. For example, a program to be executed by the smart terminal 100 can be recorded on the recording medium 22. For example, a semiconductor memory can be used as the recording medium 22. For example, the medium reader 110 stores the program and data read from the recording medium 22 in the RAM 102 or the flash memory 103 in accordance with an instruction from the processor 101.

  The communication interface 111 communicates with other devices including the streaming server 200 via the access point 21. In the example of the second embodiment, a wireless communication interface is assumed as the communication interface 111, but a wired communication interface may be used.

  Each of the front camera 104 and the rear camera 105 is an example of the imaging device 2 according to the first embodiment. Each of the display 106 and the communication interface 111 is an example of the output device 3 according to the first embodiment.

  FIG. 5 is a diagram illustrating an example of functions of a smart terminal. The smart terminal 100 includes an image storage unit 120, an AR information storage unit 130, a frame data storage unit 140, an image reception unit 151, a marker detection unit 152, an AR information management unit 153, a deformation rate calculation unit 154, a frame generation unit 155, an image It has a composition unit 156, a display control unit 157, a touch processing unit 158, a frame ratio calculation unit 159, an AR image generation unit 160, a manual operation image processing unit 161, and a communication control unit 162.

  The image storage unit 120, the AR information storage unit 130, and the frame data storage unit 140 are realized as a storage area secured in the RAM 102 or the flash memory 103. Image reception unit 151, marker detection unit 152, AR information management unit 153, deformation rate calculation unit 154, frame generation unit 155, image composition unit 156, display control unit 157, touch processing unit 158, frame ratio calculation unit 159, AR image The generation unit 160, the manual operation image processing unit 161, and the communication control unit 162 are realized by the processor 101 executing a program stored in the RAM 102. These functions may be realized by hard wired logic such as FPGA and ASIC.

  The image storage unit 120 stores an image captured by the front camera 104 or the rear camera 105. The image storage unit 120 may store an image captured by a camera externally attached to the smart terminal 100.

  The AR information storage unit 130 stores AR information. The AR information includes a work ID (IDentifier) of work performed by the worker U1, a marker ID of an AR marker read at the time of work, and device information that is a target of the work. The device information includes information such as the contour shape and size of the information device. The device information includes information on a module position (a position where the support image is arranged with respect to the information device) where the support image for the worker U1 is arranged. For example, a marker ID of an AR marker attached to the information device is assigned in advance to a task ID for identifying a certain task. The AR information storage unit 130 also stores information on the reference size of the AR marker (the ratio of the size of the AR marker to the device size in the device information from the reference size) and reference shape information in advance for each marker ID. Yes.

  The frame data storage unit 140 stores frame data for manual operation image extraction by the manual operation image processing unit 161. The frame data includes color data of each pixel of the previous frame image and the latest frame image input in time series, and a type indicating whether or not a mask target is necessary for each pixel of the current frame.

  The image receiving unit 151 receives a camera image (time-sequential frame image) output from the rear camera 105 and provides the received image to the marker detection unit 152 and the image composition unit 156. The image reception unit 151 may receive a camera image output by a camera attached to the front camera 104 or the smart terminal 100 and provide the image to the marker detection unit 152 and the image composition unit 156. Here, the worker U1 inputs the work ID of the work to be performed to the smart terminal 100 immediately before starting the maintenance work. Then, for example, the rear camera 105 starts imaging the imaging target range and starts outputting the camera image. The smart terminal 100 may accept selection of a camera that outputs a camera image by an operation input of the worker U1.

  The marker detection unit 152 detects an image (referred to as an AR marker image) of an AR marker (for example, the AR markers 11 and 12) attached to the information device from the camera image acquired from the image reception unit 151. An existing pattern recognition method can be used to detect the AR marker image. In addition, the marker detection unit 152 stores the camera image in which the AR marker is detected in the image storage unit 120.

  The AR information management unit 153 writes data to the AR information storage unit 130 and reads data from the AR information storage unit 130. The AR information management unit 153 manages data exchange between the units.

  The deformation rate calculation unit 154 compares the AR marker image included in the camera image stored in the image storage unit 120 with the reference size / shape of the AR marker image, thereby determining the AR marker image included in the camera image. The deformation rate of AR (AR marker deformation rate) is calculated. The deformation rate calculation unit 154 records the calculated AR marker deformation rate in the AR information storage unit 130 via the AR information management unit 153.

  The frame generation unit 155 reads out the work ID and device information corresponding to the detected AR marker from the AR information storage unit 130 via the AR information management unit 153, and generates a frame image that matches the shape of the work target device. The coordinates (frame image coordinates) indicating the display position of the frame image are generated. The frame image can be displayed at an arbitrary position on the display 106. For example, the frame generation unit 155 calculates the frame image coordinates so as to be displayed at the center of the display area of the display 106. The frame generation unit 155 stores the calculated frame image coordinates in the AR information storage unit 130 via the AR information management unit 153. In addition, the frame generation unit 155 provides the generated frame image to the image composition unit 156.

  The image synthesis unit 156 synthesizes the frame image generated by the frame generation unit 155 with the camera image acquired from the image reception unit 151 (superimposes the frame image on the camera image with the camera image as a background). The image composition unit 156 determines the arrangement position of the frame image based on the frame image coordinates stored in the AR information storage unit 130 (the arrangement of the frame images is fixed to the frame image coordinates).

  The image composition unit 156 adds the support image (AR image for work support) generated by the AR image generation unit 160 and the mask image generated by the manual operation image processing unit 161 to the camera image acquired from the image reception unit 151. Sometimes synthesized.

  The display control unit 157 acquires the image generated by the image composition unit 156 and outputs it to the display 106. The display 106 displays the image output by the display control unit 157.

  The touch processing unit 158 receives a touch operation on the touch panel 108 by the worker U1. For example, the worker U1 can touch a button or the like displayed on the display 106. The touch processing unit 158 receives an input corresponding to a button corresponding to the touch position. For example, the worker U1 adjusts the positional relationship between the rear camera and the work target device, and aligns the edge of the work target device image (work target device image) with the frame image displayed on the display 106. When confirming the match, the worker U1 touches a button (for example, a button with a character string such as “AR generation”) for inputting the match. Then, the touch processing unit 158 notifies the frame ratio calculation unit 159 via the AR information management unit 153 that the frame image matches the edge of the work target device image in accordance with the touch operation. .

  When receiving the notification from the touch processing unit 158, the frame ratio calculation unit 159 acquires the frame image coordinates and the AR marker coordinates and the AR marker deformation rate at the notification timing. The frame ratio calculation unit 159 displays an AR image (a work support support image or a mask image that masks other than the work target device) based on the acquired frame image coordinates, the AR marker coordinates, and the deformation rate of the AR marker. Coordinates (AR display coordinates) are calculated.

  In order to calculate the AR display coordinates, first, the frame ratio calculation unit 159 acquires the positional relationship between the AR marker image and the frame image from the acquired frame image coordinates, the AR marker coordinates, and the deformation rate of the AR marker. Then, the frame ratio calculation unit 159 obtains, from the device information, a position where the support image is arranged with respect to the work target device image (a position based on the work target device image). The frame ratio calculation unit 159 calculates the position (AR display coordinates of the support image) of the position where the support image is to be arranged based on the positional relationship between the AR marker image and the frame image, and manages AR information. The information is stored in the AR information storage unit 130 via the unit 153. Also, the frame ratio calculation unit 159 identifies an area other than the work target device image in the camera image based on the positional relationship between the marker image and the frame image, and information on the area (AR display coordinates of the mask image) Are stored in the AR information storage unit 130 via the AR information management unit 153.

  The AR image generation unit 160 generates a support image based on the AR shape information, text information, marker deformation rate, and AR display coordinates stored in the AR information storage unit 130, and provides the support image to the image composition unit 156. In addition, the AR image generation unit 160 generates a mask image for an area other than the work target area image stored in the AR information storage unit 130 and provides the mask image to the manual operation image processing unit 161. In addition, the AR image generation unit 160 acquires the camera image stored in the image storage unit 120 via the AR information management unit 153, and provides it to the manual operation image processing unit 161.

  The manual operation image processing unit 161 acquires a camera image from the AR image generation unit 160 and identifies an image of the ring marker 31 from the acquired camera image. The manual operation image processing unit 161 may acquire a camera image from the image receiving unit 151. The manual operation image processing unit 161 identifies an image region (manual operation image) in which the manual operation of the worker U1 is captured from the camera image by the image of the ring marker 31. The manual operation image processing unit 161 compares the camera image of the latest frame (latest frame image) with the camera image of the previous frame (previous frame image), and the image area moved in conjunction with the ring marker 31 Is detected. Here, the previous frame image is the frame image of the frame immediately before the latest frame image. Since the ring marker 31 is attached to the hand of the worker U1, the image area moved in conjunction with the ring marker 31 corresponds to an image area in which the hand of the worker U1 and a cable in the hand are copied. . Therefore, the manual operation image processing unit 161 identifies the detected image area as a manual operation image.

  The manual operation image processing unit 161 generates a mask image for an area other than the manual operation image and combines it with the mask image acquired from the AR image generation unit 160. Specifically, for each coordinate of both mask images, the manual operation image processing unit 161 sets the non-distribution portion to “1” (True) and the distribution portion to “0” (False), and the logical product (AND) of the coordinates. Is calculated to synthesize both mask images. As a result, the manual operation image processing unit 161 obtains a mask image that masks an area other than both the work target device image and the manual operation image. The manual operation image processing unit 161 provides a new mask image (referred to as an AR mask) obtained by combining to the image combining unit 156.

  The communication control unit 162 acquires a composite image generated by combining the camera image, the support image, and the AR mask from the image combining unit 156 by the image combining unit 156. The communication control unit 162 transmits the composite image from the communication interface 111 with the streaming server 200 as a destination.

  FIG. 6 is a diagram illustrating an example of a frame image. The frame image 51 is a frame image (an image corresponding to one frame of a moving image) captured by a camera such as the rear camera 105. The frame image 51 is generated by imaging a work area by the worker U1. The frame image 51 includes an AR marker image 51a and a ring marker image 51b. The AR marker image 51a is an image corresponding to the AR marker attached to the work target device. The ring marker image 51b is an image corresponding to the ring marker 31 of the ring 30 attached to the hand of the worker U1.

  Here, the coordinate system (XY coordinate system) of the frame image 51 is determined as follows. The origin O (0, 0) is the top left vertex of the frame image 51. The positive direction of the X axis is the direction from the origin to the right side. The positive direction of the Y axis is a direction from the origin toward the lower side. The maximum value of the X coordinate of the frame image 51 is “A-1”. The maximum value of the Y coordinate of the frame image 51 is “B−1”. In the following, it is assumed that A = 100 and B = 100 as an example.

  For example, the position of the AR marker image 51a on the frame image 51 is represented by the coordinates of the first reference position (AR marker coordinates) in the AR marker image 51a. The AR marker coordinates are, for example, one of the vertices of the AR marker image 51a or the center of gravity. Similarly, the position of the ring marker image 51b on the frame image 51 is represented by the coordinates of the second reference position (ring marker coordinates) in the ring marker image 51b. The ring marker coordinates are, for example, the center of gravity of the ring marker image 51b.

  FIG. 7 is a diagram illustrating an example of a color data table. The color data table 141 is stored in advance in the frame data storage unit 140. The color data table 141 includes items of color data and color names.

  In the color data item, a number for identifying a color (referred to as color data) is registered. The numbers indicated by the color data are positive integers such as “0, 1, 2,. In the color name item, the color name of the corresponding color data is registered. The color name is associated with actual color information (for example, color information represented by R (Red), G (Green), and B (Blue)) in the color palette.

  For example, information that color data is “0” and color name is “purple” is registered in the color data table 141. This indicates that the color data “0” corresponds to the color name “purple”. The color data “0” is a color that forms a graphic pattern in the ring marker 31.

In the color data table 141, information that the color data is “1” and the color name is “black” is registered. This indicates that the color data “1” corresponds to the color name “black”.
Here, in the color data table 141, color data is set in advance such that the smaller the difference between two color data, the higher the similarity between the colors indicated by the two color data.

  FIG. 8 is a diagram illustrating an example of the previous frame table. The previous frame table 142 is stored in the frame data storage unit 140. The previous frame table 142 includes items of coordinates (X, Y), color data, and type.

  In the item of coordinates (X, Y), the (X, Y) coordinates of the pixels in the previous frame image are registered. In the color data item, color data indicating the color of the corresponding pixel is registered. In the type item, a value indicating the type of pixel is registered. There are three types of pixels. The first type is a type indicating that the corresponding pixel is a pixel of the ring marker image. The value representing the first type is “1”. The second type is a type indicating that the pixel is a candidate for distribution target (candidate for non-mask target). The value representing the second type is “2”. The third type is a type indicating that the pixel is a candidate for non-delivery target (mask target candidate). The value representing the third type is “0”.

  For example, information that coordinates (X, Y) is “(0, 0)”, color data is “22”, and type is “0” is registered in the previous frame table 142. This is because the color data is “22” for the pixel at the coordinate (X, Y) = (0, 0) of the previous frame image, and the pixel type is “0” (candidate pixel to be masked). ).

Similarly, coordinates (X, Y), color data, and type are registered in the previous frame table 142 for other pixels of the previous frame image.
FIG. 9 is a diagram illustrating an example of the latest frame table. The latest frame table 143 is stored in the frame data storage unit 140. The latest frame table 143 includes items of coordinates (X, Y), color data, and type.

  In the item of coordinates (X, Y), the (X, Y) coordinates of the pixel in the latest frame image are registered. In the color data item, color data indicating the color of the corresponding pixel is registered. In the type item, a value indicating the type of pixel is registered. The information registered in the type item is the same as the type item in the previous frame table 142 illustrated in FIG.

  For example, information that coordinates (X, Y) is “(0, 0)”, color data is “22”, and type is “0” is registered in the latest frame table 143. This is because the color data is “22” and the type of the pixel is “0” (mask target candidate pixel) for the pixel at the coordinates (X, Y) = (0, 0) of the latest frame image. It shows that.

Similarly, coordinates (X, Y), color data, and type are registered in the latest frame table 143 for the other pixels of the latest frame image.
FIG. 10 is a diagram illustrating an example of the previous frame AR detection information. The previous frame AR detection information 144 is stored in the frame data storage unit 140. The previous frame AR detection information 144 includes items of an AR marker detection flag, an AR marker deformation rate, and an AR marker coordinate.

  The AR marker detection flag is registered in the AR marker detection flag item. The AR marker detection flag is a flag indicating whether or not an AR marker image is detected in the previous frame image. The AR marker detection flag “1” indicates that an AR marker image has been detected in the previous frame image. The AR marker detection flag “0” indicates that the AR marker image was not detected in the previous frame image. In the AR marker deformation rate item, the deformation rate with respect to the reference size of the AR marker image in the previous frame image is registered. When the AR marker detection flag is “0”, the AR marker deformation rate item is not set. The AR marker coordinate in the previous frame image is registered in the AR marker coordinate item. When the AR marker detection flag is “0”, the AR marker coordinate item is not set.

  For example, information that the AR marker detection flag is “1”, the AR marker deformation rate is “2.5”, and the AR marker coordinate is “(23, 45)” is registered in the previous frame AR detection information 144. This is because the AR marker image is detected in the previous frame image, the size of the detected AR marker image is “2.5” times the reference size, and the AR marker image on the previous frame image. Indicates that the position is “(23, 45)”.

  FIG. 11 is a diagram illustrating an example of a mask information table. The mask information table 145 is stored in the frame data storage unit 140. The mask information table 145 indicates a mask target area based on the latest mask image generated by the manual operation image processing unit 161. The mask information table 145 includes items of coordinates (X, Y), color data, and mask flags.

  In the item of coordinates (X, Y), the (X, Y) coordinates of the pixel in the latest frame image are registered. In the color data item, color data indicating the color of the corresponding pixel is registered. In the mask flag item, a mask flag indicating whether or not it is a mask target is registered. The mask flag “1” indicates that it is a mask target. The mask flag “0” indicates that the mask is not targeted.

  For example, information that coordinates (X, Y) is “(0, 0)”, color data is “22”, and mask flag is “1” is registered in the mask information table 145. This indicates that the color data of the pixel at the coordinates (X, Y) = (0, 0) of the latest frame image is “22” and is a mask target.

Similarly, coordinates (X, Y), color data, and mask flags are registered in the mask information table 145 for other pixels.
Next, a process procedure of work support by the worker U1 by the smart terminal 100 will be described.

FIG. 12 is a flowchart illustrating a processing example of the smart terminal. In the following, the process illustrated in FIG. 12 will be described in order of step number.
(S1) The smart terminal 100 receives an input of work start by the worker U1. The work start input includes a work ID for identifying the work content. The smart terminal 100 starts imaging an area including the work target device using the rear camera 105 or another camera. The image receiving unit 151 receives a captured camera image. The display control unit 157 acquires a camera image via the image reception unit 151 and the image composition unit 156 and displays the camera image on the display 106.

  (S2) The marker detection unit 152 acquires a camera image from the image reception unit 151. The marker detection unit 152 recognizes an AR marker image corresponding to the AR marker from the camera image. The marker detection unit 152 stores the acquired camera image in the image storage unit 120. When detecting the AR marker image, the marker detection unit 152 records the AR marker detection flag “True” and the AR marker coordinates in the AR information storage unit 130 via the AR information management unit 153. The marker detection unit 152 acquires an AR marker ID corresponding to the AR marker image from the AR marker image. The marker detection unit 152 records an AR marker detection flag “False” in the AR information storage unit 130 when no AR marker image is detected from the camera image. Thereafter, the marker detection unit 152 continuously sets the AR marker detection flag to “True” or “False” for the frame image of each frame.

  (S3) The frame generation unit 155 generates a frame image for the work target device. The image composition unit 156 generates a frame composite image in which the frame image generated by the frame generation unit 155 is combined with the camera image. The display control unit 157 causes the display 106 to display a frame composite image generated by image synthesis. Here, the frame generation unit 155 generates a frame image corresponding to the device information (shape, size, etc.) of the work target device and AR marker information (deformation rate of the marker image with respect to the reference shape or size). To do. The device information of the work target device and the AR marker information are stored in advance in the AR information storage unit 130 in association with the combination of the work ID and the marker ID. The display control unit 157 determines the display position of the frame image based on the frame image coordinates obtained by the frame generation unit 155.

  (S4) The display control unit 157 aligns the frame image with the outer edge of the work target device. For example, the worker U1 moves the smart terminal 100 to adjust the positional relationship between the rear camera 105 and the work target device, thereby aligning the edge of the work target device image with the frame image displayed on the display 106. it can. The display control unit 157 may align the frame image with the outer edge of the work target device in accordance with such an operation by the worker U1.

  (S5) When the frame image is aligned with the outer edge of the work target device, the touch processing unit 158 accepts an input of AR generation start by the worker U1, and confirms that the input has been accepted via the AR information management unit 153. The image generation unit 160 and the manual operation image processing unit 161 are notified. Further, the display control unit 157 stops displaying the frame image.

  (S6) The manual operation image processing unit 161 generates an AR mask in cooperation with the AR image generation unit 160. Details of the processing will be described later. The manual operation image processing unit 161 provides the generated AR mask to the image composition unit 156.

  (S7) The image composition unit 156 generates a composite image by combining the AR mask generated by the manual operation image processing unit 161 with the camera image (latest frame image) acquired from the image reception unit 151. The communication control unit 162 outputs the composite image generated by the image composition unit 156 to the communication interface 111 and transmits it to the streaming server 200. In addition, the display control unit 157 outputs the composite image generated by the image composition unit 156 to the display 106 and causes the display 106 to display the composite image.

  (S8) The touch processing unit 158 determines whether or not there is an input of work completion. If there is an input to end the work, the touch processing unit 158 ends the process. If there is no work end input, the touch processing unit 158 advances the process to step S6.

  In steps S6 to S8, the smart terminal 100 may continuously check whether the AR marker can be recognized in the camera image, and may forcibly stop the image transmission when the AR marker cannot be recognized. In addition, when the composite image cannot be generated, the communication control unit 162 does not transmit the image of the corresponding frame (does not distribute the unmasked camera image).

  FIG. 13 is a flowchart illustrating an example of the AR mask generation process. Hereinafter, the process illustrated in FIG. 13 will be described in order of step number. The procedure shown below corresponds to step S6 in FIG. 12, and is executed every frame.

  (S11) The manual operation image processing unit 161 creates the latest frame table 143 based on the latest frame image. Details of the processing will be described later. The previous frame table 142 is stored in the frame data storage unit 140.

(S12) The manual operation image processing unit 161 detects a ring marker image (ring marker image) from the latest frame image. Details of the processing will be described later.
(S13) The manual operation image processing unit 161 acquires the AR marker detection flag of the latest frame image and the detection coordinates (AR marker coordinates) of the AR marker image from the AR information storage unit 130 via the AR image generation unit 160. . The manual operation image processing unit 161 records the AR detection information about the acquired latest frame in the frame data storage unit 140. At this time, the manual operation image processing unit 161 also acquires information on the AR marker deformation rate in the latest frame image from the AR information storage unit 130 and includes it in the AR detection information on the latest frame. The frame data storage unit 140 stores previous frame AR detection information 144 recorded for the previous frame.

(S14) The manual operation image processing unit 161 acquires the previous frame table 142 from the frame data storage unit 140.
(S15) The manual operation image processing unit 161 determines whether an AR marker image and a ring marker image exist in both the latest frame image and the previous frame image. When both the AR marker image and the ring marker image exist in both the latest frame image and the previous frame image, the manual operation image processing unit 161 advances the processing to step S16. When at least one of the AR marker image and the ring marker image does not exist in at least one of the latest frame image and the previous frame image, the manual operation image processing unit 161 advances the processing to step S26.

  (S16) The manual operation image processing unit 161 calculates the translation amount, rotation amount, and enlargement / reduction rate of the ring marker image between the previous and latest frames. The parallel movement amount, rotation amount, and enlargement / reduction ratio of the ring marker image correspond to the movement amount of the ring marker image (the amount corresponding to the movement amount of the ring marker 31). The manual operation image processing unit 161 calculates the translation amount and the rotation amount of the ring marker image by detecting the positions of the ring marker image in the previous frame image and the ring marker image in the latest frame image. When obtaining the parallel movement amount, the manual operation image processing unit 161 detects a change in the position of the ring marker image based on the AR marker image in the previous and latest frame images. The manual operation image processing unit 161 calculates the enlargement / reduction ratio by detecting a change in the area of the image area of the ring marker image in the previous and latest frame images.

  (S17) As a result of the calculation in step S16, the manual operation image processing unit 161 determines whether or not there is a change in the positional relationship between the AR marker image and the ring marker image between the previous and latest frames. If there is a change, the manual operation image processing unit 161 proceeds to step S18. If there is no change, the manual operation image processing unit 161 proceeds to step S21.

  When the manual operation image processing unit 161 detects the translation, rotation, or enlargement / reduction of the ring marker image between the previous and latest frames as a result of the calculation in step S16, the AR marker image and the ring marker image are detected. It is determined that there is a change in the positional relationship. On the other hand, when the translation, rotation, or enlargement / reduction of the ring marker image between the previous and latest frames is not detected, the manual operation image processing unit 161 changes to the positional relationship between the AR marker image and the ring marker image. Judge that there is no.

(S18) The manual operation image processing unit 161 generates a mask for the manual operation image (a mask image for masking an area other than the manual operation image). Details of the processing will be described later.
(S19) The manual operation image processing unit 161 combines the mask image generated in step S18 and the mask image generated by the AR image generation unit 160 to generate a mask (AR mask) of the manual operation image. . Specifically, for each coordinate of both mask images, the manual operation image processing unit 161 sets the non-distribution portion to “1” (True) and the distribution portion to “0” (False), and the logical product (AND) of the coordinates. Is calculated to synthesize both mask images. As a result, the manual operation image processing unit 161 obtains an AR mask that masks an area other than both the work target device image and the manual operation image. Note that the mask image is generated by the AR image generation unit 160 until step S19 is executed. A method of generating a mask image by the AR image generation unit 160 will be described with reference to FIGS.

(S20) The manual operation image processing unit 161 inputs the combined AR mask to the image combining unit 156. Then, the manual operation image processing unit 161 proceeds to step S25.
(S21) The manual operation image processing unit 161 determines whether there is a change in the camera angle or zoom in the camera (for example, the rear camera 105) used for capturing the camera image. If there is a change in the camera angle or zoom, the manual operation image processing unit 161 proceeds to step S22. When there is no change in the camera angle or zoom, the manual operation image processing unit 161 advances the processing to step S24.

  (S22) The manual operation image processing unit 161 transforms the previous AR mask in accordance with the change in camera angle / zoom. Here, the manual operation image processing unit 161 acquires the information of the previous AR mask from the mask information table 145 stored in the frame data storage unit 140.

(S23) The manual operation image processing unit 161 inputs the deformed AR mask to the image composition unit 156. Then, the manual operation image processing unit 161 proceeds to step S25.
(S24) The manual operation image processing unit 161 inputs the previous AR mask to the image composition unit 156. Here, the manual operation image processing unit 161 acquires the information of the previous AR mask from the mask information table 145 stored in the frame data storage unit 140. Then, the manual operation image processing unit 161 advances the processing to step S26.

  (S25) The manual operation image processing unit 161 registers the AR mask information input this time to the image composition unit 156 in the mask information table 145 stored in the frame data storage unit 140.

  (S26) The manual operation image processing unit 161 stores the latest frame table 143 in the frame data storage unit 140. The latest frame table 143 is treated as the previous frame table in the next frame processing. Then, the manual operation image processing unit 161 ends the AR mask generation process.

  As described above, the manual operation image processing unit 161 detects the ring marker attached to the operator's hand shown in the two most recent frames. The manual operation image processing unit 161 detects the movement of the ring marker between two frames, and extracts a region other than the image (manual operation image) of the subject portion that moves in the same manner as the ring marker in the latest frame image. To do. The manual operation image processing unit 161 generates a mask image that masks an area other than the manual operation image. The manual operation image processing unit 161 synthesizes the generated mask image with the mask image created with respect to the image of the work target device to transmit the device image and the manual operation image of the work target device, An AR mask to be masked is generated. The image composition unit 156 superimposes the AR mask generated by the manual operation image processing unit 161 on the latest frame image, so that only the device image and the manual operation image of the work target device can be distributed. The distribution of the background part unrelated to the work that becomes can be prevented.

  Here, when using a camera that does not image at a fixed point, such as the rear camera 105, the manual operation image processing unit 161 includes an AR marker and a ring marker in both the previous and latest frame images, as illustrated in step S15. To see if both exist. This is because the movement of the ring marker cannot be detected with the AR marker as a reference when both are not present. When a camera that captures images at a fixed point is used, the movement of the ring marker can be detected with respect to a predetermined reference point in the camera image. Therefore, the determination in step S15 is Yes if the ring marker is detected. If no ring marker is detected, No may be used.

  In addition, when the ring marker is not detected in the camera image, the manual operation image processing unit 161 cannot detect the movement of the ring marker, and thus does not extract the manual operation image. In this case, the manual operation image processing unit 161 provides the image composition unit 156 with the mask image acquired from the AR image generation unit 160 (a mask image for masking other than the work target device) as it is. Thus, even when no ring marker is detected, a state in which nothing is displayed on the support terminal 300 can be avoided by performing distribution using the mask image created by the AR image generation unit 160.

  FIG. 14 is a flowchart illustrating an example of the latest frame table creation process. In the following, the process illustrated in FIG. 14 will be described in order of step number. The procedure shown below corresponds to step S11 in FIG.

  (S31) The manual operation image processing unit 161 creates an empty table (template of the latest frame table 143) having columns of “coordinates”, “color data”, and “type” in a predetermined area on the RAM 102. .

  (S32) The manual operation image processing unit 161 acquires the coordinates and color data of each pixel of the latest frame. The manual operation image processing unit 161 may extract the coordinate and color data information of each pixel from the latest frame image acquired from the AR image generation unit 160, or the coordinate and color of each pixel from the AR image generation unit 160. Data may be acquired.

(S33) The manual operation image processing unit 161 sets the coordinates of the processing target to (X, Y) = (0, 0).
(S34) The manual operation image processing unit 161 registers the coordinates (X, Y) of the latest frame and the color data of the coordinates in the latest frame table 143. Note that the type item is left unset.

(S35) The manual operation image processing unit 161 substitutes X + 1 for X.
(S36) The manual operation image processing unit 161 determines whether X = A (the maximum value of the X coordinate of the frame image). If X = A, the manual operation image processing unit 161 proceeds to step S37. If X <A, the manual operation image processing unit 161 proceeds to step S34.

(S37) The manual operation image processing unit 161 substitutes 0 for X.
(S38) The manual operation image processing unit 161 substitutes Y + 1 for Y.
(S39) The manual operation image processing unit 161 determines whether Y = B (the maximum value of the Y coordinate of the frame image). When Y = B, the manual operation image processing unit 161 ends the latest frame table creation process. If Y <B, the manual operation image processing unit 161 proceeds to step S34.

  FIG. 15 is a flowchart illustrating an example of the ring marker detection process. In the following, the process illustrated in FIG. 15 will be described in order of step number. The procedure shown below corresponds to step S12 in FIG.

(S41) The manual operation image processing unit 161 selects the first row of the latest frame table 143.
(S42) The manual operation image processing unit 161 determines whether or not the color data of the current coordinates (coordinates of the currently selected row) matches the color data of the ring marker 31. If they match, the manual operation image processing unit 161 proceeds to step S43. If they do not match, the manual operation image processing unit 161 advances the processing to step S44.

(S43) The manual operation image processing unit 161 sets “1” indicating the ring marker 31 to the current coordinate type of the latest frame table 143.
(S44) The manual operation image processing unit 161 determines whether or not the current row is the last row of the latest frame table 143. If it is the last line, the manual operation image processing unit 161 ends the ring marker detection process. If it is not the last line, the manual operation image processing unit 161 advances the processing to step S45.

(S45) The manual operation image processing unit 161 selects the next row of the latest frame table 143. Then, the manual operation image processing unit 161 advances the processing to step S42.
FIG. 16 is a flowchart illustrating an example of a manual operation image mask generation process. In the following, the process illustrated in FIG. 16 will be described in order of step number. The procedure shown below corresponds to step S18 in FIG.

(S51) The manual operation image processing unit 161 compares the color data between the previous frame and the latest frame. Details of the processing will be described later.
(S52) The manual operation image processing unit 161 sets the type in the latest frame table 143 of the coordinates not connected to the ring marker image in the latest frame image to “0”. “Coordinates not connected to the ring marker image” refers to the type “2” that is not adjacent to the pixel of the type “1” and is adjacent to the pixel of the type “2” among the pixels of the latest frame image. This is the coordinates of a pixel that cannot reach the pixel 1 ″. On the other hand, the “coordinates connected to the ring marker image” refers to a type “2” that is adjacent to a pixel of type “1” or is adjacent to a pixel of type “2” among the pixels of the latest frame image. This is the coordinates of a pixel that can reach the pixel 1 ″.

  (S53) Based on the latest frame table 143, the manual operation image processing unit 161 generates a mask (mask image) that covers coordinates other than the types “1” and “2”. Then, the manual operation image processing unit 161 ends the manual operation image mask generation process.

  FIG. 17 is a flowchart illustrating an example of color data comparison processing. In the following, the process illustrated in FIG. 17 will be described in order of step number. The procedure shown below corresponds to step S51 in FIG.

(S61) The manual operation image processing unit 161 selects the first coordinate of the previous frame table 142.
(S62) The manual operation image processing unit 161 applies the translation amount, the rotation amount, and the enlargement / reduction rate of the ring marker calculated in step S16 of FIG. 13 to the selected coordinates, and the latest corresponding to the corresponding coordinates. Calculate frame coordinates.

  (S63) The manual operation image processing unit 161 determines whether or not the coordinates calculated in step S62 are within the range of the latest frame (latest frame image). When the calculated coordinates are within the range of the latest frame, the manual operation image processing unit 161 advances the processing to step S64. When the calculated coordinates are not within the range of the latest frame, the manual operation image processing unit 161 advances the processing to step S66. For example, if the calculated coordinates (x, y) satisfy the condition (0, 0) ≦ (x, y) ≦ (A−1, B−1), the coordinates (x, y) are within the range of the latest frame. If the same condition is not satisfied, the coordinates (x, y) are not within the range of the latest frame.

(S64) The manual operation image processing unit 161 compares the color data of the coordinates of the previous frame with the color data of the coordinates of the latest frame that matches the coordinates.
(S65) The manual operation image processing unit 161 determines whether the color data of both pixels compared in step S64 are the same color or similar colors. When the colors are the same or similar, the manual operation image processing unit 161 advances the processing to step S66. If the color is not the same color or a similar color, the manual operation image processing unit 161 advances the processing to step S67. Here, “similar colors” indicate two colors corresponding to the two color data when the difference between the values of the two color data is equal to or smaller than a threshold value. The threshold value for determining the similar color is preset in the frame data storage unit 140, for example.

  (S66) The manual operation image processing unit 161 sets the corresponding coordinate type in the latest frame table 143 to “2”. Then, the manual operation image processing unit 161 advances the processing to step S68.

  (S67) The manual operation image processing unit 161 sets the corresponding coordinate type in the latest frame table 143 to “0”. Then, the manual operation image processing unit 161 advances the processing to step S68.

  (S68) The manual operation image processing unit 161 determines whether or not the current coordinate is the last coordinate of the previous frame table 142. If it is the last coordinate, the manual operation image processing unit 161 ends the color data comparison process. If it is not the last coordinate, the manual operation image processing unit 161 proceeds to step S69.

(S69) The manual operation image processing unit 161 selects the next coordinate in the previous frame table 142. Then, the manual operation image processing unit 161 advances the processing to step S62.
FIG. 18 is a diagram illustrating an example of rotation of the ring marker image. FIG. 18A shows a frame image 52 of the Nth (N is a positive integer) frame. The frame image 52 includes a ring marker image 52a. FIG. 18B shows a frame image 53 of the (N + 1) th frame. The frame image 53 includes a ring marker image 53a.

  In the frame image 52, lines representing the line segments L1 and L2 and the corner W1 are marked. The lines representing the line segments L1 and L2 and the corner W1 are described for convenience and are not included in the frame image 52. Further, the frame image 53 has lines representing the line segments L3 and L4 and the corner W2. The lines representing the line segments L3 and L4 and the corner W2 are described for convenience and are not included in the frame image 53.

  The line segment L1 is a line segment that passes along the center C1 of the ring marker image 52a (for example, the center of gravity of the ring marker image 52a) along the X-axis direction of the frame image 52. The line segment L2 is a line segment that passes through the center C1 of the ring marker image 52a and is orthogonal to the major axis direction of the ring marker image 52a. The corner W1 is an angle formed by the line segments L1 and L2.

  The line segment L3 is a line segment that passes along the center C2 of the ring marker image 53a (for example, the center of gravity of the ring marker image 53a) along the X-axis direction of the frame image 53. The line segment L4 is a line segment that passes through the center C2 of the ring marker image 53a and is orthogonal to the major axis direction of the ring marker image 53a. The corner W2 is an angle formed by the line segments L3 and L4.

  The rotation amount (rotation angle) of the ring marker image 53a with respect to the ring marker image 52a is a difference between the angles W1 and W2. Therefore, the manual operation image processing unit 161 can obtain the rotation amount of the ring marker image 53a with respect to the ring marker image 52a by obtaining the difference between the corners W1 and W2.

  When applying the calculated rotation amount to the previous frame image 52 with respect to the frame image 53, a parallel movement amount (for example, the difference between the centers C1 and C2) is applied to each pixel of the ring marker image 52a, and each pixel after application is applied. In addition, the rotation amount is applied with the coordinate of the center C2 as the rotation center.

  FIG. 19 is a diagram illustrating a comparative example of color data. FIG. 19 exemplifies a case where a translation amount of +1 in the X coordinate direction is detected in the previous frame and the latest frame, particularly for the ring marker image. Note that the threshold value for determining as a similar color is “3” as an example.

In this case, the manual operation image processing unit 161 compares the result of adding 1 to each X coordinate in the previous frame table 142 with the latest frame table 143.
For example, the manual operation image processing unit 161 compares the color data “22” of the coordinate (0, 0) in the previous frame table 142 with the color data “23” of the coordinate (1, 0) in the latest frame table 143. The difference between the two color data is “1” (<threshold “3”). That is, both color data are in a similar color relationship. Therefore, the manual operation image processing unit 161 sets the type of the coordinates “(1,0)” in the latest frame table 143 to “2”.

  For example, the manual operation image processing unit 161 compares the color data “35” of the coordinates (48, 50) in the previous frame table 142 with the color data “21” of the coordinates (49, 50) in the latest frame table 143. To do. The difference between the two color data is “14” (> threshold “3”). In other words, the two color data do not match each other nor have a similar color relationship. Therefore, the manual operation image processing unit 161 sets the type of the coordinates “(49, 50)” in the latest frame table 143 to “0”.

  The manual operation image processing unit 161 sets the type by comparing the other coordinates in the latest frame table 143 in the same manner. However, the manual operation image processing unit 161 keeps the type “1” for the coordinates of the latest frame that matches the previous frame in the color data “0” indicating the ring marker image.

  In FIG. 19, since only parallel movement is considered, the coordinates of the previous frame table 142 and the coordinates of the latest frame table have a one-to-one correspondence. On the other hand, in the case of enlargement, the coordinates of the previous frame table 142 and the coordinates of the latest frame table correspond one-to-many. In the case of reduction, the coordinates of the previous frame table 142 and the coordinates of the latest frame table correspond to many-to-one. In any case, for example, the representative value of the color data of the coordinate group corresponding to “many” (the color data of the pixel at a predetermined position (for example, the center) in the region represented by the coordinate group, or the coordinate group) The color data of the coordinate of the other party is compared with the average value of the color data of each coordinate to which it belongs.

The manual operation image processing unit 161 determines a manual operation image region based on the latest frame table 143 created in this way.
FIG. 20 is a diagram illustrating an example of determining the manual operation image area. Based on the latest frame table 143, the manual operation image processing unit 161 identifies an image area of type “1” or type “2” (a set of coordinates in which type “1” or type “2” is set). In the example of the latest frame image 54, there are image regions 54a and 54b, which are image regions in which pixels of type “2” are continuous. The other image area 54c is an image area of type “0” (a set of coordinates of type “0”). The image areas 54a and 54b are not adjacent to each other, and are image areas divided by an image area of type “0”. That is, the image areas 54a and 54b are not connected.

  The manual operation image processing unit 161 includes an image region 54a (including a ring marker region) adjacent to or connected to the ring marker region (a set of coordinates of type “1”) among the image regions 54a and 54b. ) As a manually operated image area. It can also be said that the image region 54a is an image region including a ring marker region. On the other hand, the manual operation image processing unit 161 does not set the image region 54b that is not connected to the ring marker region (not including the ring marker region) as the manual operation image region. In this case, the manual operation image processing unit 161 changes the type of the image area 54b from “2” to “0”.

  Then, the manual operation image processing unit 161 determines the image areas 54b and 54c that are not included in the manual operation image area as mask targets. On the other hand, the manual operation image processing unit 161 determines that the image region 54a, which is a manual operation image region, is not to be masked.

  Accordingly, the manual operation image processing unit 161 can appropriately detect the manual operation image area including the hand of the worker U1 who performs the work and the cables and tools held by the hand. In addition, by excluding pixels that are not connected to the ring marker image from the manual operation image area, pixels that have been detected by chance when interlocked with the ring marker (pixels unrelated to the manual work area) are appropriately masked, Information leakage can be prevented.

  A method for specifying a background image to be masked more strictly can be considered. For example, the background image to be masked is captured in advance by the smart terminal 100, the background image is compared with the latest frame image, the background image in the latest frame image is extracted, and the extracted background image is masked. It may be considered. Further, for example, it is also conceivable that the smart terminal 100 recognizes the manual operation image in advance and sets a region other than the recognized manual operation image as a mask target as a background image.

Next, a specific example of masking an image area other than the work target device using the frame image by the AR image generation unit 160 of the smart terminal 100 will be described.
FIG. 21 is a diagram illustrating a display example of a frame image. For example, consider a case where the worker U1 uses the smart terminal 100 to perform maintenance on the information device 10a. Information devices 10b and 10c are arranged side by side next to the information device 10a. An AR marker 11a is provided on a part of the front surface of the information device 10a that is a maintenance work target.

  In the following, an example in which the worker U1 operates the smart terminal 100 will be described, but the worker who performs the manual operation on the work target device and the worker who operates the smart terminal 100 may be different. In this case, a worker who performs manual work on the work target device performs work by attaching the ring 30 to the arm.

  The worker U1 inputs the work ID of the maintenance work to the smart terminal 100, and points the rear camera 105 toward the information device 10a. Then, the rear camera 105 starts imaging. The smart terminal 100 displays the camera image acquired by the rear camera 105 on the display 106. Here, the visual field of the rear camera 105 includes the information devices 10a, 10b, and 10c. For this reason, the camera images displayed on the display 106 include device images P11, P12, and P13 corresponding to the information devices 10a, 10b, and 10c, respectively.

  In addition, the smart terminal 100 displays the buttons Pt1 and Pt2 so as to overlap the camera image displayed on the display 106. The button Pt1 is a work end button for inputting the end of the maintenance work to the smart terminal 100. The button Pt2 is an AR generation button for inputting to the smart terminal 100 the start of generation of an AR image including a maintenance work support image and a mask image.

  The smart terminal 100 detects a marker image M11 corresponding to the AR marker 11a from the camera image. Then, the smart terminal 100 acquires the marker ID indicated by the marker image M11 and generates a frame image Fr10 of the work target device corresponding to the set of the work ID and the marker ID. The smart terminal 100 displays the frame image Fr10 so as to overlap the camera image displayed on the display 106. For example, the smart terminal 100 displays the frame image Fr10 in the center of the display area of the display 106. The smart terminal 100 fixes the arrangement of the frame image Fr10 with respect to the display area of the display 106.

At this time, the frame image Fr10 does not match the edge of the device image P11.
FIG. 22 is a diagram illustrating a frame image display example (continued). The worker U1 changes the position of the smart terminal 100 and aligns the edge of the device image P11 with the frame image Fr10. FIG. 22 illustrates a state in which the edge of the device image P11 is aligned with the frame image Fr10. The worker U1 presses the button Pt2 when the edge of the device image P11 is aligned with the frame image Fr10. Then, the smart terminal 100 calculates the positional relationship between the marker image M11 and the frame image Fr10. Specifically, the smart terminal 100 calculates the distance (represented by a pixel value) between the reference point of the marker image M11 and each side of the frame image Fr10. Thereby, the smart terminal 100 can grasp | ascertain where the marker image M11 exists in the apparatus image P11.

In addition, the smart terminal 100 generates a mask image that masks an area other than the device image P11.
FIG. 23 is a diagram illustrating a generation example of a mask image for the device image. In FIG. 23, points C10, C11, C12, C13, C21, C22, C23, and C24 in the display area of the display 106 are shown. Point C10 is a reference point (x1, y1) of marker image M11 in the XY coordinate system. Point C11 is a point at which a straight line extending from point C10 in the negative direction of the Y axis intersects with one side of frame image Fr10. Point C12 is a point where a straight line extending from point C10 in the negative direction of the X-axis and one side of frame image Fr10 intersect. Point C13 is a point where a straight line extending from point C10 in the positive direction of the X-axis and one side of frame image Fr10 intersect. Point C21 is a point where a straight line extending from point C10 in the negative direction of the Y-axis and one side of the edge of display 106 intersect. Point C22 is a point where a straight line extending from point C10 in the negative direction of the X-axis and one side of the edge of display 106 intersect. Point C23 is a point where a straight line extending in the positive direction of the X axis from point C10 and one side of display 106 intersect. Point C24 is a point where a straight line extending from point C10 in the positive direction of the Y-axis and one side of the edge of display 106 intersect.

  The AR image generation unit 160 obtains the distance (pixel value) between the following points. Specifically, the distance between the points C10 and C21 is α pixels. The distance between the points C10 and C22 is β pixels. The distance between the points C11 and C21 is γ pixels. The distance between points C12 and C22 is δ pixels. The distance between the points C22 and C23 is ε pixels. The distance between the points C21 and C24 is ζ pixels.

  FIG. 24 is a diagram illustrating a generation example (continued) of the mask image for the device image. The AR image generation unit 160 generates a rectangular mask image P21 of 2ζ pixels in the vertical direction and ε pixels in the horizontal direction that covers the region of X ≦ x1 with respect to the point C10. The AR image generation unit 160 translates the mask image P21 by (β−δ) pixels in the negative direction of the X axis. FIG. 24 shows the mask image P21 after the movement.

  The AR image generation unit 160 generates a rectangular mask image P22 of vertical ζ pixels and horizontal 2ε pixels covering the region of Y ≦ y1 for the point C10. The AR image generation unit 160 translates the mask image P22 by (α−γ) pixels in the negative direction of the Y axis. FIG. 24 shows the mask image P22 after the movement.

  Similarly, the AR image generation unit 160 generates, for the point C10, a rectangular mask image P23 having a vertical 2ζ pixel and a horizontal ε pixel that covers an area where X ≧ x1. Then, the AR image generation unit 160 translates the mask image P23 in the positive direction of the X axis (until the left side of the mask image P23 reaches the right side of the frame image Fr10).

  In addition, the AR image generation unit 150 generates a rectangular mask image P24 of vertical ζ pixels and horizontal 2ε pixels covering the region of Y ≧ y1 for the point C10. The AR image generation unit 160 translates the mask image P24 in the positive direction of the Y axis (until the upper side of the mask image P24 reaches the lower side of the frame image Fr10).

  In this way, the AR image generation unit 160 generates mask images P21, P22, P23, and P24 that cover an area other than the device image P11. The AR image generation unit 160 combines the mask images P21, P22, P23, and P24 into one mask image P20.

  After generating the mask image P20, the AR image generation unit 160 moves and deforms the mask image P20 according to changes in the position and shape of the AR marker image in the latest frame image, and follows the change in the AR marker image. Thus, the display position of the mask image P20 is changed.

  As described above, the reason why the mask image is generated according to the number of pixels in the vertical and horizontal directions of the display 106 is that it is required to generate an image having a finite size as the AR image. At that time, it is considered sufficient to cover about 2 to 3 times the number of pixels of the display 106 in the vertical and horizontal directions. The smart terminal 100 may stop the image transmission when the AR marker disappears from the view of the rear camera 105 and the AR marker is not detected.

  FIG. 25 is a diagram illustrating a synthesis example of an AR mask. The mask image P20 masks an area other than the device image P11 in the latest frame image. The mask image P30 masks an area other than the manual operation image P15 in the latest frame image. The mask image P40 is an AR mask created by combining the mask images P20 and P30, and masks areas other than both the device image P11 and the manual operation image P15.

  The manual operation image processing unit 161 sets the non-delivery part to “1” (True) and the delivery part to “0” (False) for the coordinates of the mask images P20 and P30. Then, the manual operation image processing unit 161 calculates the logical product (AND) of the same coordinates in the mask images P20 and P30, thereby synthesizing the mask images P20 and P30 to generate the mask image P40.

  FIG. 26 is a diagram illustrating a display example of the support terminal. At maintenance sites such as network equipment, in order to support work by AR, the camera is used to take pictures of the work target equipment, display superimposed data on the taken images, and send images and videos to the remote supporter U2 Sometimes.

  However, there are cases where devices from a plurality of companies are contained in a maintenance work site such as a customer's server room. Even if it is an external device or an in-house device, photographing an unworked device and distributing the image of the device may lead to information leakage, which is a security problem. In addition, when the devices are dense, there is a possibility that the adjacent devices that are not the target of work are unintentionally reflected. For this reason, it is preferable not to distribute the image around the work target device. On the other hand, it is possible to make it possible for the supporter U2 at the back to check the status of manual work by the worker 40 on the site or the other worker's hand 40 and to perform appropriate work support by the supporter U2. It is done. However, if the entire periphery of the work target device is masked, it will be difficult to confirm the state of manual work on site.

  Therefore, the smart terminal 100 transmits image information of the display image G1 including the mask image P40 to the streaming server 200. Then, the streaming server 200 distributes the image information to the support terminal 300. The support terminal 300 displays the display image G2 based on the distributed image information. The display image G2 also includes a mask image P40 as with the display image G1. For this reason, it is not necessary to show the supporter U2 a device image other than the information device 10a that is the work target. Furthermore, it is possible to check the manual work situation by the worker U1 by the supporter U2. In this way, it is possible to check the manual work status of the worker U1 while preventing information leakage related to information devices other than the work target device installed at the maintenance work site, and appropriate support by the supporter U2 becomes possible.

  Also, an operator on site (for example, the worker U1 or another worker) can distribute an image of the same movement as the hand 40 to the support terminal 300 as long as the ring marker 31 is attached to the hand 40. It is. For this reason, it is not necessary to photograph the distribution target in advance with the smart terminal 100 or register it in the smart terminal 100, and the maintenance work can be completed quickly.

  Here, in the above example, the method of detecting the mask target region for the work target device using the frame image by the smart terminal 100 is illustrated, but the mask target region for the work target device is determined by other methods. May be. For example, the smart terminal 100 stores in advance information on the relative position of the work target device with respect to the AR marker attachment position, and determines the position of the mask target region with respect to the detection position of the AR marker image based on the information. May be.

  It is also conceivable that AR markers are attached in advance to a plurality of devices. In this case, the smart terminal 100 may hold information used for identifying the ring marker 31 and information used for identifying the AR marker of the work target device corresponding to the ring marker 31 in advance in association with each other. . Then, the smart terminal 100 identifies the AR marker image of the AR marker corresponding to the ring marker 31 from the camera image based on the stored information, and masks a device other than the work target device with respect to the AR marker image. May be generated.

  Further, in the example of the second embodiment, the ring marker 31 that can be attached to the arm (wrist) of the worker U1 has been exemplified, but the ring marker 31 or a marker of the same use as the ring marker 31 is used as a finger or the back of the hand. It is also possible to attach it to another part of the. The attachment method is not limited to the case where the ring 30 is used, and other methods such as attachment by a seal adhered to the skin are also conceivable.

  Furthermore, the smart terminal 100 may be connected to a head-mounted display (head-mounted display). The head mounted display may include a camera that images a work area of the worker U1 and inputs a camera image to the smart terminal 100. In this case, the worker U1 can receive work assistance by visually recognizing an image displayed on the head mounted display while imaging an area (including a manual work portion) on which the worker U1 is working with the head mounted display. it can.

  Note that the information processing of the first embodiment can be realized by causing a processing device that functions as the detection unit 1b, the calculation unit 1c, and the image processing unit 1d to execute a program. The information processing according to the second embodiment can be realized by causing the processor 101 to execute a program. The program can be recorded on a computer-readable recording medium 22.

  For example, the program can be distributed by distributing the recording medium 22 on which the program is recorded. Alternatively, the program may be stored in another computer and distributed via a network. For example, the computer stores (installs) a program recorded on the recording medium 22 or a program received from another computer in a storage device such as the RAM 102 or the flash memory 103, and reads and executes the program from the storage device. May be.

DESCRIPTION OF SYMBOLS 1 Terminal 1a Memory | storage part 1b Detection part 1c Calculation part 1d Image processing part 2 Imaging device 3 Output device 4,5 Object 6,9 Marker 7 Hand 8 Parts P1, P2 Input image P3 Output image P1a, P1b, P2a, P2b, P3b Object image P1c, P1d, P2c, P2d, P3c, P3d Marker image P1e, P2e, P3e Hand image P1f, P2f, P3f Parts image P3a Mask image

Claims (15)

A detection unit for detecting a marker and movement of the marker from the first image and the second image;
A calculation unit for identifying a first image region corresponding to the movement of the marker from the second image;
An image processing unit that performs a first process on the first image area and performs a second process different from the first process on a second image area of the second image;
A terminal having The detection unit detects a movement amount of the marker;
The calculation unit identifies, as the first image region, an image region that is linked to the marker based on the movement amount.
The terminal according to claim 1. The calculation unit converts the coordinates of the first pixel included in the first image by the movement amount, and the color of the second pixel of the second image that is the same as the coordinate after the conversion and the color Identifying an image region associated with the marker according to a comparison with the color of the first pixel;
The terminal according to claim 2. The calculation unit determines the second pixel as a pixel interlocked with the marker when the color of the first pixel and the color of the second pixel match or are similar to each other. Among them, an area in which the second pixels are continuous is specified as an image area linked with the marker.
The terminal according to claim 3.   When there are a plurality of image areas linked to the marker, the calculation unit identifies the image area including the marker as the first image area, and sets the image area not including the marker as the first image area. The terminal according to any one of claims 2 to 4, wherein the terminal is excluded.   The terminal according to claim 2, wherein the movement amount includes at least one of a parallel movement amount, a rotation amount, and an enlargement / reduction ratio of the marker. The first process is a process of outputting the first image area without masking;
The second process is a process of masking and outputting the second image area.
The terminal according to any one of claims 1 to 6. The detection unit detects another marker included in the second image;
The calculation unit specifies a third image region according to the information of the other marker,
The image processing unit excludes the third image area from the second image area from being masked.
The terminal according to claim 7. The marker is attached to the hand;
The calculation unit identifies the first image area in which the hand is copied in accordance with the movement of the marker, and the third device in which the work target device by the hand is copied in accordance with information on the other marker. Identify the image area of
The terminal according to claim 8.   The terminal according to claim 8 or 9, wherein the detection unit detects movement of the marker with reference to images of the other markers included in both the first image and the second image. The marker is attached to the hand;
The calculation unit specifies the first image region in which the hand is copied in accordance with the movement of the marker.
The terminal according to any one of claims 1 to 8.   The terminal according to any one of claims 9 to 11, wherein the first image region includes an image of an object interlocked with the hand. A storage device for storing the first image and the second image;
A marker and a movement of the marker are detected from the first image and the second image stored in the storage device, and a first image area corresponding to the movement of the marker is specified from the second image A processing device that performs a first process on the first image area and performs a second process different from the first process on a second image area of the second image;
An image processing system. Detecting a marker and movement of the marker from the first image and the second image;
A first image region corresponding to the movement of the marker is identified from the second image;
Performing a first process on the first image area, and performing a second process different from the first process on a second image area of the second image;
An image processing program that causes a computer to execute processing. Computer
Detecting a marker and movement of the marker from the first image and the second image;
A first image region corresponding to the movement of the marker is identified from the second image;
Performing a first process on the first image area, and performing a second process different from the first process on a second image area of the second image;
Image processing method.

Images