JP2021035002A - Image specification system and image specification method - Google Patents

Abstract

To provide an image specification technique capable of accurately selecting a photographic image including an object of photography limitation and thereby suppressing the image from being leaked to the outside.SOLUTION: An image specification system 1 comprises: a camera 8 which is mounted on a movable terminal 2 and photographs images; a position information acquisition part 33 which acquires position information specifying a position of photography of an image by the camera 8; a direction information acquisition part 34 which acquires direction information specifying a direction of the photography of the image by the camera 8; an angle-of-view information acquisition part 35 which acquires an angle-of-view information specifying an angle of view of the photography of the image by the camera 8; and a photography information recording part 36 which records photography information including the position information, direction information and angle-of-view information associatively with the image.SELECTED DRAWING: Figure 2

Description

Embodiments of the present invention relate to an image identification system and an image identification method.

Conventionally, there is a technique for preventing leakage of predetermined information by using an image. For example, a technique is known for the purpose of obstructing voyeurism of exhibits such as museums. Further, there is known a technique for prohibiting shooting at a place other than the permitted place when shooting a surveillance place with a camera attached to a flying robot. Further, there is known a technique of automatically determining an object for which photography is permitted or prohibited for photography and controlling whether or not an image of the object can be saved.

Japanese Unexamined Patent Publication No. 2011-257759 Japanese Unexamined Patent Publication No. 2014-177162 Japanese Unexamined Patent Publication No. 2014-192753 Japanese Patent No. 5731083

A technique that automatically determines an object for which photography is permitted or prohibited and controls whether or not to save an image in which the object appears may inform the photographer of the location of the object including confidential information. For example, even if the photographer does not know that the object contains confidential information just by checking the object with the naked eye, it is found that the image contains confidential information by being refused to save the image. It ends up. Further, even if the image is subjected to mosaic processing so that the portion in which the object whose photography is prohibited is shown cannot be discriminated, it is found that the object is in the portion where the mosaic processing is performed. In addition, it is possible to manually select and exclude an image showing an object whose shooting is prohibited so that the image is not taken out, but there is a risk of mistakes such as oversight.

An embodiment of the present invention has been made in consideration of such circumstances, and accurately selects an image showing a target portion subject to shooting restrictions, and the image is leaked to the outside. It is an object of the present invention to provide an image identification technique capable of suppressing the above.

The image identification system according to the embodiment of the present invention is mounted on a movable terminal, and is a camera that captures an image and a position information acquisition unit that acquires position information that can identify the position at the time of capturing the image by the camera. And the direction information acquisition unit that acquires the direction information that can specify the direction when the image is taken by the camera, and the image angle that acquires the image angle information that can specify the image angle when the image is taken by the camera. It includes an information acquisition unit and a shooting information recording unit that records shooting information including the position information, the direction information, and the angle of view information in association with the image.

An embodiment of the present invention provides an image identification technique capable of accurately selecting an image in which a target portion subject to photography restriction is captured and suppressing leakage of the image to the outside. ..

The system block diagram which shows the image identification system of 1st Embodiment. The block diagram which shows the image identification system of 1st Embodiment. Top view showing the work site where the image identification system is used. A perspective view showing a state of shooting using a field terminal. Explanatory drawing which shows the reference point management table. Explanatory drawing which shows the target device management table. Explanatory drawing which shows the terminal management table. Explanatory drawing which shows the recording mode of the photographing information. The screen view which shows the image judgment result displayed on the management computer. A flowchart showing the management control process executed by the management computer. A flowchart showing an image judgment process executed by the management computer. A flowchart showing a terminal control process executed by a field terminal. A flowchart showing a marker recognition process executed by a field terminal. A flowchart showing the self-position estimation process executed by the field terminal. A flowchart showing a shooting control process executed by a field terminal. The block diagram which shows the image identification system of 2nd Embodiment. A flowchart showing an image judgment process executed by the management computer. A flowchart showing a shooting control process executed by a field terminal.

(First Embodiment)
Hereinafter, embodiments of the image identification system and the image identification method will be described in detail with reference to the drawings. First, the image identification system and the image identification method of the first embodiment will be described with reference to FIGS. 1 to 15.

Reference numeral 1 in FIG. 1 is the image identification system of the first embodiment. The image identification system 1 includes a site terminal 2 possessed by a worker W at a work site and a management computer 3 handled by an administrator M who monitors or supports the worker W.

The work site of this embodiment includes a power plant, a chemical plant, a factory, and the like. Confidential information, including confidential information or information that should be avoided, may be managed at these work sites. Confidential information includes information about facilities or equipment provided within this controlled area. For example, information such as the installation location of the target device 4 (see FIG. 4) such as a surveillance camera as the target part is included.

When it is prohibited to take confidential information out of the management area when it is taken at a predetermined shooting location in the management area for the purpose of reducing security risks or preventing leakage of technical information. There is. In particular, when taking out photography equipment, the internal memory is checked to manage information.

However, when the worker W is performing inspection work or the like at the work site, the image taken by the site terminal 2 may include information related to confidential information. Therefore, in the image identification system 1, the management computer 3 is used to select (screen) an image file in which information related to confidential information is shown and an image file in which information related to confidential information is not shown. Then, the image file in which the confidential information is included is excluded from the disclosure target, and only the image in which the confidential information is not included can be disclosed to others.

The management computer 3 is provided in a check area for checking images. Then, the administrator M operates the management computer 3 to select the image files. The image file may be automatically selected by the management computer 3 without the administrator M.

In the present embodiment, the target device 4 as the target portion subject to the shooting restriction is illustrated as a surveillance camera, but other embodiments may be used. For example, the target device 4 as the target part may be an instrument, an operation switch, a valve, a pipe, a device, or a sensor.

Further, in the present embodiment, the target device 4 is illustrated as the target part, but the target part is a predetermined part of the building, for example, a window, a door, a wall, a ceiling, a floor, a passage, an opening of a wall, or an interior space. It may be a part of the range of. Further, a part of one huge device may be a target part. Further, a plurality of locations of one device may be target portions.

In the image identification system 1, the position of the target device 4 is specified in advance. Then, based on the position of the on-site terminal 2, the shooting direction, and the angle of view, the management computer 3 determines whether or not the target device 4 is included in the shot image. By doing so, it is not necessary for the administrator M to visually select the images, and mistakes such as oversight can be prevented. Further, even if the target device 4 is not clearly shown in the image, if there is a possibility that the target device 4 is shown, the image can be selected.

The worker W takes a picture at the work site using the site terminal 2, and the image is transmitted to the management computer 3. The image of the present embodiment is a so-called moving image, and may be either a video image (moving image) or a still image. Further, the "image file" will be described as including not only a file including a still image but also a file including a moving image.

The mobile field terminal 2 handled by the worker W is composed of, for example, a smartphone 7 provided with a display 6 (see FIG. 2). The smartphone 7 is equipped with a camera 8 (see FIG. 2) for photographing the situation at the site. The display 6 of the smartphone 7 is not only a display device but also a touch panel as an input device. The input operation of the present embodiment includes a touch operation using a touch panel.

Further, the different types of field terminals 2 handled by the other worker W are composed of a wearable computer 10 (smart glasses) provided with a transmissive head-mounted display 9 worn by the worker W. The wearable computer 10 may have a structure separated from the transmissive head-mounted display 9. The wearable computer 10 is also equipped with a camera that captures the situation at the site.

The field terminal 2 may be composed of a tablet PC or a notebook PC. In addition, it is composed of devices having a shooting function, a call function, and a communication function, and may be composed of a plurality of devices having each function.

Since the system configurations of the smartphone 7 and the wearable computer 10 are almost the same, the field terminal 2 composed of the smartphone will be illustrated in the following description. The present embodiment can also be applied to the wearable computer 10.

The management computer 3 handled by the administrator M is composed of a predetermined computer such as a desktop PC, a notebook PC, or a tablet PC, for example. In this embodiment, a desktop PC is illustrated. The management computer 3 includes a computer main body 11, a display 12 visually recognized by the administrator M, and a wireless communication device 13 that wirelessly communicates with the field terminal 2. Here, an example of performing wireless communication is described, but wired communication is also possible. The display 12 may be separate from the computer main body 11 or may be integrated with the computer main body 11.

In the present embodiment, the field terminal 2 can specify its own position and its own posture. For example, it may have a position measurement system such as GPS (Global Positioning System).

Further, known techniques such as SLAM (Simultaneous Localization and Mapping) and SfM (Structure from Motion) can be used as the position estimation technique used for calculating the displacement of the self-position and the self-posture of the field terminal 2.

The field terminal 2 of the present embodiment uses VSLAM (Visual Simultaneous Localization and Mapping) to calculate the displacement of its position and posture based on the image taken by the field terminal 2. That is, the field terminal 2 can estimate its own position even in a place where GPS cannot be used, such as indoors.

In the VSLAM technology, feature points of surrounding objects are extracted using information acquired by a predetermined device such as a camera 8 and a sensor of a field terminal 2. Then, the moving image taken by the camera 8 is analyzed, and the feature points (for example, parts of the object such as corners) of the object are tracked in real time. Then, the three-dimensional information of the position or posture of the field terminal 2 is estimated.

As shown in FIG. 3, a predetermined building 14 is illustrated as a work site where the worker W performs the work. For example, a building 14 provided in a predetermined section of a power plant is a work site. This work site has a plurality of rooms 15 surrounded by walls. Of these plurality of rooms 15, one room 15 is preset as a work site. That is, the room 15 is preset as a permitted area 16 in which the field terminal 2 is permitted to acquire an image. It should be noted that other than the permitted area 16, the image acquisition is not permitted (non-permitted area).

The entrance / exit 17 of the building 14 is set as a reference point that serves as a reference for estimating the position of the site terminal 2. A marker 5 is arranged at the entrance / exit 17 of the building 14. Further, the entrance / exit 18 of each room 15 is also set as a reference point as a reference for estimating the position of the site terminal 2. Markers 5 are also arranged at the doorways 18 of these rooms 15. The marker 5 may be provided on the wall surface near the entrances 17 and 18, or may be provided on the door surface of the entrances 17 and 18.

When the worker W starts the work, the worker W first photographs the marker 5 at the doorway 17 of the building 14. Here, the field terminal 2 estimates its own position and its own posture based on the captured image in which the marker 5 is captured. By starting the shooting of the marker 5 at the entrance / exit 17 of the building 14 and continuing the shooting even during the movement, the movement route 19 in which the site terminal 2 has moved can be estimated from the reference point.

The marker 5 is a figure capable of image recognition. For example, a matrix type two-dimensional code, a so-called QR code (registered trademark), is used as the marker 5. Further, the marker 5 includes information indicating a marker ID that can individually identify the corresponding marker 5. In this way, the plurality of markers 5 can be identified respectively.

In the management computer 3, for example, information about the target device 4 is registered corresponding to the marker ID of the marker 5 provided at the entrance / exit 18 of each room 15. If the marker 5 at the entrance / exit 18 of the room 15 is not photographed to obtain a reference point, it may be controlled so that the image cannot be photographed inside the room 15.

When the worker W arrives at the doorway 18 of the room 15, the worker W first reads the marker 5 with the site terminal 2. Then, the coordinates of the reference point acquired based on the marker 5 and the coordinates of the reference point set in advance corresponding to the room 15 are matched with each other. Then, when the image is saved on the site terminal 2, the position information and the direction information at the time of shooting are acquired based on the coordinates of the reference point.

As an embodiment in which the marker 5 is provided, for example, a panel on which the marker 5 is printed is created, and this panel is arranged on a wall surface or the like of a work site before the start of work. The marker 5 may be printed directly on the wall surface or the like.

As shown in FIG. 4, in the permitted area 16, the worker W performs various shootings using the on-site terminal 2. The image taken at this time is stored in the field terminal 2 as an image file. In the image file stored in the field terminal 2, shooting information (see FIG. 8) including the position of the field terminal 2 at the time of shooting, the shooting direction, and the angle of view is recorded in association with each other. The shooting information may include other information. For example, the shooting information also includes information such as a viewpoint (angle) or a magnification (scale).

Further, the image file including the image taken by the field terminal 2 is transmitted to the management computer 3. The timing of transmitting the image to the management computer 3 may be during the work or after the work is completed. Further, when the transmission to the management computer 3 is completed, the image stored in the field terminal 2 may be deleted.

In this embodiment, shooting information is recorded as an attribute of an image file including an image. In this way, the image file and the shooting information can be directly associated with each other. For example, when an image is saved in the Exif (Exchangeable image file format) format, shooting information can be recorded as its property (attribute). Even if the image file is moved, the shooting information is also moved, which facilitates the management of the image file.

For example, as shown in FIG. 8, the property of the image file includes the file name, the terminal ID, the user ID, the shooting date / time (shooting time), the position information (coordinate value), the direction information (angle value), and the angle of view information (angle value). Angle value) and is recorded. When the judgment is made by the management computer, the judgment result is also recorded. That is, various information can be added to the property.

Further, the shooting information may be generated as a predetermined identifier. The identifier is, for example, output as a predetermined mathematical expression when each value of shooting information is input to a predetermined calculation function. Then, the shooting information can be reproduced based on the identifier. In this way, the shooting information can be recorded in a defined manner called an identifier. For example, the identifier may be recorded in the attribute of the image file.

Further, the identifier may be recorded in a manner of being included in the image. In this way, the image itself can include shooting information. Therefore, image management becomes easy. The mode in which the identifier is included in the image may be, for example, a mode in which the identifier is generated as a matrix type two-dimensional code and the two-dimensional code is synthesized in the image. Also. The identifier may be included in the image as a digital watermark.

Further, the shooting information may be recorded in a recording file different from the image file including the image. For example, a recording file 21 having the same file name as the image file 20 may be generated (see FIG. 9), and shooting information may be recorded in this recording file 21. In this way, even when the amount of shooting information data is large, the recording can be performed. For example, in the case of a moving image, the amount of shooting information data is large, but even in this case, the shooting information can be recorded.

In the present embodiment, the image file 20 and the recording file 21 are associated with each other by the file name, but other embodiments may be used. For example, the image file 20 may be associated with the image file 20 by generating a specific ID that can identify the image file 20 and recording the specific ID in the recording file 21. Further, the image file 20 and the recording file 21 may be associated with each other by using the file system of the computer.

Next, the system configuration of the image identification system 1 will be described with reference to the block diagram shown in FIG.

The field terminal 2 and the management computer 3 of the present embodiment have hardware resources such as a CPU, ROM, RAM, and HDD, and when the CPU executes various programs, information processing by software uses the hardware resources. It consists of a computer to be realized. Further, the image identification method of the present embodiment is realized by causing a computer to execute a program.

The field terminal 2 includes a camera 8, an acceleration sensor 22, an angular velocity sensor 23, a three-dimensional measurement sensor 24, a display 6, an operation input unit 25, a communication unit 26, an image storage unit 27, and a terminal control unit 28.

The camera 8 is mounted on the field terminal 2 and photographs an object around the field terminal 2 with visible light. The camera 8 is a stereo camera as the device of the present embodiment. For example, it includes two image sensors with lenses. The camera 8 can acquire information in the depth direction to the object by simultaneously photographing the object from two different directions.

Further, although the image acquired by each image element of the camera 8 is a two-dimensional image, it is possible to generate a three-dimensional photograph capable of grasping the three-dimensional space based on the difference between the images captured in these two two-dimensional images. It is possible. The captured image captured by the camera 8 is input to the terminal control unit 28.

The acceleration sensor 22 is mounted on the field terminal 2 and detects the acceleration generated when the field terminal 2 moves. Further, the acceleration sensor 22 can also detect the gravitational acceleration. The acceleration information indicating the value of the acceleration detected by the acceleration sensor 22 is input to the terminal control unit 28.

The angular velocity sensor 23 (gyro sensor) is mounted on the field terminal 2 and detects the angular velocity generated when the posture of the housing of the field terminal 2 changes. The shooting direction of the camera 8 can be grasped from the posture of the housing of the field terminal 2. The angular velocity information indicating the value of the angular velocity detected by the angular velocity sensor 23 is input to the terminal control unit 28.

The shooting direction is acquired, for example, based on the measurement of the horizontal and vertical tilts of the field terminal 2. When two or more cameras 8 having different orientations are mounted on the housing of the field terminal 2, the position and orientation of the camera 8 used for shooting in the housing are set, and the shooting direction of the camera 8 is set. To get.

The field terminal 2 may be equipped with a motion sensor, which is a 9-axis sensor that combines an inertial sensor (3-axis acceleration sensor and 3-axis angular velocity sensor) and a 3-axis geomagnetic sensor.

The three-dimensional measurement sensor 24 measures the three-dimensional shape of an object around the field terminal 2. As the three-dimensional measurement sensor 24, for example, a depth sensor is used. A laser sensor such as an infrared sensor or LiDAR may be used as the three-dimensional measurement sensor 24.

The three-dimensional measurement sensor 24 is provided on the back side of the housing of the field terminal 2 where the camera 8 is provided. The three-dimensional measurement sensor 24 can measure the distance from the field terminal 2 to the object, for example, by projecting a laser onto the object and receiving the reflected light by the light receiving element. In the present embodiment, the three-dimensional measurement sensor 24 measures the distance from the field terminal 2 to surrounding objects by using a ToF (Time of Flight) method that converts the delay time of the received light pulse with respect to the floodlight pulse into a distance. To make a three-dimensional point group. The imaging direction by the camera 8 and the measurement direction by the three-dimensional measurement sensor 24 are the same.

The display 6 displays a captured image captured by the camera 8. It should be noted that the captured image taken by one of the two image elements with a lens may be displayed.

The operation input unit 25 is a device operated by the operator W so that a predetermined information input operation can be performed. In the present embodiment, the operation input unit 25 is composed of a touch panel provided integrally with the display 6. When the wearable computer 10 is used as the field terminal 2, the operation input unit 25 is arranged at the hand of the worker W, and the input operation can be appropriately performed.

The communication unit 26 communicates with the management computer 3 via the communication line. The communication unit 26 of the present embodiment includes a network device, for example, a wireless LAN antenna. In addition, communication may be performed using a LAN cable or a USB cable. The field terminal 2 accesses the management computer 3 via the communication unit 26.

The image storage unit 27 stores an image file including an image taken by the camera 8. The image stored in the image storage unit 27 may be a moving image or a still image. Further, the recording file associated with the image file may be stored together with the image file. Further, predetermined data related to the image file may be stored.

The terminal control unit 28 comprehensively controls the field terminal 2. The terminal control unit 28 includes a position / orientation estimation unit 29, a reference acquisition unit 30, an area identification unit 31, an image acquisition control unit 32, a position information acquisition unit 33, a direction information acquisition unit 34, an angle of view information acquisition unit 35, and shooting information. A recording unit 36 and an identifier generation unit 37 are provided. These are realized by executing the program stored in the memory or the HDD by the CPU.

The position / attitude estimation unit 29 controls to estimate the position and attitude of the site terminal 2 based on the captured image, the acceleration information, and the angular velocity information. That is, the position and posture of the field terminal 2 are estimated based on the information obtained by the device mounted on the field terminal 2. The position / orientation estimation unit 29 estimates the position and orientation of the mobile terminal and at the same time creates an environment map including information on the surrounding environment of the site terminal 2. That is, the position / orientation estimation unit 29 uses the VSLAM technology. By doing so, it is possible to improve the acquisition accuracy of the position and direction at the time of taking an image.

In the VSLAM technology, the feature points of the objects around the field terminal 2 can be extracted by using the information acquired by the camera 8 and the three-dimensional measurement sensor 24 of the field terminal 2. A collection of these feature points is called three-dimensional feature point cloud data. Then, the captured image (moving image) captured by the camera 8 is analyzed, and the feature points of the object (for example, the sides or corners of the box-shaped object) are tracked in real time. Based on this three-dimensional feature point cloud data, it is possible to estimate the three-dimensional information of the position and orientation of the field terminal 2.

In addition, the three-dimensional feature point cloud data of the objects around the field terminal 2 is detected at predetermined time intervals, and the position and orientation of the field terminal 2 are based on the displacement between the three-dimensional feature point cloud data before and after in the time series. The displacement of can be calculated. Further, from a series of self-positions and self-postures obtained in time series, the movement path 19 of the field terminal 2 prior to the current position and the current posture can be obtained.

That is, the position / orientation estimation unit 29 controls to estimate the position and orientation of the site terminal 2 based on the captured images continuously captured when the site terminal 2 moves from the reference point. In this way, the path 19 (see FIG. 3) in which the site terminal 2 has moved from the reference point can be estimated from the image, so that the estimation accuracy of the position and posture of the site terminal 2 can be improved.

In this embodiment, the VSLAM technique for estimating the position and orientation of the field terminal 2 based on the information acquired by the camera 8 and the three-dimensional measurement sensor 24 is illustrated, but other embodiments may be used. .. For example, VSLAM technology that estimates the position and orientation of the field terminal 2 based on the information acquired by the RGB camera, the fisheye camera, the gyro sensor, and the infrared sensor may be used. This VSLAM technology can measure the position indoors, and it is the most accurate 3D coordinate among the position information measurement technologies that can be used indoors such as beacon or pedestrian autonomous navigation (PDR). Can be calculated.

The position and posture of the field terminal 2 are recorded in association with the time at the time of calculation or update, respectively.

The reference acquisition unit 30 acquires a reference point in the three-dimensional space when the marker 5 provided in advance corresponding to the imaging location (target device 4) is photographed by the camera 8. The position of the marker 5 may be used as a reference point, or a predetermined position away from the marker 5 may be used as a reference point. In this way, the acquisition accuracy can be improved by a simple operation of providing the marker 5 at the work site and a simple process of photographing the marker 5.

The area identification unit 31 identifies whether or not the site terminal 2 is in the permitted area 16 where the acquisition of images is permitted, based on the current position of the site terminal 2. The permission area 16 associated with the site terminal 2 is set by accessing the management computer 3 in advance before starting work.

The image acquisition control unit 32 controls that the image acquisition is permitted when the site terminal 2 is in the shooting permission area 16 and the image acquisition is not permitted when the site terminal 2 is not in the shooting permission area 16. That is, at the work site, control is performed to limit the photography function. The photography function refers to a function of storing an image taken by the camera 8 of the field terminal 2 in the image storage unit 27.

For example, when the field terminal 2 is in the shooting permission area 16 and the camera 8 of the field terminal 2 takes a picture, the image is displayed on the display 6 and is not stored in the image storage unit 27. The image exemplifies a moving image. Here, when the worker W touches the photography button displayed on the display 6, the image taken by the camera 8 at that time is stored in the image storage unit 27. The stored image is an example of a still image.

On the other hand, if the worker W performs a photography operation when the on-site terminal 2 is not in the photography permission area 16, a photography prohibition message such as "shooting is prohibited in this area" is displayed on the display 6. It is displayed and the image is not stored in the image storage unit 27.

In this way, it is possible to limit the acquisition of an image of the target device 4 that is subject to shooting restrictions. Therefore, it is possible to prevent the acquisition of unnecessary images and reduce the number of images that must be determined by the management computer 3.

In the present embodiment, the term "image acquisition" includes the meaning of "image storage" or "image transfer". In addition, the term "not permitting the acquisition of an image" includes the meaning of "prohibiting the storage (memory) of an image" or "prohibiting the shooting of an image". Further, the image acquisition may be performed by the field terminal 2 or may be performed by the management computer 3.

It should be noted that the availability of the photography function may be switched depending on whether or not the photography button is displayed. For example, when the on-site terminal 2 is in the photography permission area 16, the photography button may be displayed on the display 6, and when the on-site terminal 2 is not in the photography permission area 16, the photography button may not be displayed on the display 6. ..

The position information acquisition unit 33 acquires position information that can specify the position when the image is taken by the camera 8. For example, the position information acquisition unit 33 can acquire the position at the time of taking an image based on the environment map created by the position / orientation estimation unit 29 using the VSLAM technology. Further, the position information acquisition unit 33 can acquire the position at the time of shooting the image based on the reference point acquired by the reference acquisition unit 30.

The direction information acquisition unit 34 acquires direction information capable of specifying the direction when the image is taken by the camera 8. For example, the direction information acquisition unit 34 can acquire the direction at the time of taking an image based on the environment map created by the position / orientation estimation unit 29 using the VSLAM technology. Further, the direction information acquisition unit 34 can acquire the direction at the time of shooting the image based on the reference point acquired by the reference acquisition unit 30. In addition, the direction information includes information such as a viewpoint (angle).

If there is an error between the position information or direction information based on the environmental map and the position information or direction information based on the reference point, the position information or direction information based on the reference point may be preferentially applied. In this way, the worker W can correct the information by photographing the marker 5 at the work site. Since the VSLAM technique is a method of calculating the relative movement amount, an error may be accumulated by using it for a long time. Therefore, by reading the marker 5 which is the reference point by the field terminal 2, the position information in the area 16 can be accurately grasped.

The angle of view information acquisition unit 35 acquires the angle of view information capable of specifying the angle of view when the image is taken by the camera 8. The angle of view information is acquired based on the settings of the camera 8. The angle of view information also includes information such as the enlargement ratio (scale).

The shooting information recording unit 36 records the shooting information including the position information, the direction information, and the angle of view information in association with the image. The shooting information may be recorded as an attribute of the image file, or may be recorded in a recording file different from the image file. The worker W can set in advance which method to record.

The management computer 3 can accurately select the image in which the target device 4 subject to the shooting restriction is captured, based on the shooting information recorded in association with the image. Therefore, it is possible to prevent the image from being leaked to the outside.

The identifier generation unit 37 generates an identifier based on the shooting information. The worker W can set in advance whether or not to record the shooting information as an identifier. In the case of the setting in which the identifier is generated, the shooting information recording unit 36 records the identifier in association with the image. The shooting information recording unit 36 may record the image in a manner in which the identifier is included in the image.

As shown in FIG. 2, the management computer 3 includes a target information database 38, a display 12, an operation input unit 39, a communication unit 40, an image storage unit 41, and a management control unit 42.

The target information database 38 is a collection of information stored in a memory or HDD and organized so that it can be searched or stored. The target information database 38 stores in advance target information that can specify the three-dimensional position of the target device 4 that is subject to shooting restrictions. In addition, the design information of the plant that is the work site is stored. For example, 3D-CAD data related to design information is stored in advance. The 3D-CAD data includes information regarding the position (coordinates) where the target device 4 is arranged. It also contains information about the arrangement of rooms 15 in the building 14.

The target device 4 subject to shooting restrictions may be set using 3D-CAD data, or the data may be displayed on the display 12 of the management computer 3 and individually selected or specified. Further, it may be automatically selected or specified based on the information such as the name or model of the target device 4 possessed by the 3D-CAD data.

The display 12 is a display device that outputs an image determination result (see FIG. 9). That is, the management computer 3 controls the image displayed on the display 12. The management computer 3 may control the image displayed on the display of another computer connected via the network.

In this embodiment, the display 12 is illustrated as the display device, but other modes may be used. For example, information may be displayed using a projector. Further, a printer that prints information on a paper medium may be used instead of the display 12. That is, a projector or a printer may be included as an object controlled by the management computer 3.

The operation input unit 39 is composed of an input device such as a mouse or a keyboard used for inputting predetermined information according to the operation of the administrator M. The input operation of the present embodiment includes a click operation using a mouse or a touch operation using a touch panel. That is, predetermined information is input to the management computer 3 according to the operation of these input devices.

The communication unit 40 communicates with the field terminal 2 via a communication line. The communication unit 40 is mounted on the wireless communication device 13. Further, the communication unit 40 may be mounted on a predetermined network device, for example, a wireless LAN access point or an antenna. The communication unit 40 may communicate with the field terminal 2 via a WAN (Wide Area Network), an Internet line, or a mobile communication network. In addition, communication may be performed by connecting to the field terminal 2 with a LAN cable or a USB cable.

The image storage unit 41 stores the image file received from the field terminal 2. The image stored in the image storage unit 41 may be a moving image or a still image. Further, the recording file associated with the image file may be stored together with the image file. Further, predetermined data related to the image file may be stored.

The management control unit 42 comprehensively controls the management computer 3. The management control unit 42 includes an image determination unit 43, a determination information recording unit 44, a determination result display unit 45, and a target information setting unit 46. These are realized by executing the program stored in the memory or the HDD by the CPU.

The image determination unit 43 determines whether or not the target device 4 (target portion) is reflected in the image based on the shooting information associated with the image file and the target information stored in the target information database 38. .. In this way, the captured range is specified by the position, direction, and angle of view of the image taken by the camera 8, and whether or not the target portion is reflected in the image in relation to the three-dimensional position of the target device 4. Can be determined. Further, since the determination can be made based on the three-dimensional positional relationship between the camera 8 and the target portion, the images can be accurately selected. The judgment can be made with higher accuracy than the case where the target device shown in the image is identified by human or artificial intelligence.

The field terminal 2 acquires position information and direction information based on the coordinates of the reference point when the image is taken. Here, the shooting range is calculated based on the position information, the direction information, and the angle of view. For example, the position (coordinates) of the site terminal 2 and the position (coordinates) of the target device 4 in the room 15 are calculated by referring to the 3D-CAD data of the target information database 38. After calculating the position of the site terminal 2, the shooting direction of the site terminal 2 is calculated at that point, and the shooting range is calculated in consideration of the angle of view of the camera 8 in that direction. It is determined whether or not the target device 4 is included in this photographing range. Based on this determination result, the possibility of restriction may be displayed for each image file, or the storage location of the image file may be divided.

The determination information recording unit 44 records the determination information that can identify the determination result by the image determination unit 43 in association with the image file. In the present embodiment, as the determination information, information indicating whether the file is a restricted file whose disclosure to the outside is restricted or an unrestricted file whose disclosure to the outside is not restricted is recorded (see FIG. 8).

In this way, it is possible to specify from the determination information whether or not the target device 4 (target portion) subject to the shooting restriction is captured in the image file. For example, even when the image file is referred to by a computer other than the management computer 3, it is possible to grasp whether or not the target device 4 (target portion) is included in the image file based on the determination information.

The determination result display unit 45 displays the determination result by the image determination unit 43 on the display 12 so as to be identifiable. For example, as shown in FIG. 9, the image determination result screen 47 is displayed on the display 12 of the management computer 3. The restricted file display area 48 and the non-restricted file display area 49 are provided side by side on the image determination result screen 47.

In the restricted file display area 48, the file name of the restricted file 20 which is the image file 20 including the image of the target device 4 and whose disclosure to the outside is restricted is displayed. In the non-restricted file display area 49, the file name of the non-restricted file which is the image file 20 including the image in which the target device 4 is not shown and whose disclosure to the outside is not restricted is displayed.

By doing so, the restricted file display area 48 and the unrestricted file display area 49 are separated and displayed side by side at the same time, so that the administrator M of the image file 20 can see the image file in which the target device 4 is shown. The 20 and the image file 20 in which the target device 4 is not shown can be easily identified.

When the image file 20 is displayed by an icon, the display color of the image file 20 may be different between the restricted file and the non-restricted file. Further, the extension of the image file 20 may be different between the restricted file and the non-restricted file.

The administrator M can display the image included in the image file by clicking the file name with the mouse cursor 50. Then, the administrator M can visually confirm whether or not the target device 4 is captured.

The administrator M can drag the restricted file from the restricted file display area 48 to the unrestricted file display area 49. The determination information of the restricted file dragged to the unrestricted file display area 49 is rewritten as an unrestricted file. In addition, the administrator M can drag the unrestricted file from the unrestricted file display area 49 to the restricted file display area 48. The determination information of the non-restricted file dragged to the restricted file display area 48 is rewritten as a restricted file. If there is a recording file 21 corresponding to the image file 20, the recording file 21 is moved along with the movement of the image file 20.

As shown in FIG. 2, the target information setting unit 46 sets the target information database 38 to store the target information. For example, the reference point management table for managing the reference points, the target device management table for managing the target device 4, and the terminal management table for managing the site terminal 2 are set to store various information.

As shown in FIG. 5, 3D position information, a marker ID, and a target device ID are registered in the reference point management table in association with the reference point ID.

The reference point ID is identification information that can individually identify a plurality of reference points. One reference point ID is assigned to one reference point.

The 3D position information registered in the reference point management table is information on three-dimensional coordinates that specify the position where the reference point is provided. The coordinate information includes information for designating positions in the horizontal direction (X-axis, Y-axis) and the vertical direction (Z-axis). Further, the 3D position information includes information regarding the posture or orientation of the reference point.

In the present embodiment, the coordinates are set with one marker 5 provided at the entrance / exit 17 of the building 14 as a reference point (origin, starting point). The coordinates do not have to use the marker 5 as a reference point, and a predetermined position of the building 14 may be used as a reference point for the coordinates. Further, the 3D position information may be configured based on the three-dimensional CAD created at the design stage of the building 14.

The marker ID registered in the reference point management table is identification information that can individually identify a plurality of markers 5. One marker ID is assigned to one marker 5. Further, in the reference point management table, one marker ID is assigned to one reference point.

The target device ID is identification information that can individually identify a plurality of target devices 4. One target device ID is assigned to one target device 4.

In the reference point management table, a plurality of target device IDs may be registered corresponding to one reference point ID. For example, the target device IDs of all the target devices 4 arranged inside the building 14 are registered in association with the marker IDs of the markers 5 at the entrance / exit 17 of the building 14. Further, the target device IDs of all the target devices 4 arranged inside the room 15 are registered in association with the marker IDs of the markers 5 of the doorway 18 of the room 15.

As shown in FIG. 6, 3D position information is registered in the target device management table in association with the target device ID.

The 3D position information registered in the target device management table is position information indicating coordinates that specify the position of the target device 4. It includes information for designating the positions in the horizontal direction (X-axis, Y-axis) and the vertical direction (Z-axis). Further, the 3D position information includes information regarding the posture or orientation of the target device 4.

In this way, in the target information database 38, position information relating to the marker 5 provided in advance in the building 14 at the work site (shooting location) is registered in association with the target device 4. Further, the position information of the target device 4 is also registered. In this way, the target information regarding the target device 4 can be centrally managed by using the target information database 38.

As shown in FIG. 7, a user ID and a permission area are registered in the terminal management table in association with the terminal ID.

The terminal ID is identification information that can individually identify a plurality of field terminals 2. In the target device management table, the terminal ID of the field terminal 2 that is permitted to provide the support information of the corresponding target device 4 is registered in association with the target device ID.

The user ID is identification information that can individually identify a plurality of workers W. In the target device management table, the user ID of the worker W who is permitted to use the corresponding field terminal 2 is registered in association with the terminal ID. The user ID may be used when logging in to the site terminal 2, or may be registered in the ID card possessed by each worker W.

In the item of permitted area, the name of the area where the acquisition of the image is permitted by the site terminal 2 corresponding to the terminal ID is registered. The manager M can centrally manage each site terminal 2 by setting the items of the permitted area based on the information such as the work plan. The on-site terminal 2 sets a permitted area by accessing the management computer 3 in advance before starting work. The work time corresponding to the permitted area may be set. In this way, it is possible to prevent the worker W from acquiring an image at an unnecessary place unrelated to the work. Therefore, it is possible to efficiently manage the taking-out of images.

The management computer 3 can search various data registered in the target information database 38 based on the reference point ID, the target device ID, the terminal ID, and the marker ID.

As shown in FIG. 2, the image determination unit 43 includes a machine learning unit 51. The machine learning unit 51 can discriminate between an image in which the target device 4 is captured and an image in which the target device 4 is not captured by machine learning. That is, in the present embodiment, not only the determination of the image based on the shooting information but also the image can be analyzed by using the machine learning unit 51 by superimposing on this determination. In this way, not only the shooting information at the time of shooting the image by the camera 8 but also whether or not the target device 4 is reflected in the image is supplementarily identified by machine learning, so that the determination can be made with higher accuracy. Can be done. The worker W can set in advance whether or not the machine learning unit 51 analyzes the image.

The machine learning unit 51 is equipped with artificial intelligence (AI). For the analysis using the computer of the present embodiment, an analysis technique based on learning of artificial intelligence can be used. For example, a learning model generated by machine learning by a neural network, another learning model generated by machine learning, a deep learning algorithm, a mathematical algorithm such as regression analysis can be used. Further, the form of machine learning includes forms such as clustering and deep learning.

The image identification system 1 of the present embodiment includes a computer provided with artificial intelligence that performs machine learning. For example, the image identification system 1 may be configured by one computer provided with a neural network, or the image identification system 1 may be configured by a plurality of computers provided with a neural network.

Here, the neural network is a mathematical model that expresses the characteristics of brain function by computer simulation. For example, we show a model in which artificial neurons (nodes) that form a network by synaptic connection change the synaptic connection strength by learning and have problem-solving ability. In addition, neural networks acquire problem-solving abilities through deep learning.

For example, a neural network is provided with an intermediate layer having six layers. Each layer of this intermediate layer is composed of 300 units. In addition, by having a multi-layer neural network learn in advance using training data, it is possible to automatically extract features in a pattern of changes in the state of a circuit or system. In the multi-layer neural network, any number of intermediate layers, the number of units, the learning rate, the number of learnings, and the activation function can be set on the user interface.

Note that deep reinforcement learning may be used in which a reward function is set for various information to be learned and the one having the highest value is extracted based on this reward function.

For example, a CNN (Convolution Neural Network) with a proven track record in image recognition is used. In this CNN, the intermediate layer is composed of a convolution layer and a pooling layer. The convolution layer acquires a feature map by filtering nearby nodes in the previous layer. The pooling layer further reduces the feature map output from the convolutional layer into a new feature map. At this time, it is possible to absorb some deviation of the image depending on which value of the region of interest is used.

The convolution layer extracts the local features of the image, and the pooling layer performs a process of summarizing the local features. In these processes, the image is reduced while maintaining the characteristics of the input image. That is, in CNN, the amount of information contained in an image can be significantly compressed (abstracted). Then, using the abstracted image stored in the neural network, the input image can be recognized and the image can be classified.

There are various methods for deep learning such as autoencoder, RNN (Recurrent Neural Network), RSTM (Long Short-Term Memory), and GAN (Generative Adversarial Network). These methods may be applied to the deep learning of the present embodiment.

In the present embodiment, the machine learning unit 51 is made to learn a large amount of images in which various patterns of the target device 4 are captured and images in which the target device 4 is not captured in advance. These learning images may be images actually taken by the camera 8 or images synthesized by computer graphics (CG). In this way, the machine learning unit 51 is made to learn a large number of images in advance until the shape of the target device 4 and the like can be identified.

The machine learning method of the present embodiment can be roughly divided into two processes. First, the first process of generating a learning image is performed. In this first process, a learning image is acquired, and the image of the target device 4 included in the learning image is classified according to the feature. Then, one learning image is adjusted so that one or more images of different types are included. Next, the second process of creating the learning model is performed. In this second process, the learning image is trained by the neural network to create an image learning model. This learning model is set for image determination.

Next, the processing executed by the image identification system 1 will be described with reference to the flowcharts of FIGS. 10 to 15. The block diagram shown in FIG. 2 will be referred to as appropriate.

The management computer 3 executes the management control process shown in FIG. The field terminal 2 executes the terminal control process shown in FIG. These processes are processes that are repeated at regular intervals. By repeating these processes, the image identification method 1 is executed in the image identification system 1.

The management control process executed by the management computer 3 will be described with reference to the flowchart of FIG.

First, in step S11, the management control unit 42 executes the main control process. In this main control process, a process of controlling various devices such as the display 12 of the management computer 3 is performed. In addition, when there is an access from the field terminal 2, predetermined information is transmitted and received. For example, when an image file is transmitted from the field terminal 2, a process of storing the image file in the image storage unit 41 is performed.

In the next step S12, the management control unit 42 executes the operation acceptance process. In this operation acceptance process, the operation input unit 39 performs a process of accepting a predetermined input operation.

In the next step S13, the target information setting unit 46 executes the pre-registration process. In this pre-registration process, a process of storing the target information in the target information database 38 is performed. For example, information about the reference point is registered in the reference point management table (see FIG. 5). In addition, information about the target device 4 is registered in the target device management table (see FIG. 6). Further, information about the field terminal 2 is registered in the terminal management table (see FIG. 7).

In the next step S14, the image determination unit 43 executes an image determination process (see FIG. 11) described later. In this image determination process, it is determined whether or not the target device 4 (target portion) is reflected in the image received from the field terminal 2.

In the next step S15, the determination result display unit 45 executes the image determination result display process. In this image determination result display process, a process (see FIG. 9) is performed in which the determination result by the image determination unit 43 is identifiablely displayed on the display 12. Then, the management control process is terminated.

Next, the terminal control process executed by the field terminal 2 will be described with reference to the flowchart of FIG.

First, in step S31, the terminal control unit 28 executes the main control process. In this main control process, a process of controlling various devices such as a display 6 mounted on the field terminal 2 is performed.

In the next step S32, the terminal control unit 28 executes the operation acceptance process. In this operation acceptance process, the operation input unit 25 performs a process of accepting a predetermined input operation.

In the next step S33, the terminal control unit 28 executes the database access process. In this database access process, the target information database 38 of the management computer 3 is accessed, and various data download processes are performed. For example, the data of the reference point management table is downloaded.

In the next step S34, the terminal control unit 28 executes a marker recognition process (see FIG. 13) described later. In this marker recognition process, the marker 5 is recognized based on the captured image captured by the camera 8.

In the next step S35, the terminal control unit 28 executes a self-position estimation process (see FIG. 14), which will be described later. In this self-position estimation process, the position and orientation of the field terminal 2 are estimated based on the captured image, the acceleration information, and the angular velocity information. When this self-position estimation process is executed, the camera 8 is activated and shooting is started.

In the next step S36, the terminal control unit 28 executes a shooting control process (see FIG. 15) described later. In this shooting control process, shooting is controlled by the camera 8.

In the next step S37, the terminal control unit 28 executes the image transmission process. In this image transmission process, a process of transmitting an image file stored in the image storage unit 27 to the management computer 3 is performed. The process of transmitting the image file may be performed based on the operation of the worker W, or may be automatically performed at regular intervals. Then, the terminal control process is terminated.

Next, the marker recognition process executed by the field terminal 2 will be described with reference to the flowchart of FIG.

First, in step S41, the terminal control unit 28 acquires a captured image captured by the camera 8.

In the next step S42, the terminal control unit 28 determines whether or not the marker 5 is included in the acquired captured image. Here, if the marker 5 is not shown in the captured image (NO in step S42), the marker recognition process ends. On the other hand, if the marker 5 appears in the captured image (YES in step S42), the process proceeds to step S43.

In the next step S43, the terminal control unit 28 reads the marker ID included in the figure of the marker 5.

In the next step S44, the reference acquisition unit 30 obtains 3D position information of the reference point ID corresponding to the read marker ID based on the data of the reference point management table downloaded in advance from the target information database 38 of the management computer 3. Identify. Based on the 3D position information of this reference point, the position information of the site terminal 2 is acquired or corrected. Then, the marker recognition process is terminated.

Next, the self-position estimation process executed by the field terminal 2 will be described with reference to the flowchart of FIG.

First, in step S51, the position / orientation estimation unit 29 performs coordinate confirmation processing based on the setting of the reference point that is the origin of the coordinates. For example, a reference point is set with one marker 5 provided at the entrance / exit 17 of the building 14 as the origin. Further, the reference point may be set with one marker 5 provided at the entrance / exit 18 of the room 15 as the origin. In addition, until the photographed image of the marker 5 is acquired, the display 6 may indicate that the image of the marker 5 is urged to be photographed.

In the next step S52, the terminal control unit 28 acquires a captured image captured by the camera 8.

In the next step S53, the position / orientation estimation unit 29 acquires acceleration information indicating the value of the acceleration detected by the acceleration sensor 22.

In the next step S54, the position / orientation estimation unit 29 acquires the angular velocity information indicating the value of the angular velocity detected by the angular velocity sensor 23.

In the next step S55, the position / orientation estimation unit 29 measures the three-dimensional shape of the object around the site terminal 2 based on the captured image of the camera 8. Further, the three-dimensional shape of the object around the field terminal 2 may be acquired based on the measurement of the three-dimensional measurement sensor 24.

In the next step S56, the position / orientation estimation unit 29 creates an environment map including information on the surrounding environment of the site terminal 2.

In the next step S57, the position / orientation estimation unit 29 estimates the position and orientation of the site terminal 2 based on the captured images continuously captured when the site terminal 2 moves from the reference point. By recording the position and posture of the on-site terminal 2 during this movement together with the time information, the movement route 19 (see FIG. 3) of the on-site terminal 2 is recorded. Further, the movement route inside the room 15 may be recorded.

In the next step S58, the position / orientation estimation unit 29 estimates the current position and orientation of the field terminal 2. Then, the self-position estimation process is completed.

Next, the photographing control process executed by the field terminal 2 will be described with reference to the flowchart of FIG.

First, in step S61, the terminal control unit 28 determines whether or not the camera 8 has already been activated. Here, if the camera 8 is not activated (NO in step S61), the shooting control process ends. On the other hand, when the camera 8 is activated (YES in step S61), the process proceeds to step S62.

In the next step S62, the terminal control unit 28 displays the image captured by the camera 8 on the display 6.

In the next step S63, the image acquisition control unit 32 determines whether or not the image acquisition operation, which is an operation of saving the image captured by the camera 8 in the image storage unit 27, has been accepted. The shooting operation is, for example, a touch operation of a photo shooting button. Here, if the shooting operation is not accepted (NO in step S63), the shooting control process ends. On the other hand, when the shooting operation is accepted (YES in step S63), the process proceeds to step S64.

In the next step S64, the area identification unit 31 determines whether or not the site terminal 2 is in the permitted area where image acquisition is permitted based on the current position of the site terminal 2 estimated by the position / orientation estimation unit 29. To judge. Here, if it is not within the permitted area (NO in step S64), the process proceeds to step S65. On the other hand, if it is within the permitted area (YES in step S64), the process proceeds to step S66.

In step S65, the image acquisition control unit 32 displays a shooting prohibition message on the display 6. Then, the shooting control process is terminated.

In step S66, the position information acquisition unit 33 acquires position information capable of specifying the position when the image is taken by the camera 8.

In the next step S67, the direction information acquisition unit 34 acquires direction information capable of specifying the direction when the image is taken by the camera 8.

In the next step S68, the angle of view information acquisition unit 35 acquires the angle of view information capable of specifying the angle of view when the image is taken by the camera 8.

In the next step S69, the shooting information recording unit 36 records the shooting information including the position information, the direction information, and the angle of view information in association with the image. The identifier generation unit 37 may generate an identifier based on the shooting information and record the identifier in association with the image. Here, the image file is stored in the image storage unit 27. Then, the shooting control process is terminated.

Next, the image determination process executed by the management computer 3 will be described with reference to the flowchart of FIG.

First, in step S21, the management control unit 42 determines whether or not there is an undetermined image file in the image storage unit 41. Here, if there is no undetermined image file (NO in step S21), the image determination process ends. On the other hand, if there is an undetermined image file (YES in step S21), the process proceeds to step S22.

In the next step S22, the image determination unit 43 identifies the imaging position of the image by the camera 8 based on the position information included in the imaging information associated with the undetermined image file.

In the next step S23, the image determination unit 43 identifies the shooting direction of the image by the camera 8 based on the direction information included in the shooting information associated with the undetermined image file.

In the next step S24, the image determination unit 43 specifies the angle of view at the time of photographing the image by the camera 8 based on the angle of view information included in the shooting information associated with the undetermined image file.

In the next step S25, the image determination unit 43 captures the target device 4 (target portion) in the image based on the shooting information associated with the image file and the target information stored in the target information database 38. Judge whether or not. Here, if the target device 4 is shown in the image (YES in step S25), the process proceeds to step S26. On the other hand, if the target device 4 is not shown in the image (NO in step S25), the process proceeds to step S27.

In step S26, the image determination unit 43 sets the image file as the restriction file. Then, the process proceeds to step S28.

In step S27, the image determination unit 43 sets the image file as an unrestricted file.

In the next step S28, the determination information recording unit 44 records the determination information indicating the determination result in association with the image file.

In the next step S29, the image determination unit 43 determines whether or not the machine learning unit 51 is set to determine the image. Here, if the determination by the machine learning unit 51 is not being set (NO in step S29), the image determination process ends. On the other hand, if the determination by the machine learning unit 51 is being set (YES in step S29), the process proceeds to step S30.

In the next step S30, the image determination unit 43 performs a machine learning determination process for determining an image by the machine learning unit 51. In this machine learning determination process, the machine learning unit 51 analyzes whether or not the target device 4 is shown in the image. If the determination result by the machine learning unit 51 and the determination result in step S25 described above match, the image determination process is terminated. On the other hand, if they do not match, the determination result by the machine learning unit 51 may be prioritized to update the determination information, or a message may be displayed so that the administrator M can visually confirm. Then, the image determination process is terminated.

(Second Embodiment)
Next, the image identification system 1A and the image identification method of the second embodiment will be described with reference to FIGS. 16 to 18. The same components as those shown in the above-described embodiment are designated by the same reference numerals, and duplicate description will be omitted.

As shown in FIG. 16, in the image identification system 1A of the second embodiment, unlike the case of the first embodiment (see FIG. 2), the image storage unit 27, the photographing information recording unit 36, and the identifier generation unit are stored in the field terminal 2A. 37 is not provided. Instead, the management computer 3A is provided with a shooting information recording unit 52 and an identifier generation unit 53.

In the second embodiment, the field terminal 2A is provided in a communication environment capable of constantly connecting to the management computer 3A. Then, when the worker W performs a shooting operation using the on-site terminal 2A, the acquired image is transferred to the management computer 3A in real time. The shooting information including the position information, the direction information, and the angle of view information at the time of shooting is also transferred at the same time. Then, it is stored in the image storage unit 41 of the management computer 3A. As described above, in the second embodiment, since the image is not stored in the field terminal 2A, the image does not have to be taken out even when the field terminal 2A is taken out to the outside.

When the image is received from the site terminal 2A, the photographing information recording unit 52 of the management computer 3A records the photographing information including the position information, the direction information, and the angle of view information in association with the image. The shooting information may be recorded as an attribute of the image file, or may be recorded in a recording file different from the image file. The administrator M can set in advance which method to record.

The identifier generation unit 53 of the management computer 3A generates an identifier based on the received shooting information. The administrator M can set in advance whether or not to record the shooting information as an identifier. In the case of the setting in which the identifier is generated, the shooting information recording unit 52 records the identifier in association with the image. The shooting information recording unit 52 may record the image in a manner in which the identifier is included in the image.

Next, the photographing control process executed by the field terminal 2A will be described with reference to the flowchart of FIG. In the shooting control process of the second embodiment, only step S69A is different from the shooting control process of the first embodiment (see FIG. 15), and steps S61 to S68 are the same as the shooting control process of the first embodiment. The same is true.

As shown in FIG. 18, in step S69A following step S68, the terminal control unit 28 manages the image taken by the camera 8 and the shooting information including the position information, the direction information, and the angle of view information of the management computer 3A. Send to. Then, the shooting control process is terminated.

Next, the image determination process executed by the management computer 3A will be described with reference to the flowchart of FIG. In the image determination process of the second embodiment, only steps S21A and S21B are different from the image determination process of the first embodiment (see FIG. 11), and steps S22 to S30 are images of the first embodiment. It is the same as the judgment process.

As shown in FIG. 17, first, in step S21A, the management control unit 42 determines whether or not an image has been received from the field terminal 2A. Here, if the image has not been received (NO in step S21A), the image determination process ends. On the other hand, when the image is received (YES in step S21A), the process proceeds to step S21B.

In the next step S21B, the shooting information recording unit 52 records the shooting information received from the field terminal 2A together with the image in association with the image. The identifier generation unit 53 may generate an identifier based on the shooting information and record the identifier in association with the image. Here, the image file is stored in the image storage unit 41. Then, the process proceeds to step S22.

Although the image identification system and the image identification method according to the present embodiment have been described based on the first to second embodiments, the configuration applied in any one embodiment may be applied to other embodiments. Alternatively, the configurations applied in each embodiment may be combined.

Although the flowchart of the present embodiment illustrates a mode in which each step is executed in series, the context of each step is not necessarily fixed, and even if the context of some steps is exchanged. good. Also, some steps may be executed in parallel with other steps.

The system of this embodiment includes a control device in which a dedicated chip, a controller such as an FPGA (Field Programmable Gate Array), a GPU (Graphics Processing Unit), or a CPU (Central Processing Unit) is highly integrated, and a ROM (Read Only). Storage devices such as Memory) or RAM (Random Access Memory), external storage devices such as HDD (Hard Disk Drive) or SSD (Solid State Drive), display devices such as displays, and input devices such as mice or keyboards. , With a communication interface. This system can be realized by a hardware configuration using a normal computer.

The program executed by the system of the present embodiment is provided by incorporating it into a ROM or the like in advance. Alternatively, the program may be a computer-readable, non-transient storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD) that is a file in an installable or executable format. It may be stored and provided in.

Further, the program executed by this system may be stored on a computer connected to a network such as the Internet, and may be downloaded and provided via the network. The system can also be configured by connecting separate modules that independently perform the functions of the components to each other via a network or a dedicated line, and combining them.

In this embodiment, the location information of the field terminal 2 is acquired by using the VSLAM technology, but other embodiments may be used. For example, the position information may be acquired using GPS. Location information may be acquired using wireless communication such as Wi-Fi (registered trademark). Position information may be acquired by PDR using an acceleration sensor or the like.

The field terminal 2 may be provided with a camera with a fisheye lens. For example, two cameras with fisheye lenses may be used to simultaneously capture a 360-degree panoramic image of the operator W in all directions, up, down, left, and right. Then, it may be determined whether or not the target device 4 is shown in the spherical image. In that case, the determination may be made based only on the position information without referring to the shooting direction and the angle of view information.

The field terminal 2 of the present embodiment may have a size and weight that can be carried by an operator. For example, at the work site, the site terminal 2 may be fixedly installed and used using a fixing device such as a tripod.

According to at least one embodiment described above, by providing a shooting information recording unit that records shooting information including position information, direction information, and angle of view information in association with an image, it is subject to shooting restrictions. It is possible to accurately select an image in which a target portion is captured and prevent the image from being leaked to the outside.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, changes, and combinations can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, as well as in the scope of the invention described in the claims and the equivalent scope thereof.

1 (1A) ... image identification system, 2 (2A) ... field terminal, 3 (3A) ... management computer, 4 ... target device, 5 ... marker, 6 ... display, 7 ... smartphone, 8 ... camera, 9 ... transmissive type Head mount display, 10 ... wearable computer, 11 ... computer body, 12 ... display, 13 ... wireless communication device, 14 ... building, 15 ... room, 16 ... permitted area, 17, 18 ... doorway, 19 ... movement route, 20 ... Image file, 21 ... Recording file, 22 ... Acceleration sensor, 23 ... Angle speed sensor, 24 ... Three-dimensional measurement sensor, 25 ... Operation input unit, 26 ... Communication unit, 27 ... Image storage unit, 28 ... Terminal control unit, 29 ... Position / orientation estimation unit, 30 ... reference acquisition unit, 31 ... area identification unit, 32 ... image acquisition control unit, 33 ... position information acquisition unit, 34 ... direction information acquisition unit, 35 ... angle of view information acquisition unit, 36 ... shooting information Recording unit, 37 ... Identifier generation unit, 38 ... Target information database, 39 ... Operation input unit, 40 ... Communication unit, 41 ... Image storage unit, 42 ... Management control unit, 43 ... Image judgment unit, 44 ... Judgment information recording unit , 45 ... Judgment result display unit, 46 ... Target information setting unit, 47 ... Image judgment result screen, 48 ... Restricted file display area, 49 ... Unrestricted file display area, 50 ... Mouse cursor, 51 ... Machine learning unit, 52 ... Shooting information recording unit, 53 ... identifier generation unit, M ... administrator, W ... worker.

The image identification system according to the embodiment of the present invention is mounted on a movable terminal, and is a camera that captures an image and a position information acquisition unit that acquires position information that can identify the position when the image is captured by the camera. And the direction information acquisition unit that acquires the direction information that can specify the direction when the image is taken by the camera, and the image angle that acquires the image angle information that can specify the image angle when the image is taken by the camera. An information acquisition unit, a shooting information recording unit that records shooting information including the position information, the direction information, and the angle of view information in association with the image, and a target portion subject to shooting restriction are arranged. It is determined whether or not the target portion is reflected in the image based on the target information database in which the target information including the three-dimensional position information related to the design information of the shooting location is stored and the shooting information and the target information. An image determination unit is provided.

The position / attitude estimation unit 29 controls to estimate the position and attitude of the site terminal 2 based on the captured image, the acceleration information, and the angular velocity information. That is, the position and posture of the field terminal 2 are estimated based on the information obtained by the device mounted on the field terminal 2. The position and orientation estimation unit 29 performs creation of environmental map including the position and orientation estimation simultaneously with site information of the ambient environment of the terminal 2 of the field terminal 2. That is, the position / orientation estimation unit 29 uses the VSLAM technology. By doing so, it is possible to improve the acquisition accuracy of the position and direction at the time of taking an image.

In the next step S44, the reference acquisition unit 30 obtains 3D position information of the reference point ID corresponding to the read marker ID based on the data of the reference point management table downloaded in advance from the target information database 38 of the management computer 3. Identify. Based on the 3D positional information of the reference point, to acquire or correction of the positional information field terminal 2. Then, the marker recognition process is terminated.

The image identification system according to the embodiment of the present invention is mounted on a movable terminal, and is a position information acquisition unit that acquires a camera that captures an image and position information that can identify the position at the time of capturing the image by the camera. And the direction information acquisition unit that acquires the direction information that can specify the direction when the image is taken by the camera, and the image angle that acquires the image angle information that can specify the image angle when the image is taken by the camera. An information acquisition unit, a shooting information recording unit that records shooting information including the position information, the direction information, and the angle of view information in association with the image, and a target portion subject to shooting restriction are arranged. It is determined whether or not the target portion is reflected in the image based on the target information database in which the target information including the three-dimensional position information related to the design information of the shooting location is stored and the shooting information and the target information. an image determination section that, the determination result display unit for displaying the identifiably displays the determination result by the image determining unit, Bei example, said determination result display section, an image including the image in which the target portion is captured The restricted file display area in which the file is displayed and the non-restricted file display area in which the image file including the image in which the target portion is not displayed are displayed side by side .

In this embodiment, the position information of the field terminal 2 is acquired by using the VSLAM technology, but other embodiments may be used. For example, the position information may be acquired using GPS. Location information may be acquired using wireless communication such as Wi-Fi (registered trademark). Position information may be acquired by PDR using an acceleration sensor or the like.

Claims (13)

A camera mounted on a mobile terminal that captures images,
A position information acquisition unit that acquires position information that can specify the position when the image is taken by the camera, and a position information acquisition unit.
A direction information acquisition unit that acquires direction information that can specify the direction when the image is taken by the camera, and a direction information acquisition unit.
An angle of view information acquisition unit that acquires angle of view information that can specify the angle of view when the image is taken by the camera, and an angle of view information acquisition unit.
A shooting information recording unit that records shooting information including the position information, the direction information, and the angle of view information in association with the image.
To prepare
Image identification system. A target information database that stores target information that can identify the three-dimensional position of the target part that is subject to shooting restrictions, and
An image determination unit that determines whether or not the target portion is reflected in the image based on the shooting information and the target information.
To prepare
The image identification system according to claim 1. It is provided with a judgment information recording unit that records judgment information that can identify the judgment result by the image judgment unit in association with the image.
The image identification system according to claim 2. A judgment result display unit for displaying the judgment result by the image judgment unit on the display in an identifiable manner is provided.
The determination result display unit displays a restricted file display area in which an image file including the image in which the target portion is shown is displayed and an unrestricted file display in which an image file including the image in which the target portion is not shown is displayed. Display side by side with the area,
The image identification system according to claim 2 or 3. The image determination unit includes a machine learning unit capable of discriminating between the image in which the target portion is captured and the image in which the target portion is not captured by machine learning.
The image identification system according to any one of claims 2 to 4. The shooting information recording unit records the shooting information as an attribute of an image file including the image.
The image identification system according to any one of claims 1 to 5. The shooting information recording unit records the shooting information in a recording file different from the image file containing the image.
The image identification system according to any one of claims 1 to 5. It is provided with an identifier generation unit that generates a predetermined identifier based on the shooting information.
The shooting information recording unit records the identifier.
The image identification system according to any one of claims 1 to 7. The shooting information recording unit records in such a manner that the identifier is included in the image.
The image identification system according to claim 8. An area identification unit that identifies whether or not the terminal is in a permitted area where acquisition of the image is permitted, and
An image acquisition control unit that permits acquisition of the image when the terminal is in the permitted area and does not permit acquisition of the image when the terminal is not in the permitted area.
To prepare
The image identification system according to any one of claims 1 to 9. It is provided with a position / orientation estimation unit that estimates the position and orientation of the terminal based on the information obtained by the device mounted on the terminal and creates an environment map including information on the surrounding environment of the terminal.
Based on the environment map, the position information acquisition unit acquires the position at the time of shooting the image, and the direction information acquisition unit acquires the direction at the time of shooting the image.
The image identification system according to any one of claims 1 to 10. It is equipped with a reference acquisition unit that acquires a reference point in a three-dimensional space by photographing a marker provided in advance corresponding to the shooting location with the camera.
Based on the reference point, the position information acquisition unit acquires the position at the time of shooting the image, and the direction information acquisition unit acquires the direction at the time of shooting the image.
The image identification system according to any one of claims 1 to 11. Steps to take an image with a camera mounted on a mobile terminal,
A step of acquiring position information capable of specifying the position of the image at the time of shooting by the camera, and
A step of acquiring direction information capable of specifying the direction when the image is taken by the camera, and
The step of acquiring the angle of view information that can specify the angle of view at the time of shooting the image by the camera, and
A step of recording shooting information including the position information, the direction information, and the angle of view information in association with the image.
including,
Image identification method.

Images