JP2022021009A - Site video management system and site video management method - Google Patents

Abstract

To provide a site video management technology capable of displaying video of an optional point on a movement path of a terminal in reproducing the video photographed by using a moving terminal.SOLUTION: A site video management system 1 includes a video acquisition part 24 for acquiring video including at least a moving image photographed by a camera 6 mounted on a movable terminal 2, a position information acquisition part 28 for acquiring position information of the terminal 2 with video photographing, a video recording part 26 for associating the video with the position information to record them, a map allocation part 39 for specifying a coordinate position of a map showing the layout of a photographing site on the basis of the position information and allocating the video to the specified coordinate position, and a video display control part 40 for performing control for receiving the selection of the coordinate position of the map and displaying the video allocated to the selected coordinate position.SELECTED DRAWING: Figure 2

Description

Application for application of Article 30, Paragraph 2 of the Patent Law Meeting name: Japan Society for Conservation 16th Academic Lecture, Date: July 24, 2019, Venue: Link Station Hall Aomori (Aomori City) 1-4-1, Tsutsumicho), Publication: Abstracts of the 16th Annual Meeting of the Conservation Society of Japan, Publisher: Japan Conservation Society, Publication date: July 24, 2019

Embodiments of the present invention relate to on-site video management technology.

Conventionally, in plants such as power plants, regular patrol inspections are carried out for construction progress management or abnormality detection. In this patrol inspection, the items to be confirmed on the recording paper are recorded, and the inspection points are recorded as photographs (still images). However, when a photograph is recorded of a place where there are many structures with similar shapes such as a plant, it may be difficult to identify the photographed location from the image after the patrol inspection. In addition, there may be a request to confirm what the surrounding environment of the target place was after the patrol inspection. In that case, the inspector needs to go to the site again to check and record.

Further, a technique for recording a cityscape by using a 360-degree image obtained by recording an image in all directions is known. For example, Street View provided by Google has an omnidirectional camera (360 degree camera) mounted on a car and links with GPS location information so that 360 degree images of selected points on the map can be viewed. ing.

https://www.google.com/streetview https://www.google.com/maps

In the conventional technology, a 360-degree image can be recorded in cooperation with the GPS position information, and an omnidirectional image of a selected point on the map can be confirmed. In addition, past images at the same point can be displayed so that the situation can be confirmed retroactively. However, Google Street View needs to be in an environment where GPS can be used at the time of shooting. In addition, the points where the images are recorded are at intervals of about 5 m to 10 m, and the images taken at the points between them cannot be viewed. Further, since the recorded video is a still image, it is not possible to grasp the change in the situation in a short time at the recorded point. Therefore, this technique cannot be used for the recording work of the above-mentioned patrol inspection of the plant.

The embodiment of the present invention has been made in consideration of such circumstances, and can display an image at an arbitrary point on the moving path of the terminal when playing back an image taken by using a moving terminal. The purpose is to provide on-site video management technology.

The on-site video management system according to the embodiment of the present invention has a video acquisition unit that acquires a video including at least a moving image taken by a camera mounted on a movable terminal, and position information of the terminal together with the shooting of the video. Based on the position information acquisition unit to be acquired, the video recording unit that records the image in association with the position information, and the position information, the coordinate position of the map showing the layout of the shooting site is specified and specified. It is provided with a map allocation unit that allocates the video to a coordinate position, and a video display control unit that accepts selection of the coordinate position of the map and controls to display the video assigned to the selected coordinate position. ..

According to an embodiment of the present invention, there is provided a field image management technique capable of displaying an image at an arbitrary point on a moving path of a terminal when reproducing an image taken by using a moving terminal.

A system configuration diagram showing a site video management system. A block diagram showing an on-site video management system. A conceptual diagram showing the on-site video management method. An image diagram showing the inspection site taken by a camera with a fisheye lens. A coordinate diagram showing the movement trajectory of the shooting terminal. An explanatory diagram showing a layout map of the inspection site. A screen view showing a viewer for viewing video. Explanatory drawing showing inspection information. A flowchart showing the terminal-side processing executed by the shooting terminal. A flowchart showing the frame rate adjustment process. A flowchart showing the management side processing executed by the management computer. An explanatory diagram showing the relationship between the frame, the position, and the posture. Explanatory drawing which shows the relationship between the frame of the modification 1 and the position, and the posture. Explanatory drawing which shows the relationship between the frame of the modification 2 and the position, and the posture.

Hereinafter, embodiments of the on-site video management system and the on-site video management method will be described in detail with reference to the drawings.

Reference numeral 1 in FIG. 1 is the on-site video management system of the present embodiment. The on-site video management system 1 includes a photographing terminal 2 worn by a worker W at the inspection site and a management computer 3 handled by an administrator M who manages the inspection site. The on-site video management system 1 may include other configurations. For example, the worker W may have a tablet terminal 4 or the like used for inspection work.

The inspection site (shooting site) of this embodiment includes a power plant, a chemical plant, a factory, and the like. These plants are equipped with a large number of equipment or structures to be inspected. Worker W goes to the inspection site, confirms the equipment or structure, makes a predetermined record, and shoots an image of the equipment or structure.

The device includes, for example, a valve, a motor, a switchboard, an electric component, a computer, and the like. Further, the structure includes, for example, a tank, a pipe, a support member, a skeleton constituting a building, a through hole provided in the skeleton, a room, a passage, a site, and the like. In the following description, these are collectively referred to as inspection objects.

Worker W wears a helmet 5 and performs inspection work. In this embodiment, the helmet 5 constitutes a part of the photographing terminal 2. A camera 6 having two image sensors with a fisheye lens is mounted on the crown of the helmet 5. The camera 6 can capture a wide range around the worker W.

The two image sensors are arranged so as to be back to back, and can simultaneously capture the scenery before and after. That is, it is possible to simultaneously shoot an omnidirectional image (omnidirectional image) which is a 360-degree panoramic image of the operator W in all directions, up, down, left, and right, using the camera 6.

In the present embodiment, the camera 6 having two image sensors is illustrated, but a camera having one image sensor may be used. For example, a convex mirror or the like may be used to guide the surrounding landscape to one image sensor, and a 360-degree panoramic image may be taken. That is, the helmet 5 may be equipped with an omnidirectional camera capable of acquiring a 360-degree omnidirectional image in one shot. With an omnidirectional camera, it is possible to shoot an omnidirectional image with one camera without changing the direction of the camera at the shooting site. Further, a spherical image which is a 360-degree panoramic image may be acquired by using two or more cameras having one image sensor and synthesizing the images taken by these cameras. The 360-degree panoramic image does not have to be a spherical image, but may be a horizontal 360-degree range (horizontal and horizontal directions).

The worker W takes a picture at the work site using the movable picture terminal 2, and the image is transmitted to the management computer 3. The video (image) shot by the shooting terminal 2 mainly exemplifies a moving image. A still image may be captured by the photographing terminal 2.

In the present embodiment, the photographing terminal 2 provided with the helmet 5 and the camera 6 is exemplified as the “movable terminal”, but other embodiments may be used. For example, the predetermined camera itself held by the worker W may be a “movable terminal”. Further, when a predetermined robot is equipped with a camera 6, the robot becomes a "movable terminal".

As shown in FIG. 3, in the on-site video management system 1 of the present embodiment, the image 7 taken by the camera 6, the position information (coordinate diagram 8) showing the movement trajectory of the shooting terminal 2 at the time of shooting, and the inspection site. The layout map 9 and is acquired. Here, the video 7 and the position information are acquired in synchronization. Then, the video 7 is superimposed and recorded on the layout map 9 according to the position information. The administrator M can display and view the omnidirectional video 7 at an arbitrary position of the layout map 9 at the inspection site by using the video viewing viewer 13.

As shown in FIG. 4, the image 7 taken by the camera 6 with a fisheye lens captures the scenery around the helmet 5 of the worker W as a base point. Then, by synthesizing the images 7 acquired by the two image sensors arranged back to back, it is possible to generate an omnidirectional image around the worker W.

As shown in FIG. 5, the position information of the photographing terminal 2 is recorded corresponding to the predetermined coordinate diagram 8. For example, the position information of the X coordinate and the Y coordinate corresponding to the plan view of the inspection site is recorded in the coordinate diagram 8. Although not particularly shown, the position information of the Z coordinate indicating the height position of the photographing terminal 2 is also recorded in the coordinate diagram 8. The movement locus of the photographing terminal 2 can be grasped from the position information acquired at predetermined time intervals. In this coordinate diagram 8, the coordinates obtained by the photographing terminal 2 are displayed as circles. In addition to the video, information obtained by laser scanning the inspection site (for example, three-dimensional feature point cloud data) may be recorded in association with the coordinates.

As shown in FIG. 6, in the layout map 9 of the inspection site, information indicating the equipment 10, the piping 11, and the like, as well as information indicating the inspection point 12 to be inspected by the worker W are registered. This information is registered in advance before the inspection of the worker W. The reference point of the coordinates of the layout map 9 coincides with the reference point of the position information of the photographing terminal 2. In this embodiment, a plan view (2D data) is exemplified as the layout map 9. The layout map 9 may be 3D data in which the state of the inspection site is three-dimensionally recorded.

In the shooting terminal 2, the shot video and the position information at the time of shooting are recorded together. Here, the shooting time of the image and the recording time of the position are matched and recorded. Then, the management computer 3 performs a process of allocating the video and the position information recorded at the same time to the layout map 9. The photographing terminal 2 may perform the process of allocating (allocating) to the layout map 9.

The "allocation" of the present embodiment is to divide a video (moving image) into frames (still images) and record each frame in association with the coordinate position of the layout map 9.

The position information of the video and the layout map 9 can be aligned with each other according to the reference point. By doing so, the locus of movement of the photographing terminal 2 can be displayed on the layout map 9.

As shown in FIG. 7, after the worker W completes the inspection, the manager M can view the video of the inspection site using the management computer 3. For example, the viewer 13 for viewing a video is provided with a first display field 14 in which a layout map 9 is displayed. Then, by selecting an arbitrary portion of the layout map 9 of the first display column 14 with the mouse cursor 16, an omnidirectional image of that portion can be displayed in the second display column 15. The first display field 14 and the second display field 15 can be displayed side by side at the same time.

Even if the image is taken by the camera 6 with a fisheye lens, when it is displayed on the viewer 13 for viewing the image, it is corrected to the image of the normal mode and displayed. Further, the administrator M can change the viewpoint of the image by using the mouse cursor 16. Therefore, the administrator M can confirm the situation of the worker W in the front, back, left, and right. Further, the viewer 13 for viewing a video may be used to display the information obtained by laser scanning the inspection site instead of the video.

The video displayed in the second display field 15 of the viewer 13 for viewing a video mainly exemplifies a still image. The moving image may be displayed in the second display field 15.

In the present embodiment, the video image taken by the camera 6 is recorded as a moving image, and the movement locus of the photographing terminal 2 is recorded in conjunction with the moving image. Therefore, the administrator M can select an arbitrary coordinate position at the inspection site with the viewer 13 for viewing the video. For example, it is possible to select arbitrary points arranged at intervals of 1 m or less, and it is possible to confirm a complete on-site image. The coordinate positions that can be selected by the viewer 13 for viewing the video are located on the path that the worker W has moved.

It is also possible to confirm the situation of moving to the inspection site by playing the moving image in the second display field 15 of the viewer 13 for viewing the image. Therefore, it is possible to effectively confirm the situation at the inspection site. Further, in the present embodiment, since the situation at the inspection site is recorded as a moving image, if the moving speed of the photographing terminal 2 is slow, or if the photographing terminal 2 stays at a predetermined point, the surroundings are viewed from the same viewpoint. It is also possible to grasp changes in the situation.

Further, by recording the moving image in cooperation with the position information on the shooting terminal 2, it is possible to compare it with the past image at the same place. Therefore, the manager M can surely grasp the state of change at the inspection site.

In the shooting terminal 2 of the present embodiment, it is possible to reduce the amount of data when shooting a moving image. For example, in the prior art, when recording an omnidirectional video as a moving image, there is a problem that the data capacity is increased. Generally, image compression technology that uses change information of moving images is adopted to reduce the amount of moving image data. However, the increase in the capacity of moving image data due to the recent increase in the resolution of the small camera 6 has become a problem again. Therefore, in the present embodiment, the capacity of the moving image data is reduced by utilizing the characteristics of the inspection work.

For example, in inspection work, the equipment to be inspected is often set in advance. Then, the worker W patrols each inspection point based on the inspection plan. The worker W moves while checking whether there is any change in the surroundings such as the presence or absence of water leakage in the pipe even on the way to the inspection point on foot. With the method of leaving only the image of the inspection point as in the prior art, the worker W cannot leave the moving image. Therefore, the administrator M cannot grasp the surrounding situation while the worker W is moving.

Therefore, in the present embodiment, the moving image is recorded even when the worker W is moving. Here, in order to perform the work according to the inspection item at the inspection point, the worker W will stop at the place and perform the work. For example, when the worker W reaches a predetermined point and stops for a predetermined time, the acquisition of the omnidirectional image is completed. When the worker W stops at the inspection point for a long time, it is not necessary to acquire an additional image, and the shooting is temporarily stopped. In this way, the capacity of the moving image data can be reduced by changing the recording mode of the moving image when the worker W is moving toward the inspection point and when he / she is at the inspection point.

Further, when the worker W is moving at a predetermined moving speed or higher, there are many cases where there is no abnormality in the periphery and the worker W is moving smoothly. Therefore, when the worker W is moving at a predetermined moving speed or higher, the frame rate when shooting a moving image is lowered. In this way, the capacity of the moving image data can be reduced by changing the frame rate according to the state of the worker W.

As shown in FIG. 8, in the present embodiment, the coordinate positions of the inspection points are set in advance by associating the inspection points with the inspection points numbers that can be individually identified. Furthermore, the inspection items to be performed at each inspection location are set in advance. Then, the photographing terminal 2 determines whether or not the predetermined inspection point has been reached, and changes the frame rate or suspends photographing. As a result, it is possible to reduce the amount of video data while leaving the video necessary for grasping the situation at the inspection site.

As shown in FIG. 1, the management computer 3 handled by the administrator M is composed of a predetermined computer such as a desktop PC, a notebook PC, or a tablet PC. In this embodiment, a desktop PC is exemplified. The management computer 3 includes a computer main body 17, a display 18 for visual recognition by the administrator M, and a wireless communication device 19 for wireless communication with the photographing terminal 2. Here, an example of performing wireless communication is described, but wired communication is also possible. The display 18 may be separate from the computer main body 17 or may be integrated with the computer body 17.

The photographing terminal 2 can specify the self-position and the self-posture. For example, it may have a position measurement system such as GPS (Global Positioning System). Further, as the position estimation technique used for calculating the self-position and the displacement of the self-posture of the photographing terminal 2, for example, known techniques such as SLAM (Simultaneous Localization and Mapping) and SfM (Structure from Motion) can be used. With such a position estimation technique, it is possible to obtain a three-dimensional movement amount when the photographing terminal 2 is moved.

The photographing terminal 2 uses VSLAM (Visual Simultaneous Localization and Mapping) in particular. Then, the displacement of the position and the posture is calculated based on the image taken by the photographing terminal 2. That is, the photographing terminal 2 can estimate its own position even in a place where GPS cannot be used, such as indoors.

In the VSLAM technology, the feature points of surrounding objects are extracted by using the information acquired by a predetermined device such as the camera 6 of the photographing terminal 2. Then, the image taken by the camera 6 is analyzed, and the feature points of the object (for example, the part of the object such as a corner) are tracked in real time. Then, the three-dimensional information of the position or the posture of the photographing terminal 2 is estimated. By using this VSLAM technology, it is possible to record an image and record a position only with the camera 6. Therefore, the equipment of the worker W can be simplified.

This VSLAM technology extracts feature points of an object appearing in an omnidirectional image, and calculates position information and posture information of the photographing terminal 2 based on these feature points. For example, the current position and posture of the photographing terminal 2 are calculated by calculating the movement locus of the photographing terminal 2 from a predetermined position whose position is known as a starting point. According to this VSLAM technology, it is not necessary to acquire the 3D feature point cloud data or the 3D CAD data of the inspection site (shooting site) in advance. Then, at the inspection site, by calculating the feature points of the objects appearing in the plurality of frames constituting the omnidirectional image, the self-position and the self-posture can be estimated from the movement amount of the feature points. ..

The photographing terminal 2 of the present embodiment estimates its own position and its own posture with a predetermined marker as a reference point (starting point). For example, a marker as a specific subject of the present embodiment is arranged at the entrance / exit of a building that is an inspection site. When the worker W starts the work, the worker W first shoots the marker with the shooting terminal 2. Here, the photographing terminal 2 acquires the reference point of the self-position and the self-posture based on the image in which the marker is captured. By taking a picture of this marker and continuing the picture taking while moving, it is possible to estimate the moving path of the moving terminal 2 from the reference point.

The marker is provided in a state of being fixed at a predetermined position as a reference. The marker is a figure capable of image recognition. For example, a matrix type two-dimensional code, a so-called QR code (registered trademark), is used as a marker. In addition, information that can identify the location of a building or equipment may be used as a marker. For example, a name plate of an entrance / exit, a name plate of a room, a name plate on which the device number or name of a device is described, or the like may be used as a marker. The name plate is a small flat plate on which the brand is displayed.

The position of the photographing terminal 2 is specified based on the three-dimensional coordinates with respect to the marker. At this time, the moving distance of the photographing terminal 2 is calculated by using the image of the portion where the size of the marker or the distance is known in advance. The posture (shooting direction) of the shooting terminal 2 is obtained based on the orientation (yaw angle) in the horizontal plane at the position at the time of shooting, the pitch angle, and the roll angle of the shooting terminal 2.

The photographing terminal 2 of the present embodiment can record the video and the position information in cooperation with each other in an environment where GPS cannot be used, and can record the video as a moving image. Further, the video captured by the photographing terminal 2 is transmitted to the management computer 3. The video transmission timing to the management computer 3 may be during the work or after the work is completed.

Further, when the 3D feature point cloud data or the 3D CAD data of the inspection site is acquired in advance, the position information and the attitude information of the photographing terminal 2 are obtained based on one frame constituting the omnidirectional image. Can be calculated. The 3D feature point cloud data at the inspection site may be generated based on the video acquired in the previous inspection work. The 3D CAD data may be generated based on the design drawing of the inspection site.

Here, the photographing terminal 2 compares the feature points of the object captured in one frame constituting the omnidirectional image with the feature points of the three-dimensional feature point cloud data or the three-dimensional CAD data acquired in advance. Then, the parts that match each other are specified. For example, search for the same or similar part as the feature point of the object in the frame of the captured image. By specifying the matchable portion, it is possible to estimate the position and orientation of the photographing terminal 2 when the frame of the image is photographed.

If the 3D feature point cloud data or 3D CAD data of the inspection site is acquired in advance, not only does it not need to use the marker as a reference point (starting point), but also shooting is performed while the shooting terminal 2 is moving. There is no need to continue.

Further, when the 3D feature point cloud data or the 3D CAD data of the inspection site is acquired in advance, the position and orientation of the photographing terminal 2 are estimated based on not only one frame but also a plurality of frames. May be.

Next, the system configuration of the on-site video management system 1 will be described with reference to the block diagram shown in FIG.

The on-site video management system 1 of the present embodiment has hardware resources such as a CPU, ROM, RAM, and HDD, and when the CPU executes various programs, information processing by software is realized by using the hardware resources. It consists of a computer. Further, the on-site video management method of the present embodiment is realized by causing a computer to execute various programs.

The photographing terminal 2 includes a camera 6, a motion sensor 20, a storage unit 21, a communication unit 22, and a terminal control unit 23.

The camera 6 is mounted on the photographing terminal 2 and photographs an object around the photographing terminal 2 with visible light. The image taken by the camera 6 is input to the terminal control unit 23. The camera 6 shoots a moving image at a predetermined frame rate.

Further, the frame rate at the time of shooting by the camera 6 can be adjusted based on the control of the terminal control unit 23. The camera 6 can shoot at any frame rate of, for example, 1 FPS, 5 FPS, 10 FPS, 30 FPS, and 60 FPS. The frame rate during normal shooting is 30 FPS. Based on the control of the terminal control unit 23, the frame rate at the time of shooting of the camera 6 can be increased or decreased.

The "moving image" of the present embodiment includes not only a video shot at a constant frame rate but also a video recorded with the frame rate changed during shooting. Further, the "moving image" includes at least a part of an image taken at 1 FPS or more. For example, assuming that the moving speed of the worker W during walking is about 1 m / s, if the image is taken at 1 FPS or more, the situation at the inspection site can be recorded at intervals of 1 m or less. Further, if the image is taken at 1 FPS or more, the operation of the object to be reflected in the image can be recorded. Further, the "moving image" may include an image taken at less than 1 FPS, that is, an image obtained by continuously shooting a still image.

The motion sensor 20 is a 9-axis sensor that combines an inertial sensor (3-axis acceleration sensor and 3-axis angular velocity sensor) and a 3-axis geomagnetic sensor. The motion sensor 20 is mounted on the photographing terminal 2 and detects the acceleration generated when the photographing terminal 2 moves. Further, the motion sensor 20 can also detect the gravitational acceleration. Further, the motion sensor 20 detects the angular velocity generated when the posture of the housing of the photographing terminal 2 changes. By specifying the posture of the housing of the shooting terminal 2 by combining the value of the angular velocity and the value of the acceleration, the shooting direction of the camera 6 can be grasped. It is also possible to grasp the posture of the camera 6 by the geomagnetism. The acceleration value and the angular velocity value detected by the motion sensor 20 are input to the terminal control unit 23.

The storage unit 21 stores an image taken by the camera 6. The video stored in the storage unit 21 may be a moving image or a still image. Further, the storage unit 21 stores predetermined data related to the video. For example, the storage unit 21 stores 3D feature point cloud data or 3D CAD data at the inspection site, and information that can specify the coordinate position of the inspection location.

The communication unit 22 communicates with the management computer 3 via the communication line. The communication unit 22 of the present embodiment includes a network device, for example, a wireless LAN antenna. In addition, communication may be performed using a LAN cable or a USB cable. The photographing terminal 2 accesses the management computer 3 via the communication unit 22.

Although not particularly shown, the photographing terminal 2 may be provided with a three-dimensional measurement sensor. The three-dimensional measurement sensor can measure the distance from the photographing terminal 2 to the object, for example, by projecting a laser on the object and receiving the reflected light by the light receiving element. The imaging direction by the camera 6 and the measurement direction by the three-dimensional measurement sensor are the same. The three-dimensional measurement sensor measures the distance from the photographing terminal 2 to surrounding objects by using the ToF (Time of Flight) method that converts the delay time of the received light pulse with respect to the floodlight pulse into a distance, and forms a three-dimensional point group. can do.

The terminal control unit 23 comprehensively controls the photographing terminal 2. The terminal control unit 23 includes a video acquisition unit 24, a video synthesis unit 25, a video recording unit 26, a data distribution unit 27, a position information acquisition unit 28, a posture information acquisition unit 29, an environment map creation unit 30, and a frame rate adjustment unit 31. The moving speed acquisition unit 32 and the inspection location determination unit 33 are provided. These are realized by executing the program stored in the memory or the HDD by the CPU.

The video acquisition unit 24 acquires a video including at least a moving image shot by the camera 6 mounted on the shooting terminal 2.

The image compositing unit 25 joins the respective images taken by the plurality of image sensors and synthesizes a 360-degree omnidirectional image. In the present embodiment, one camera 6 is equipped with two image sensors, and an omnidirectional image is synthesized based on each image taken by these image sensors. It should be noted that the two cameras may be arranged back to back, and the images in all directions may be combined based on the images taken by these cameras. By doing so, it is possible to acquire an omnidirectional image using a general camera. The video compositing unit 25 may be provided by the management computer 3.

The video recording unit 26 controls to record the video captured by the camera 6 in the storage unit 21. The video recording unit 26 controls to record the position information and the posture information in the storage unit 21 in association with the video.

The data distribution unit 27 controls to transmit the data including the video, the position information, and the posture information acquired by the shooting terminal 2 to the management computer 3.

The position information acquisition unit 28 acquires the position (coordinates) of the photographing terminal 2 at the time of photographing the image by the camera 6 based on the information acquired by the motion sensor 20 or the information acquired by the SLAM technique. The position information indicating the position (shooting position) of the shooting terminal 2 is acquired together with the shooting of the image by the camera 6. This position information is stored in the storage unit 21 in association with the time at the time of calculation or update.

By using the VSLAM technology, the position information acquisition unit 28 can acquire position information based on at least one frame of the image captured by the camera 6. By doing so, it is possible to acquire the position information of the photographing terminal 2 by using only the image captured by the camera 6.

The posture information acquisition unit 29 acquires the posture of the photographing terminal 2 at the time of photographing the image by the camera 6 based on the information acquired by the motion sensor 20 or the information acquired by the SLAM technique. The posture information indicating the posture (shooting direction) of the shooting terminal 2 is acquired together with the shooting of the image by the camera 6. This posture information is stored in the storage unit 21 in association with the time at the time of calculation or update.

The posture information acquisition unit 29 can acquire posture information based on at least one frame of the image captured by the camera 6 by using the VSLAM technology. By doing so, it is possible to acquire the posture information of the photographing terminal 2 by using only the image captured by the camera 6.

The environmental map creation unit 30 creates an environmental map including information on the surrounding environment of the photographing terminal 2 by using SLAM technology or VSLAM technology. By doing so, it is possible to improve the acquisition accuracy of the position (shooting position) of the shooting terminal 2 and the orientation (shooting direction) of the housing at the time of shooting a moving image.

In the SLAM technique of the present embodiment, the feature points of the objects around the photographing terminal 2 are extracted by using only the information acquired by the camera 6 of the photographing terminal 2. It is also possible to extract feature points of objects around the photographing terminal 2 by using the information acquired by the camera 6 and the three-dimensional measurement sensor. A collection of feature points is referred to as three-dimensional feature point cloud data. The terminal control unit 23 analyzes the image captured by the camera 6 and tracks the feature points of the object (for example, the sides or corners of the box-shaped object) in real time. Based on this 3D feature point cloud data, 3D information on the position and orientation of the photographing terminal 2 can be estimated. At this time, the moving distance of the photographing terminal 2 is calculated by using the image of the portion where the size of the marker or the distance is known in advance.

Further, the 3D feature point cloud data of the object around the shooting terminal 2 is detected at predetermined time intervals, and the position and orientation of the shooting terminal 2 are based on the displacement between the 3D feature point cloud data before and after in the time series. The displacement of can be calculated. Further, from a series of self-positions and self-postures obtained in time series, a movement path of the photographing terminal 2 prior to the current position and the current posture can be obtained.

That is, the terminal control unit 23 controls to acquire the position and posture of the shooting terminal 2 based on the continuously shot images taken when the shooting terminal 2 moves from the reference point. By doing so, the path that the photographing terminal 2 has moved from the reference point can be estimated from the image, so that the estimation accuracy of the position and the posture of the photographing terminal 2 can be improved.

In this embodiment, the SLAM technique using the camera 6 is exemplified, but other embodiments may be used. For example, the self-position and self-posture of the photographing terminal 2 may be estimated based on the information acquired by the stereo camera, the gyro sensor, and the infrared sensor. The SLAM technology of the present embodiment can measure the position indoors, and is the most accurate among the position information measurement technologies that can be used indoors such as beacon or pedestrian autonomous navigation (PDR: Pedestrian Dead Reckoning). You can often calculate 3D coordinates.

If there is an error between the position information or posture information based on the environmental map and the position information or posture information based on the reference point, the position information or posture information based on the reference point may be preferentially applied. By doing so, the worker W can correct the information by photographing the marker at the work site. Since the VSLAM technique is a method of calculating the relative movement amount, an error may be accumulated by using it for a long time. Therefore, by reading the marker serving as the reference point with the photographing terminal 2, the position information at the inspection site can be accurately grasped.

When the camera 6 shoots a moving image, the frame rate adjusting unit 31 controls to make the frame rate different between a part of the shooting period and another period. By doing so, the frame rate can be adjusted according to the shooting period.

Further, the frame rate adjusting unit 31 controls to change the frame rate according to the moving speed of the shooting terminal 2 when shooting a moving image with the camera 6. By doing so, the capacity of the moving image data can be reduced. For example, the frame rate can be lowered while the shooting terminal 2 is moving to reduce the amount of moving image data.

When performing inspection work or the like at the inspection location, the photographing terminal 2 does not move and the surrounding conditions do not change, so that the imaging is temporarily stopped. When the photographing terminal 2 is at the inspection point, the frame rate may be increased. By doing so, it is possible to acquire a detailed video of the inspection work.

The moving speed acquisition unit 32 acquires the moving speed of the photographing terminal 2 based on the detection of the acceleration and the angular velocity by the motion sensor 20. The moving speed acquisition unit 32 can also acquire the moving speed of the photographing terminal 2 by using the VSLAM technology.

The inspection location determination unit 33 determines whether or not the photographing terminal 2 is at the inspection location. In response to this determination, the frame rate adjusting unit 31 controls to make the frame rate different depending on whether the photographing terminal 2 is at the inspection point or not at the inspection point. By doing so, the frame rate can be adjusted between the inspection location and the inspection location other than the inspection location. The photographing terminal 2 accesses the management computer 3 before the start of the inspection, and acquires information (FIG. 8) capable of specifying the coordinate position of the inspection location in advance. The inspection point determination unit 33 makes a determination based on the acquired coordinate position of the inspection point. The inspection point is not limited to a predetermined point, but refers to a point having a predetermined area (range). That is, the inspection point determination unit 33 determines whether or not the photographing terminal 2 is within the range of the inspection point.

As shown in FIG. 2, the management computer 3 includes a communication unit 34, a storage unit 35, an input unit 36, a display 18, and a management control unit 37.

The system configuration of the management computer 3 of the present embodiment does not necessarily have to be provided in one computer. For example, one system may be realized by using a plurality of computers connected to each other by a network.

The communication unit 34 communicates with the photographing terminal 2 via the communication line. The communication unit 34 is mounted on the wireless communication device 19. Further, the communication unit 34 may be mounted on a predetermined network device, for example, a wireless LAN access point or an antenna. The communication unit 34 may communicate with the photographing terminal 2 via a WAN (Wide Area Network), an Internet line, or a mobile communication network. The communication unit 34 may communicate with another computer via a communication line.

The storage unit 35 records the image sent from the photographing terminal 2. In addition, the position information and the posture information associated with the video are recorded. In addition, a preset layout map (Fig. 6) and the like are also recorded.

The storage unit 35 may include a database. A database is a collection of information stored in memory or HDD and organized for retrieval or storage.

The input unit 36 inputs predetermined information according to the operation of the administrator M who uses the management computer 3. The input unit 36 includes an input device such as a mouse or a keyboard. That is, predetermined information is input to the input unit 36 according to the operation of these input devices. The input operation of the present embodiment includes a click operation using a mouse or a touch operation using a touch panel. That is, predetermined information is input to the management computer 3 according to the operation of these input devices.

The display 18 is a device that outputs predetermined information. For example, it is a device that displays the viewer 13 (FIG. 7) for viewing a video. The management computer 3 controls the image displayed on the display 18. The management computer 3 may control the image displayed on the display of another computer connected via the network.

In the present embodiment, the display 18 is exemplified as a device for displaying an image, but other embodiments may be used. For example, information may be displayed using a head-mounted display or a projector. Further, a printer that prints information on a paper medium may be used instead of the display. That is, the object controlled by the management computer 3 may include a head-mounted display, a projector, or a printer.

The management control unit 37 comprehensively controls the management computer 3. The management control unit 37 includes a data acquisition unit 38, a map allocation unit 39, and a video display control unit 40. These are realized by executing the program stored in the memory or the HDD by the CPU.

The data acquisition unit 38 controls to receive data including video, position information, and posture information from the photographing terminal 2. The data received from the photographing terminal 2 is stored in the storage unit 35. The data acquisition unit 38 acquires information about the inspection site such as the layout map 9 in advance. Then, a process of recording information such as the layout map 9 in the storage unit 35 is performed.

The map allocation unit 39 specifies the coordinate position of the layout map 9 (FIG. 6) of the inspection site (shooting site) based on the position information, and executes a process of allocating the image to the specified coordinate position. That is, the process of recording the video in association with the specified coordinate position is performed.

The map allocation unit 39 specifies the orientation of the housing of the photographing terminal 2 at the time of photographing based on the posture information, and executes a process of allocating an image according to the specified orientation. That is, a process of recording an image in association with the orientation of the housing of the photographing terminal 2 at the time of photographing is performed.

The video display control unit 40 controls the display of the viewer 13 (FIG. 7) for viewing video on the display 18. Further, the video display control unit 40 receives the selection of the coordinate position of the layout map 9 by the viewer 13 for viewing the video, and controls to display the video assigned to the selected coordinate position. For example, as shown in FIG. 7, the administrator M selects a predetermined inspection point 12 in the first display field 14 of the viewer 13 for viewing a video. Based on this selection, the image of the selected inspection point 12 is displayed in the second display column 15.

Next, the terminal-side processing executed by the photographing terminal 2 of the on-site video management system 1 will be described with reference to the flowchart of FIG. Refer to the above drawings as appropriate. This process is a process that is repeated at regular time intervals. By repeating this process, the on-site video management method is executed. It should be noted that this process may be interrupted and executed while the photographing terminal 2 is executing another main process. The following steps are at least a part of the processes included in the terminal-side processing, and other steps may be included in the terminal-side processing.

First, in step S11, the terminal control unit 23 executes the self-position estimation process. In this self-position estimation process, a process of estimating the position and posture of the photographing terminal 2 is performed based on the information acquired by the motion sensor 20 or the information acquired by the SLAM technique. When this self-position estimation process is executed, the camera 6 is activated and shooting is started. Here, the position information acquisition unit 28 acquires the position information of the photographing terminal 2. The posture information acquisition unit 29 acquires the posture information of the photographing terminal 2. The environmental map creation unit 30 creates an environmental map including information on the surrounding environment of the photographing terminal 2.

In the next step S12, the terminal control unit 23 (frame rate adjustment unit 31) executes the frame rate adjustment process (FIG. 10) described later.

In the next step S13, the video acquisition unit 24 executes the video acquisition process. In this video acquisition process, first, the camera 6 mounted on the shooting terminal 2 shoots a video including at least a moving image. Then, the image acquisition unit 24 performs a process of acquiring an image captured by the camera 6.

In the next step S14, the video compositing unit 25 executes the video compositing process. In this image compositing process, the images taken by the plurality of image sensors of the camera 6 are joined together to synthesize an omnidirectional image of 360 degrees. When two cameras are used, the images taken by these cameras are joined together to synthesize a 360-degree omnidirectional image. The video compositing process may be executed by the management computer 3.

In the next step S15, the video recording unit 26 executes the video recording process. In this video recording process, the video acquired by the video acquisition unit 24 is recorded in the storage unit 21. Here, the video recording unit 26 records the video, the position information, and the posture information in association with each other.

In the next step S16, the data distribution unit 27 executes the data distribution process. In this data distribution process, data including a video acquired by the photographing terminal 2, position information, and posture information is transmitted to the management computer 3. Then, the photographing terminal 2 ends the processing on the terminal side.

Next, the frame rate adjustment process executed by the terminal control unit 23 (frame rate adjustment unit 31) will be described with reference to the flowchart of FIG.

First, in step S21, the inspection location determination unit 33 determines whether or not the current position of the photographing terminal 2 is the inspection location based on the position information acquired by the position information acquisition unit 28. Here, if the current position of the photographing terminal 2 is the inspection point (YES in step S21), the process proceeds to step S25 described later. On the other hand, if the current position of the photographing terminal 2 is not the inspection point (NO in step S21), the process proceeds to step S22.

In step S22, the moving speed acquisition unit 32 acquires the moving speed of the photographing terminal 2 based on the detection of the acceleration and the angular velocity by the motion sensor 20. The moving speed acquisition unit 32 can also acquire the moving speed of the photographing terminal 2 by using the VSLAM technology.

In the next step S23, the frame rate adjusting unit 31 determines whether or not the moving speed acquired by the moving speed acquisition unit 32 is equal to or higher than a preset threshold value. Here, if the moving speed is less than the threshold value (NO in step S23), the process proceeds to step S25 described later. On the other hand, when the moving speed is equal to or higher than the threshold value (YES in step S23), the process proceeds to step S24.

In step S24, the frame rate adjusting unit 31 lowers the frame rate at the time of shooting by the camera 6. For example, when the normal frame rate is 30 FPS, the frame rate is reduced to 10 FPS. Further, the frame rate may be gradually lowered each time the moving speed is gradually increased. Then, the terminal control unit 23 ends the frame rate adjustment process.

In step S25, which proceeds to the case of YES in step S21 or NO in step S23, the frame rate adjusting unit 31 determines whether or not the photographing terminal 2 is stopped on the spot for a certain period of time or more without moving. To judge. That is, it is determined whether or not the state where the moving speed is almost zero continues for a certain period of time or more. An arbitrary time is set in advance for the fixed time used for the determination. Here, if the vehicle has not been stopped for a certain period of time or longer (NO in step S25), the process proceeds to step S27 described later. On the other hand, if the vehicle has been stopped for a certain period of time or longer (YES in step S25), the process proceeds to step S26.

In step S26, the frame rate adjusting unit 31 stops shooting by the camera 6. The frame rate at the time of shooting by the camera 6 may be lowered to 1 FPS or less. Then, the terminal control unit 23 ends the frame rate adjustment process.

In step S27, which proceeds in the case of NO in step S25 described above, the frame rate adjusting unit 31 sets the frame rate of the camera 6 at the time of shooting to a normal frame rate (for example, 30 FPS) for shooting. Then, the terminal control unit 23 ends the frame rate adjustment process.

Next, the management-side processing executed by the management computer 3 of the on-site video management system 1 will be described using the flowchart of 11. Refer to the above drawings as appropriate. This process is a process that is repeated at regular time intervals. By repeating this process, the on-site video management method is executed. It should be noted that this process may be interrupted and executed while the management computer 3 is executing another main process. The following steps are at least a part of the processes included in the management side process, and other steps may be included in the management side process.

First, in step S31, the data acquisition unit 38 executes the data acquisition process. In this data acquisition process, data including a video, position information, and posture information is received from the photographing terminal 2.

In the next step S32, the map allocation unit 39 executes the map allocation process. In this map allocation process, the coordinate position of the layout map 9 (FIG. 6) of the inspection site is specified based on the position information, and the image is assigned to the specified coordinate position.

In the next step S33, the video display control unit 40 executes the video display control process. In this video display control process, first, a viewer 13 (FIG. 7) for viewing a video is displayed on the display 18. In addition, the selection of the predetermined inspection point 12 in the first display column 14 is accepted. Further, the image acquired at the selected inspection point 12 is displayed in the second display column 15. Then, the management computer 3 ends the processing on the management side.

Next, the relationship between each frame constituting the video (moving image) in the video compositing process, the video recording process, and the map allocation process, the position information, and the posture information will be described with reference to FIGS. 12 to 14.

As shown in FIG. 12, each frame of the omnidirectional video is recorded in association with the shooting time. Further, the position information (coordinate position) of the photographing terminal 2 is recorded in association with the recording time. Further, the posture information of the photographing terminal 2 is recorded in association with the recording time.

In the video recording process, the recording time corresponding to the shooting time of the frame is specified, and the position information and the posture information corresponding to the specified recording time are recorded as the position information and the posture information corresponding to the frame.

Further, in the map allocation process, the coordinate position of the layout map (FIG. 6) is specified based on the position information corresponding to the frame. Further, the orientation of the frame at the time of reproduction is specified based on the posture information corresponding to the frame. Then, a frame is assigned to the specified coordinate position on the layout map.

As shown in the first modification of FIG. 13, in the video recording process, the position information and the posture information may be recorded in association with each frame at the time of shooting the video. By doing so, it is not necessary to perform a process of matching the frame, the position information, and the posture information based on the shooting time and the recording time.

In the second modification of FIG. 14, an embodiment in which a camera 6 with a fisheye lens is attached to the forehead of the worker W to take a picture is illustrated. It is assumed that the camera 6 captures the front side of the worker W.

For example, on the outbound route where the worker W heads for the inspection point, it is assumed that the worker W walks in the north direction. Then, on the return route in which the worker W returns from the inspection point, the worker W shall walk toward the south. Then, the camera 6 attached to the forehead of the worker W captures both the north direction and the south direction images at a predetermined position. Then, by synthesizing the images of the outward route and the return route at the same position, it is possible to acquire the images in all directions at that position.

In the video recording process of the second modification, the position information and the posture information are recorded in association with each frame when the video is shot. Then, in the video compositing process, it is specified that the position information at the time of shooting on the outward route and the position information at the time of shooting on the return route match. Then, frames with matching position information are combined to generate an omnidirectional image.

A general mode of reproducing a moving image is to reproduce frames in chronological order in which they were photographed. However, in the present embodiment, the time axis of the moving image is ignored, and the frames are reproduced in the position order (coordinate axis order) at the time of shooting. By doing so, it is possible to construct a system for confirming the situation at the shooting site based on the shooting of the moving image.

In the present embodiment, the determination of an arbitrary value (movement speed or movement stop time) using the reference value (threshold value or fixed time) may be the determination of "whether or not the arbitrary value is equal to or greater than the reference value". , "Whether or not an arbitrary value exceeds the reference value" may be determined. Alternatively, it may be a determination of "whether or not an arbitrary value is equal to or less than a reference value" or a determination of "whether or not an arbitrary value is less than a reference value". Further, the reference value is not fixed but may change. Therefore, a value in a predetermined range may be used instead of the reference value, and it may be determined whether or not an arbitrary value falls within the predetermined range. Further, the error generated in the apparatus may be analyzed in advance, and a predetermined range including the error range centered on the reference value may be used for the determination.

Although the flowchart of the present embodiment illustrates a mode in which each step is executed in series, the context of each step is not necessarily fixed, and even if the context of some steps is exchanged. good. Also, some steps may be executed in parallel with other steps.

The system of this embodiment includes a control device in which a dedicated chip, a controller such as an FPGA (Field Programmable Gate Array), a GPU (Graphics Processing Unit), or a CPU (Central Processing Unit) is highly integrated, and a ROM (Read Only). Storage devices such as Memory) or RAM (Random Access Memory), external storage devices such as HDD (Hard Disk Drive) or SSD (Solid State Drive), display devices such as displays, and input devices such as mice or keyboards. , With a communication interface. This system can be realized with a hardware configuration using a normal computer.

The program executed by the system of the present embodiment is provided by incorporating it into a ROM or the like in advance. Alternatively, the program may be a non-transient storage medium that is a computer-readable file such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD) in an installable or executable format. It may be stored in the computer and provided.

Further, the program executed by this system may be stored on a computer connected to a network such as the Internet, and may be downloaded and provided via the network. The system can also be configured by connecting and combining separate modules that independently exert the functions of the components to each other via a network or a dedicated line.

In the present embodiment, the image and the position information are acquired by the camera 6 mounted on the helmet 5, but other embodiments may be used. For example, the image and the position information may be acquired by the camera 6 mounted on the tablet terminal 4.

In this embodiment, the position of the photographing terminal 2 is acquired by the SLAM technique, but other embodiments may be used. For example, when the inspection site is outdoors, the position of the photographing terminal 2 may be acquired using GPS.

According to the embodiment described above, by providing a video display control unit that accepts the selection of the coordinate position of the map and controls the display of the video assigned to the selected coordinate position, a moving terminal is used. It is possible to display an image at an arbitrary point on the movement path of the terminal when playing back the captured image.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, changes, and combinations can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope of the invention described in the claims and the equivalent scope thereof, as are included in the scope and gist of the invention.

1 ... On-site video management system, 2 ... Shooting terminal, 3 ... Management computer, 4 ... Tablet terminal, 5 ... Helmet, 6 ... Camera, 7 ... Video, 8 ... Coordinate map, 9 ... Layout map, 10 ... Equipment, 11 ... Piping, 12 ... Inspection points, 13 ... Viewer, 14 ... First display column, 15 ... Second display column, 16 ... Mouse cursor, 17 ... Computer body, 18 ... Display, 19 ... Wireless communication device, 20 ... Motion sensor, 21 ... Storage unit, 22 ... Communication unit, 23 ... Terminal control unit, 24 ... Video acquisition unit, 25 ... Video synthesis unit, 26 ... Video recording unit, 27 ... Data distribution unit, 28 ... Position information acquisition unit, 29 ... Attitude Information acquisition unit, 30 ... Environmental map creation unit, 31 ... Frame rate adjustment unit, 32 ... Movement speed acquisition unit, 33 ... Inspection location determination unit, 34 ... Communication unit, 35 ... Storage unit, 36 ... Input unit, 37 ... Management Control unit, 38 ... Data acquisition unit, 39 ... Map allocation unit, 40 ... Video display control unit, M ... Administrator, W ... Worker.

Claims (10)

A video acquisition unit that acquires video including at least video shot by a camera mounted on a mobile terminal, and a video acquisition unit.
A position information acquisition unit that acquires the position information of the terminal together with shooting the image,
A video recording unit that records the video in association with the location information,
A map allocation unit that specifies the coordinate position of the map showing the layout of the shooting site based on the position information and allocates the image to the specified coordinate position.
An image display control unit that accepts selection of the coordinate position of the map and controls to display the image assigned to the selected coordinate position.
To prepare
On-site video management system. The image taken by the camera is taken by an omnidirectional camera capable of acquiring the image in all directions of 360 degrees in one shooting.
The on-site video management system according to claim 1. It is provided with a video compositing unit that joins the respective images taken by the plurality of cameras and synthesizes the 360-degree omnidirectional video.
The on-site video management system according to claim 1 or 2. The position information acquisition unit acquires the position information based on at least one frame of the image captured by the camera.
The on-site video management system according to any one of claims 1 to 3. It is provided with a frame rate adjusting unit that makes the frame rate different between a part of the shooting period and another period when the moving image is shot by the camera.
The on-site video management system according to any one of claims 1 to 4. A moving speed acquisition unit for acquiring the moving speed of the terminal is provided.
The frame rate adjusting unit makes the frame rate different according to the moving speed.
The on-site video management system according to claim 5. The shooting site is a place including an inspection point in the plant, and is provided with an inspection point determination unit for determining whether or not the terminal is in the inspection point.
The frame rate adjusting unit makes the frame rate different depending on whether the terminal is at the inspection point or not.
The on-site video management system according to claim 5 or 6. The moving image includes at least a part shot at 1 FPS or more.
The on-site video management system according to any one of claims 1 to 7. It is provided with an environment map creation unit that creates an environment map including information on the surrounding environment of the terminal based on the information obtained by the device mounted on the terminal.
The on-site video management system according to any one of claims 1 to 8. Steps to get at least video, including video, taken with a camera mounted on a mobile device,
The step of acquiring the position information of the terminal together with the shooting of the video, and
A step of associating and recording the video with the position information,
A step of specifying the coordinate position of the map showing the layout of the shooting site based on the position information and allocating the image to the specified coordinate position.
A step of accepting the selection of the coordinate position of the map and controlling the display of the image assigned to the selected coordinate position.
including,
On-site video management method.

Images