WO2024084873A1 - Photographing method and image processing method - Google Patents

Abstract

A photographing method for photographing, while moving, an object by a photographing device installed in a mobile unit is characterized in that the photographing device photographs a region to be photographed in the object located in a direction intersecting a moving direction, the method comprises an installation position setting step for setting the installation position of the photographing device in the mobile unit, and when, using an XYZ orthogonal coordinate system with the moving direction of the mobile unit as an X-axis, the direction intersecting the moving direction as a Z-axis, and a direction orthogonal to the X-axis and the Z-axis as a Y-axis, the length from a position at which a photographing reference position of the photographing device installed in the mobile unit is projected onto an X-Z plane on which the mobile unit moves to a position on the X-Z plane at which the X-Z plane and the object intersect is defined as the distance from the photographing reference position to the object, and the length on the Y-axis from the X-Z plane to the region to be photographed in the object is defined as the height of the region to be photographed, in the installation position setting step, the installation position is adjusted in the Z-axis direction intersecting the moving direction according to the distance from the photographing reference position to the object and/or the height of the region to be photographed.

Description

Photographing method and information processing method

The present invention relates to a photography method and an information processing method.

For example, Patent Document 1 describes a method of inspecting a slope by inspecting the condition of the slope based on visual observation, and recording the condition and location of any abnormalities found.

In this slope inspection method, a vehicle traveling on the flat surface at the top or to the side of the slope uses multiple cameras with different focal lengths to divide the slope in the height direction and continuously photograph it with an almost uniform resolution, while a position measurement means mounted on the vehicle records the photographing positions. This slope inspection method also creates three-dimensional measurement data of the slope through photogrammetry using the continuously photographed images and the photographing positions and orientations. This slope inspection method also uses the three-dimensional measurement data to externally observe the condition of the slope based on a direct projection image that has been projected in a direction directly facing the slope, and if any abnormality is found, identifies its condition and position.

However, there is room for improvement in Patent Document 1 in terms of obtaining an image that is in focus over a wide area of the slope.

The present invention has been made in consideration of the above points, and aims to provide an imaging method and information processing method that can obtain an image that is in focus over a wide area of an object (e.g., a slope).

The photographing method of this embodiment is a photographing method in which an object is photographed while moving by a photographing device installed on a moving body, and the photographing device photographs a photographing target area of the object located in a direction intersecting with the moving direction, and includes an installation position setting step for setting the installation position of the photographing device on the moving body, and using an XYZ Cartesian coordinate system in which the moving direction of the moving body is the X axis, the direction intersecting the moving direction is the Z axis, and the direction perpendicular to the X axis and the Z axis is the Y axis, the length from the position where the photographing reference position of the photographing device installed on the moving body is projected onto the XZ plane on which the moving body moves to the position on the XZ plane where the XZ plane and the object intersect is defined as the distance from the photographing reference position to the object, and the length on the Y axis from the XZ plane to the photographing target area of the object is defined as the height of the photographing target area, and the installation position setting step is characterized in that the installation position is adjusted in the Z axis direction intersecting with the moving direction in accordance with at least one of the distance from the photographing reference position to the object and the height of the photographing target area.

The present invention provides an imaging method and information processing method that can obtain an image that is in focus over a wide area of an object (e.g., a slope).

FIG. 1 is a diagram showing an example of the overall configuration of a state inspection system according to the present embodiment. FIG. 2 is a diagram showing an example of how a slope condition is inspected using the condition inspection system. FIG. 3 is a diagram showing an example of a state of a slope. FIG. 4 is a diagram illustrating an example of a hardware configuration of the data acquisition device. FIG. 5 is a diagram for explaining a photographed image acquired by the mobile system. FIG. 6 is an explanatory diagram of a photographed image and a distance measurement image. FIG. 7 is an explanatory diagram of a plurality of shooting regions. FIG. 8 is a diagram showing an example of the configuration of a moving body equipped with the imaging device of this embodiment. FIG. 9 is a diagram illustrating an example of the configuration of the variable installation position photographing unit. FIG. 10 is a diagram showing an example of adjustment of the installation position of the installation position variable imaging unit. FIG. 11 is a first conceptual diagram showing an example of the allowable deviation from the theoretical value of the Scheimpflug angle. FIG. 12 is a second conceptual diagram showing an example of the allowable deviation from the theoretical value of the Scheimpflug angle. FIG. 13 is a third conceptual diagram showing an example of the allowable deviation from the theoretical value of the Scheimpflug angle. FIG. 14 is a sequence diagram showing an example of a display process in the state inspection system. FIG. 15 is an explanatory diagram of operations on the display screen of the status inspection system. FIG. 16 is a flowchart showing an example of the process of the information processing method. FIG. 17 is a diagram showing an example of a display screen after the processing shown in FIG.

The imaging method and information processing method of this embodiment will be described in detail below with reference to the drawings. In the following description, the X-axis, Y-axis, and Z-axis directions are based on the arrow directions shown in the drawings. The X-axis, Y-axis, and Z-axis directions define a three-dimensional space that is orthogonal to each other. The X-axis direction may be interpreted as the moving direction of the moving body (direction of progress, direction of travel). The Y-axis direction may be interpreted as the height direction of the slope of the object (vertical direction, perpendicular direction). The Z-axis direction may be interpreted as the direction intersecting with the moving direction of the moving body (depth direction from the moving body toward the slope, imaging direction). In this way, the imaging method and information processing device of this embodiment use an XYZ Cartesian coordinate system in which the moving direction of the moving body is the X-axis, the direction intersecting the moving direction is the Z-axis, and the direction perpendicular to the X-axis and Z-axis is the Y-axis.

FIG. 1 is a diagram showing an example of the overall configuration of a condition inspection system 1 for implementing the imaging method and information processing method of this embodiment. The condition inspection system 1 is an example of an information processing system, and is a system for inspecting the condition of road earthwork structures using various data acquired by a mobile system 60. Road earthwork structures are a general term for structures constructed to build roads using ground materials such as soil and rocks as their main materials, and structures associated with them, and refer to cut and slope stabilization facilities, embankments, culverts, and similar. Hereinafter, road earthwork structures may be referred to as "objects," "slope," and "objects including slopes."

The condition inspection system 1 is composed of a mobile system 60, an evaluation system 4, a terminal device 1100 of the national or local government, and a terminal device 1200 of a contractor. The mobile system 60 is an example of an imaging system, and is composed of a data acquisition device 9 and a mobile body 6 such as a vehicle equipped with the data acquisition device 9. The vehicle may be a vehicle that runs on a road or a vehicle that runs on a railroad. The data acquisition device 9 has an imaging device 7, which is an example of a measuring device that measures a structure, as well as a distance sensor 8a and a GNSS (Global Navigation Satellite System) sensor 8b. GNSS is a general term for satellite positioning systems such as the Global Positioning System (GPS) or the Quasi-Zenith Satellite System (QZSS).

The imaging device 7 is a line camera equipped with a line sensor in which photoelectric conversion elements are arranged in one or more rows. The imaging device 7 captures images of positions along a predetermined imaging range on an imaging surface aligned in the traveling direction (X-axis direction) of the moving body 6. Note that the imaging device 7 is not limited to a line camera, and may be a camera equipped with an area sensor in which photoelectric conversion elements are arranged in a planar manner. The imaging device 7 may also be composed of multiple cameras. Details of the imaging device 7 composed of multiple cameras will be described later.

The distance sensor 8a is a ToF (Time of Flight) sensor that measures the distance to the subject photographed by the photographing device 7. The GNSS sensor 8b is a positioning means that receives signals transmitted at each time by multiple GNSS satellites and calculates the distance to the satellite from the difference in the time at which each signal was received, thereby measuring a position on the earth. The positioning means may be a device dedicated to positioning, or may be an application dedicated to positioning installed on a PC (Personal Computer), smartphone, etc. The distance sensor 8a and the GNSS sensor 8b are examples of sensor devices. The distance sensor 8a is also an example of a three-dimensional sensor.

The ToF sensor used as distance sensor 8a measures the distance from a light source to an object by irradiating the object with laser light from the light source and measuring the scattered and reflected light.

In this embodiment, the distance sensor 8a is a LiDAR (Light Detection and Ranging) sensor. LiDAR is a method of measuring the time of flight of light using pulses, but as another method of the ToF sensor, the distance may be measured using a phase difference detection method. In the phase difference detection method, a laser light amplitude modulated at a fundamental frequency is irradiated onto the measurement range, the reflected light is received, and the phase difference between the irradiated light and the reflected light is measured to obtain time, and the distance is calculated by multiplying this time by the speed of light. The distance sensor 8a may also be configured with a stereo camera, etc.

By using a three-dimensional sensor, the mobile system 60 can obtain three-dimensional information that is difficult to obtain from two-dimensional images, such as the height, inclination angle, or overhang of a slope.

The mobile system 60 may further include an angle sensor 8c. The angle sensor 8c is a gyro sensor or the like for detecting the angle (attitude) or angular velocity (or each acceleration) of the shooting direction of the image capture device 7.

The evaluation system 4 is constructed by an evaluation device 3 and a data management device 5. The evaluation device 3 and data management device 5 constituting the evaluation system 4 can communicate with a mobile system 60, a terminal device 1100 and a terminal device 1200 via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), etc. In addition, the communication network 100 may include not only wired communication but also networks using wireless communication such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution). In addition, the evaluation device 3 and the data management device 5 may have a communication function using a short-range communication technology such as NFC (Near Field Communication) (registered trademark).

The data management device 5 is an example of an information processing device, and is a computer such as a PC that manages various data acquired by the data acquisition device 9. The data management device 5 receives various acquired data from the data acquisition device 9, and transfers the received various acquired data to the evaluation device 3 that performs data analysis. The method of transferring the various acquired data from the data management device 5 to the evaluation device 3 may be manual transfer using a USB (Universal Serial Bus) memory or the like.

The evaluation device 3 is a computer such as a PC that evaluates the condition of the slope based on various acquired data transferred from the data management device 5. A dedicated application program for evaluating the condition of the slope is installed in the evaluation device 3. The evaluation device 3 detects the type or structure of the slope from the captured image data and sensor data, extracts shape data, and performs a detailed analysis by detecting the presence or absence of deformation and the degree of deformation. The evaluation device 3 also generates a report to be submitted to a road administrator such as the country, local government, or a commissioned business operator, using the captured image data, sensor data, evaluation target data, and the detailed analysis results. The data of the report generated by the evaluation device 3 is submitted to the country or local government via the commissioned business operator in the form of electronic data or printed paper. The report generated by the evaluation device 3 is called an investigation record sheet, inspection sheet, investigation ledger, or report. The evaluation device 3 is not limited to a PC, and may be a smartphone or tablet terminal. The evaluation system 4 may be configured to construct the evaluation device 3 and the data management device 5 as a single device or terminal.

The terminal device 1200 is provided to the commissioned business operator, and the terminal device 1100 is provided to the national or local government. The evaluation device 3, the terminal device 1100, and the terminal device 1200 are examples of communication terminals capable of communicating with the data management device 5, and various data managed by the data management device 5 can be viewed.

FIG. 2 is a diagram showing an example of how the condition of a slope is inspected using a condition inspection system 1 for implementing the imaging method and information processing method of this embodiment. The mobile system 60 drives a mobile body 6 equipped with a data acquisition device 9 along a road while capturing images of a predetermined area of the slope (the slope of the target object) using an imaging device 7.

As shown in Figure 2, a cut slope is a slope where the soil has been piled up, and a bank slope is a slope where the soil has been piled up. In addition, the slope on the side of a road that runs along the side of a mountain is called a natural slope. Cut slopes and bank slopes can be made more durable by planting plants on the surface of the slope, and can be left unchanged for decades. However, this is not always the case. When cut slopes, bank slopes, and natural slopes deteriorate due to wind and rain, surface collapses occur, causing rocks and soil to fall, or the mountain collapses, causing road closures. To prevent such situations, methods are used to spray mortar on the surface of the slope (mortar spraying) or to install and harden concrete structures to slow the rate at which the slope deteriorates when exposed to wind and rain. Structures constructed using such methods are called earthwork structures. Earthwork structures include retaining walls that are installed between natural slopes and roads, and rockfall protection fences that prevent rocks from falling onto the road, but both are intended to prevent road closures or human injury caused by soil or falling rocks flowing onto the road.

In recent years, the deterioration of earthwork structures that were constructed decades ago has been significant, making the development of social infrastructure a major issue. For this reason, it is important to detect deterioration of earthwork structures early and to inspect and protect them from deterioration in order to prolong their life. Conventionally, inspections of natural slopes and earthwork structures have been carried out by visual inspections by experts, investigating rock falls, collapses, landslides, or debris flows on the slopes and creating repair plans.

However, visual inspections by experts have problems with efficiency, such as the inability to inspect all of the large number of earthwork structures throughout Japan in a given period of time and the inability to inspect embankments at high altitudes or along rivers. In addition, visual inspections cannot quantitatively grasp the degree of deterioration, such as cracks or peeling that have occurred on the surface of earthwork structures.

The condition inspection system 1 according to the embodiment acquires photographed image data of the slope of an earthwork structure using the photographing device 7, and acquires sensor data including three-dimensional information using a three-dimensional sensor such as a distance sensor 8a. The evaluation system 4 then combines the acquired photographed image data and sensor data to evaluate the condition of the slope, thereby detecting shape data indicating the three-dimensional shape of the slope and detecting abnormalities such as cracks and peeling. This allows the condition inspection system 1 to efficiently perform evaluations that are difficult to inspect with the human eye.

Figure 3 shows an example of the condition of a slope. Figure 3(a) is an image showing the surface of the slope five years before the collapse, and Figure 3(b) is an explanatory diagram of the image shown in Figure 3(a). At this stage, cracks in the surface layer of the slope are noticeable, and image analysis shown in an unfolded view, etc. is effective in detecting changes or signs of changes in the surface layer, such as cracks, peeling, and seepage.

Figure 3(c) is an image showing the surface of the slope two years before the collapse, and Figure 3(d) is an explanatory diagram of the image shown in Figure 3(c). In this state, the inside of the slope has turned to soil, the soil has pushed against the surface of the slope, and the slope has bulged. In order to detect three-dimensional deformations such as cracks, steps, and bulges, three-dimensional analysis of images such as development drawings + cross-sections is effective.

Figure 3(e) is an image showing the surface of the collapsed slope, and Figure 3(f) is an explanatory diagram of the image shown in Figure 3(e). In this state, the surface layer of the slope is unable to contain the soil and has collapsed.

FIG. 4 is a diagram showing an example of the hardware configuration of the data acquisition device 9. The data acquisition device 9 includes the image capture device 7 and the sensor device 8 as shown in FIG. 1, as well as a controller 900 that controls the processing or operation of the data acquisition device 9.

The controller 900 includes an imaging device I/F (Interface) 901, a sensor device I/F 902, a bus line 910, a CPU (Central Processing Unit) 911, a ROM (Read Only Memory) 912, a RAM (Random Access Memory) 913, a HD (Hard Disk) 914, a HDD (Hard Disk Drive) controller 915, a network I/F 916, a DVD-RW (Digital Versatile Disk Rewritable) drive 918, a media I/F 922, an external device connection I/F 923, and a timer 924.

Of these, the imaging device I/F 901 is an interface for transmitting and receiving various data or information to and from the imaging device 7. The sensor device I/F 902 is an interface for transmitting and receiving various data or information to and from the sensor device 8. The bus line 910 is an address bus, data bus, etc. for electrically connecting each component such as the CPU 911 shown in FIG. 4.

The CPU 911 also controls the operation of the entire data acquisition device 9. The ROM 912 stores programs used to drive the CPU 911, such as IPL. The RAM 913 is used as a work area for the CPU 911. The HD 914 stores various data such as programs. The HDD controller 915 controls the reading and writing of various data from the HD 914 under the control of the CPU 911. The network I/F 916 is an interface for data communication using the communication network 100.

The DVD-RW drive 918 controls the reading and writing of various data from and to a DVD-RW 917, which is an example of a removable recording medium. Note that the medium is not limited to a DVD-RW, and may be a DVD-R or a Blu-ray (registered trademark) Disc, etc.

The media I/F 922 controls the reading and writing (storing) of data from and to a recording medium 921 such as a flash memory. The external device connection I/F 923 is an interface for connecting an external device such as an external PC 930 having a display, a reception unit, and a display control unit. The timer 924 is a measurement device with a time measurement function. The timer 924 may be a computer-based software timer. It is preferable that the timer 924 be synchronized with the time of the GNSS sensor 8b. This makes it easy to synchronize the time and associate the positions of each sensor data and captured image data.

Figure 5 is a diagram explaining the captured images acquired by the mobile system.

The mobile body system 60 uses the imaging device 7 provided in the data acquisition device 9 to capture images of slopes on the road while the mobile body 6 is traveling. The X-axis direction shown in FIG. 5 indicates the direction of movement of the mobile body 6, the Y-axis direction is the vertical direction, and the Z-axis direction is perpendicular to the X-axis and Y-axis directions and indicates the depth direction from the mobile body 6 toward the slope.

As the mobile unit 6 travels, the data acquisition device 9 acquires photographed image 1, distance-measured image 1, photographed image 2, and distance-measured image 2 in chronological order, as shown in FIG. 5. Distance-measured image 1 and distance-measured image 2 are images acquired by distance sensor 8a. At this time, the photographing device 7 and sensor device 8 are time-synchronized, so photographed image 1 and distance-measured image 1, and photographed image 2 and distance-measured image 2 are images of the same area of the slope. In addition, tilt correction (image correction) of the photographed image is performed based on the attitude of the vehicle at the time of shooting, and the image data and positioning data (north latitude and east longitude) are linked based on the time of the photographed image.

In this way, the mobile system 60 acquires photographed image data of the slope and sensor data acquired in response to photographing by the photographing device 7 while driving the vehicle as the mobile body 6, and uploads them to the data management device 5. Note that the data acquisition device 9 may acquire the ranging images and the photographed images while driving separately, but considering changes in the slope shape due to collapses, etc., it is preferable to acquire the ranging images and the photographed images for the same slope shape while driving at the same time.

Figure 6 is an explanatory diagram of the captured image and the distance measurement image.

FIG. 6(a) shows captured image data 7A of captured images 1, 2, etc. shown in FIG. 5. Each pixel 7A1 of the captured image data 7A acquired by the photographing device 7 is arranged at coordinates corresponding to the X-axis and Y-axis directions shown in FIG. 5, and has brightness information corresponding to the amount of stored power. In other words, the captured image data 7A is an example of a brightness image.

Then, the brightness information of each pixel 7A1 of the captured image data 7A is stored in the storage unit as captured image data in association with coordinates corresponding to the X-axis and Y-axis directions shown in FIG. 5.

FIG. 6(b) shows distance measurement image data 8A such as distance measurement images 1 and 2 shown in FIG. 5. Each pixel 8A1 of distance measurement image data 8A acquired by distance sensor 8a is arranged at coordinates corresponding to the X-axis and Y-axis directions shown in FIG. 5, and has distance information in the Z-axis direction shown in FIG. 5 corresponding to the amount of stored power. Note that distance measurement image data 8A is three-dimensional point cloud data, but is generally referred to as distance measurement image data because it is visually displayed with luminance information added when viewed by a user. The captured image data 7A and distance measurement image data 8A are collectively referred to as image data.

The distance information for each pixel 8A1 in the distance measurement image data 8A is then stored in the storage unit as three-dimensional data included in the sensor data, in association with coordinates corresponding to the X-axis and Y-axis directions shown in FIG. 5.

Here, the captured image data 7A shown in FIG. 6(a) and the distance measurement image data 8A shown in FIG. 6(b) are images of the same area of the slope, so the brightness information and distance information are stored in the memory unit in association with the coordinates corresponding to the X-axis and Y-axis directions shown in the figure.

FIG. 7 is an explanatory diagram of multiple shooting areas. As shown in FIG. 7(a), the shooting device 7 moves together with the moving body 6 and captures images of the target area 70, which is the shooting range of the slope 80, divided into multiple shooting areas d11, d12, etc., at a constant shooting interval t along the X-axis direction, which is the moving direction of the moving body 6.

As shown in FIG. 7(b), the captured images of multiple shooting areas d11, d12, etc. are long slit-shaped images in the Y-axis direction, and by stitching together the images of multiple shooting areas d11, d12, etc., it is possible to obtain a captured image of the target area 70 that is continuous in the X-axis direction.

FIG. 7(c) is a diagram showing a case where the entire target area 70 is divided into a plurality of target areas and imaged when the entire target area 70 is imaged. In FIG. 7(c), the entire target area 70 is imaged by dividing the captured image into four target areas, namely, target areas 701A, 702A, 701B, and 702B.

As in the case shown in FIG. 7(b), each of the multiple target areas 701A, 702A, 701B, and 702B is divided into multiple shooting areas d11, d12, etc., and images of the multiple shooting areas d11, d12, etc. are stitched together to obtain images of each of the multiple target areas 701A, 702A, 701B, and 702B. Then, by stitching together images of the multiple target areas 701A, 702A, 701B, and 702B, an image of the entire target area 70 can be obtained.

In this case, the image capture device 7 is equipped with multiple image capture devices, and the target areas 702A and 702B are captured by an image capture device different from the image capture device that captures the target areas 701A and 701B.

In addition, the target area 701B may be photographed by the same imaging device that photographed the target area 701A, and the target area 702B may also be photographed by the same imaging device that photographed the target area 702A.

As shown in FIG. 7(a), when the image capturing device 7 captures the target area of the slope 80 divided into a plurality of image capturing areas d11, d12, etc., it is desirable that the distance sensor 8a also acquires distance information indicating the distance from the distance sensor 8a to each of the plurality of image capturing areas d11, d12, etc.

As a result, as explained in FIG. 6, it is possible to easily associate the luminance information of each pixel 7A1 of the photographed image data 7A acquired by the photographing device 7 with the distance information of each pixel 8A1 of the distance measurement image data 8A acquired by the distance sensor 8a. Then, by associating the luminance information of each image of the captured image capturing the target area of the slope 80 with the distance information of each pixel of the distance measurement image capturing the target area of the slope 80, it is possible to perform a highly accurate inspection of the target area of the slope 80.

In this way, the condition inspection system 1 of this embodiment photographs the slope while a mobile body (vehicle) equipped with an imaging device (camera) travels along a road beside the slope. In this case, multiple imaging devices (cameras) are prepared, and the slope is divided and photographed in the traveling direction (X-axis direction) of the mobile body (vehicle) by each imaging device (camera). In addition, the imaging timing of each imaging device (camera) is synchronized to divide and photograph the area in the height direction (Y-axis direction) of the slope. In this way, the slope is sliced and photographed in the traveling direction (X-axis direction) of the mobile body (vehicle), and each slice image is stitched together after shooting to obtain a surface image of the entire slope at a specified resolution (e.g. 4K). This makes it possible to inspect the slope for deformation, and by reducing the stitched image, it becomes possible to check the condition of a wide area of the slope.

FIG. 8 is a diagram showing an example of the configuration of a moving body equipped with an image capture device of this embodiment. As shown in FIG. 8, a moving body (vehicle) 6 moving in the X-axis direction is equipped with multiple image capture devices (cameras) 7A, 7B, 7C, 7D, and 7E at different positions in the X-axis direction. The image capture devices 7A, 7B, 7C, 7D, and 7E are arranged in this order from the front to the rear of the moving body 6 in the direction of travel. The image capture devices 7A, 7B, 7C, 7D, and 7E may also be called cameras 1, 2, 3, 4, and 5. A distance sensor 8a (e.g., LiDAR) is mounted on the rear side of the image capture device 7E. Note that the arrangement of the image capture devices 7A to 7E and the distance sensor 8a is not limited to that shown in FIG. 8, and various design changes are possible.

There are cases where the height direction of the slope (object including a slope) of the object does not fit within the angle of view of a single camera, so multiple camera devices, here five camera devices 7A to 7E, are used to divide and photograph the area of the slope (object including a slope) of the object in the height direction. When the height direction of the slope (object including a slope) of the object is divided from bottom to top into a first area, a second area, a third area, a fourth area, and a fifth area, the first area is photographed by camera device 7A, the second area is photographed by camera device 7B, the third area is photographed by camera device 7C, the fourth area is photographed by camera device 7D, and the fifth area is photographed by camera device 7E. In other words, camera device 7A is the camera that photographs the lowest part of the slope, and camera device 7E is the camera that photographs the highest part of the slope. In the following explanation, it is assumed that the height of the slope (object including a slope) of the object is 15m or close to that.

In this way, the photographing method of this embodiment involves photographing an object (slope) while moving using photographing devices (cameras) 7A, 7B, 7C, 7D, and 7E installed on a moving body (vehicle) 6. Furthermore, the photographing devices (cameras) 7A, 7B, 7C, 7D, and 7E photograph a "photographed target area" of the object (slope) located in the Z-axis direction that intersects with the moving direction of the moving body (vehicle) 6. This "photographed target area" includes a "first target area" and a "second target area" located above the first target area in the Y-axis direction. Furthermore, the "photographing devices" include a "first photographing device" that photographs the "first target area" and a "second photographing device" that photographs the "second target area". In the example of this embodiment, all or part of the photographing devices 7A-7D correspond to the "first photographing device" that photographs the "first target area", and the photographing device 7E corresponds to the "second photographing device" that photographs the "second target area".

Here, the photographing device 7E (camera 5) that photographs the uppermost part of the slope (around 15 m) is an "installation position variable photographing unit" whose installation position in the direction (Z axis direction) intersecting with the moving direction (X axis direction) relative to the moving body 6 is adjusted. In the "installation position setting step" described later, the installation position in the Z axis direction of the photographing device 7E (second photographing device) on the moving body (vehicle) 6 is set (adjusted).

Figures 9(a), (b), and (c) are diagrams showing an example of the configuration of the variable installation position imaging unit. The direction perpendicular to the paper surface in the figure is the X-axis direction, i.e., the direction of movement of the moving body 6, and the rightward direction in the figure is the Z-axis direction, i.e., the direction in which the slope of the target object (target object including a slope) exists. Note that the configuration of the variable installation position imaging unit (configuration of the installation position adjustment mechanism) is not limited to the examples shown in Figures 9(a) to (c), and various design modifications are possible.

In FIG. 9(a), two camera mounting parts 71 and 72 are provided on the ceiling of the moving body 6, spaced apart in the Z-axis direction, and the installation position in the direction (Z-axis direction) intersecting the moving direction (X-axis direction) relative to the moving body 6 is adjusted depending on which of the two camera mounting parts 71 and 72 the imaging device 7E (camera 5) is attached to. When the imaging device 7E (camera 5) is attached to the camera mounting part 71, the distance to the target slope (target including a slope) can be reduced, and when the imaging device 7E (camera 5) is attached to the camera mounting part 72, the distance to the target slope (target including a slope) can be increased (two-stage setting is possible). In addition, the imaging devices (camera mounting parts 71 and 72) may be moved not only in the Z-axis direction but also in the Y-axis direction (position adjustment may be possible). FIG. 9(a) illustrates an example in which the imaging device 7E (camera 5) is attached to the camera mounting part 71.

In FIG. 9(b), three camera mounting parts 71, 72, 73 spaced apart in the Z-axis direction are provided on the ceiling of the moving body 6, and the installation position in the direction (Z-axis direction) intersecting the direction of movement (X-axis direction) relative to the moving body 6 is adjusted depending on which of the three camera mounting parts 71, 72, 73 the imaging device 7E (camera 5) is attached to. When the imaging device 7E (camera 5) is attached to the camera mounting part 71, the distance to the target slope (target including a slope) can be made the closest, when the imaging device 7E (camera 5) is attached to the camera mounting part 72, the distance to the target slope (target including a slope) can be made the furthest, and when the imaging device 7E (camera 5) is attached to the camera mounting part 73, the distance to the target slope (target including a slope) can be set to an intermediate position (three-stage setting is possible). In addition, the photographing devices ( camera mounting parts 71, 72, 73) may be moved not only in the Z-axis direction but also in the Y-axis direction (their positions may be adjustable). Figure 9(b) shows an example in which the photographing device 7E (camera 5) is mounted on the camera mounting part 73.

In FIG. 9(c), a camera mounting part 74 with a sliding mechanism capable of sliding in the Z-axis direction is provided on the ceiling of the moving body 6, and the installation position in the direction (Z-axis direction) intersecting the moving direction (X-axis direction) relative to the moving body 6 is adjusted (stepless setting is possible) by attaching the imaging device 7E (camera 5) to the camera mounting part 74 with a sliding mechanism and sliding it. FIG. 9(c) illustrates an example in which the imaging device 7E (camera 5) is attached to the camera mounting part 74 with a sliding mechanism and slid to be farthest from the target slope (target including a slope). In addition, the installation position may be adjusted not only in the direction (Z-axis direction) intersecting the moving direction (X-axis direction) relative to the moving body 6, but also in the Y-axis direction by tilting the imaging device and attaching it to the camera mounting part 74 with a sliding mechanism and sliding it.

The imaging devices 7A, 7B, 7C, and 7D (cameras 1, 2, 3, and 4) other than the imaging device 7E (camera 5) are "fixed installation position imaging units" whose installation positions are fixed in the direction (Z-axis direction) that intersects with the moving direction (X-axis direction) relative to the moving body 6.

Therefore, the variable installation position photographing unit (photographing device 7E (camera 5)) photographs a relatively high position on the slope of the object, and the fixed installation position photographing unit (photographing devices 7A to 7D (cameras 1 to 4)) photographs a relatively low position on the slope of the object.

Furthermore, of the five imaging devices 7A-7E (cameras 1-5), imaging devices 7A and 7B (cameras 1 and 2) are so-called "normal cameras" in which the Scheimpflug angle is not set, while imaging devices 7C-7E (cameras 3-5) are so-called "Scheimpflug cameras" in which the Scheimpflug angle is set. A Scheimpflug camera is an example of a special camera in which the entire plane facing each other diagonally appears in focus, and may be read as a "tilt camera" or a "tilt-mount camera."

The photographing devices 7A to 7E (cameras 1 to 5) have an image sensor and a photographing lens, and in a Scheimpflug camera, the Scheimpflug angle θ', which indicates the angle between the perpendicular to the sensor surface of the image sensor and the central axis (optical axis) of the photographing lens, is set to a value other than 0 degrees. Conversely, in a normal camera, the perpendicular to the sensor surface of the image sensor and the central axis (optical axis) of the photographing lens are coaxial (parallel) and have an angle of 0 degrees.

The imaging devices 7C to 7E (cameras 3 to 5), which are Scheimpflug cameras, have Scheimpflug angles that correspond to the imaging position in the height direction of the slope of the object. For example, in this embodiment, the Scheimpflug angle of the imaging devices 7C and 7D (cameras 3 and 4) (imaging units with fixed installation positions) is set to 0.7 degrees, and the Scheimpflug angle of the imaging device 7E (camera 5) (imaging unit with variable installation position) is set to 1.1 degrees. Note that the Scheimpflug angles of the imaging devices 7C to 7E (cameras 3 to 5) are merely examples and are set appropriately depending on the range of inclination angles of the slope of the object. The imaging device used in this embodiment may be equipped with a mechanism that dynamically changes the Scheimpflug angle by adjusting the positional relationship between the image sensor and the imaging lens (the relationship of the optical axis of the imaging lens to the imaging surface of the image sensor). However, in this embodiment, the Scheimpflug angles of the imaging devices 7C to 7E (cameras 3 to 5) are uniquely determined at the time of manufacture, and no mechanism is equipped to dynamically change the Scheimpflug angle (the Scheimpflug angle is invariable when focusing on the camera alone). For example, the imaging device 7E (second imaging device) has a preset Scheimpflug angle in which the optical axis of the imaging lens in the imaging device 7E (second imaging device) is tilted with respect to an axis perpendicular to the imaging surface of the imaging device 7E (second imaging device).

FIG. 10 is a diagram showing an example of adjusting the installation position of the variable installation position photographing unit, i.e., the photographing device 7E (camera 5). FIG. 10 corresponds to an "installation position setting step" for setting (adjusting) the installation position in the Z-axis direction of the photographing device 7E (second photographing device) on the moving body (vehicle) 6. FIG. 10 illustrates an example in which the height of the slope, which is the subject, is 15 m, and the inclination angle of the slope, which is the subject, is approximately 45 degrees to 73.3 degrees (the inclination angle of the slope, which is the subject, is drawn as if it were constant, but this is for convenience of drawing).

The slope of the target object has a variety of angles, from gentle to steep, so even when using a Scheimpflug camera, it can be difficult to obtain an image that is in focus on the slope surface. For example, if the overall image to be included in the report and the image showing the deformation are not in focus, they will not be able to serve as an inspection report. Also, because the crack width is estimated from an image of the crack, the focus affects the error in the crack width estimation results.

In the example of Figure 10, when the slope angle is about 45 degrees to 73.3 degrees and the height of the slope is 15 m or thereabouts, if the horizontal distance from the shooting reference position of the moving body 6 (for example, the installation position (lens object side principal point position) of the shooting device 7E (camera 5) at the right end in Figure 10) to the end of the slope (the "distance from the shooting reference position to the subject" described below) is in the range of 1.75 m to 3.75 m, it may be difficult to capture an image in focus on the surface of the slope, even with the shooting device 7E (camera 5) with the Scheimpflug angle set to 1.1 degrees.

Therefore, in this embodiment, an "installation position setting step" is provided for setting (adjusting) the installation position in the Z-axis direction of the imaging device 7E (second imaging device) on the moving body (vehicle) 6. Specifically, in this embodiment, an XYZ Cartesian coordinate system is used, in which the movement direction of the moving body (vehicle) 6 is the X-axis, the direction intersecting the movement direction of the moving body (vehicle) 6 is the Z-axis, and the direction perpendicular to the X-axis and Z-axis is the Y-axis.

Here, the length from the position where the shooting reference position of the shooting device 7E (second shooting device) installed on the moving body (vehicle) 6 is projected onto the XZ plane on which the moving body (vehicle) 6 moves to the position on the XZ plane where the XZ plane intersects with the object (e.g. a slope) is defined as the "distance from the shooting reference position to the object." Also, the length on the Y axis from the XZ plane to the shooting target area on the object (e.g. a slope) is defined as the "height of the shooting target area." The shooting reference position may be interpreted as the object-side principal point position of the lens.

In this embodiment, the object is a slope, the moving body is a vehicle, and the XZ plane on which the moving body moves is the road surface on which the vehicle moves. In this case, the "distance from the shooting reference position to the object" can be interpreted as the "distance from the position where the shooting reference position of the shooting device installed on the moving body (vehicle) is projected onto the road surface to the end of the slope where the slope meets the road surface."

Then, in the "installation position setting step", the installation position of the imaging device 7E (second imaging device) is set (adjusted) in the Z-axis direction that intersects with the direction of movement of the moving body (vehicle) 6, depending on at least one of the "distance from the imaging reference position to the target object" and the "height of the imaging target area".

The "distance from the shooting reference position to the object" and the "height of the object to be shot" are parameters for determining the theoretical Scheimpflug angle for focusing on the object to be shot, and the installation position of the shooting device 7E (second shooting device) is set (adjusted) according to the difference between the theoretical Scheimpflug angle and the set Scheimpflug angle that is set in advance in the shooting device 7E (second shooting device). Specifically, when the difference between the theoretical Scheimpflug angle and the set Scheimpflug angle is equal to or greater than a specified range, the installation position of the shooting device 7E (second shooting device) is changed in the Z-axis direction (for example, away from the slope of the object) to change the theoretical Scheimpflug angle, thereby bringing the difference between the theoretical Scheimpflug angle and the set Scheimpflug angle within a specified range.

In Figure 10, when the "distance W from the shooting reference position to the target object" is 2.75 m or more and 3.75 m or less, an image focused on the slope surface can be captured even if the shooting device 7E (camera 5) is installed at the right end of Figure 10. However, when the "distance W from the shooting reference position to the target object" is 1.75 m or more and less than 2.75 m, it becomes difficult to capture an image focused on the slope surface if the shooting device 7E (camera 5) is installed at the right end of Figure 10.

Therefore, in this embodiment, the installation position of the imaging device 7E (camera 5) in the direction (Z-axis direction) intersecting with the moving direction (X-axis direction) relative to the moving object 6 is adjusted according to the "distance W from the imaging reference position to the object". When the "distance W from the imaging reference position to the object" is 2.75 m or more and 3.75 m or less, the installation position of the imaging device 7E (camera 5) is adjusted to the right end in FIG. 10. On the other hand, when the "distance W from the imaging reference position to the object" is 1.75 m or more and less than 2.75 m, the installation position of the imaging device 7E (camera 5) is adjusted to the left end in FIG. 10. In this way, the installation position of the imaging device 7E (camera 5) as an installation position variable imaging unit is adjusted in the direction (Z-axis direction) intersecting with the moving direction (X-axis direction) relative to the moving object 6 according to the "distance W from the imaging reference position to the object".

Also, considering that the photographing device 7E (camera 5) is for photographing the highest area of the slope, the photographing device 7E (camera 5) as a position-variable photographing unit has its position adjusted in the direction (Z-axis direction) intersecting with the moving direction (X-axis direction) relative to the moving body 6 depending on the "height of the photographed area (for example, whether the photographed area is the first or second target area)". If the photographing device (camera) is for photographing a low position on the slope, there is little need to make the photographing device (camera) a position-variable photographing unit in the first place, and a fixed-position photographing unit is sufficient. In Figures 10 and 11, the "height of the photographed area" is marked with the symbol D (the height D of the photographed area is set within a maximum range of 15 m). The height D of the photographed area in Figure 11 may be read as the "height of the intersection of the optical axis and the slope".

In this way, the installation position of the imaging device 7E (camera 5) as a position-adjustable imaging unit is set (adjusted) in the Z-axis direction that intersects with the direction of movement of the moving body (vehicle) 6, depending on at least one of the "distance W from the imaging reference position to the target object" and the "height D of the imaging target area."

The installation position of the imaging device 7E (camera 5) as a position-adjustable imaging unit is adjusted in the direction (Z-axis direction) that intersects with the moving direction (X-axis direction) relative to the moving body 6 based on the relationship of the "distance W from the imaging reference position to the target object" to a predetermined distance (whether the distance W is in the range of 1.75m to 3.75m, or whether it is less than 2.75m or 2.75m or more) and the relationship of the "height D of the imaging target area" to a predetermined height (whether it is 15m or close to that, or whether it is the highest slope height, or whether the imaging target area is the first or second target area).

In other words, the installation position of the imaging device 7E (camera 5) in the direction (Z axis direction) intersecting the moving direction (X axis direction) relative to the moving body 6 is adjusted based on whether or not the slope angle, etc., according to the "distance W from the imaging reference position to the object" and the "height D of the imaging target area" exceeds the allowable range for the theoretical value of the Scheimpflug angle calculated from the Scheimpflug angle formula (details will be described later). Keeping the Scheimpflug angle constant refers to one camera, and does not mean that the Scheimpflug angle is made common to the other cameras. Each camera (here, imaging devices 7C to 7E (cameras 3 to 5)) has its own Scheimpflug angle set. In other words, the optimal Scheimpflug angle is set for each camera within the allowable range for the theoretical value of the Scheimpflug angle calculated from the Scheimpflug angle formula described later. In addition, the Scheimpflug angle set for each camera is not changed.

Generally, a mechanism for changing the Scheimpflug angle involves preparing a lens adapter for each Scheimpflug angle and replacing the lens adapter to obtain the desired Scheimpflug angle. In contrast, in this embodiment, a mechanism for moving the imaging device (second imaging device) 7E in the Z-axis direction is provided, eliminating the need to replace the lens adapter and improving work efficiency. In addition, in-focus photography is possible even with a camera that does not have a mechanism for dynamically changing the Scheimpflug angle. Furthermore, since the imaging devices 7C to 7E (cameras 3 to 5) have a structure in which the Scheimpflug angle is fixed, a high level of rigidity can be ensured (guaranteed) for the cameras and lenses.

As already mentioned, the "photographing reference position" is the installation position of the camera on the moving body 6 (in this embodiment, the object-side principal point position of the lens), and may be interpreted as a "predetermined photography reference position", for example, uniquely defined by the driving line of the moving body 6 at the time of shooting. In that case, the "distance W from the photography reference position to the object", that is, the "distance W from the position where the photography reference position of the camera installed on the moving body (vehicle) is projected onto the road surface to the toe of the slope where the slope meets the road surface", may differ depending on, for example, differences in the environment between the slope and the road along which the vehicle is traveling, that is, the presence or absence of a sidewalk, a gutter, or a guardrail. In this embodiment, before photographing the slope, the "distance W from the photographing reference position to the object," that is, the "distance W from the position where the photographing reference position of the photographing device installed on the moving body (vehicle) is projected onto the road surface to the end of the slope where the slope meets the road surface," can be grasped in advance by using the output of the GNSS sensor 8b, checking in advance on a map, etc., driving once before photographing, using the results of the previous photographing, or any other means. Then, once the installation position of the photographing device 7E (camera 5) as the position-variable photographing unit is adjusted, it is not moved during photographing, and photographing is performed with the photographing conditions related to focus fixed (constant).

Generally, when setting the Scheimpflug angle in normal photography, the Scheimpflug angle is set according to the inclination angle of the subject. However, considering the effort required for photography, the inventors have noticed that when photographing slopes, there is a demand to artificially change the set Scheimpflug angle (to cover a Scheimpflug angle with a certain range) rather than simply changing the Scheimpflug angle setting as appropriate according to the inclination angle of the slope.

As mentioned above, the "distance W from the shooting reference position to the object," i.e., the "distance W from the position where the shooting reference position of the shooting device installed on the moving body (vehicle) is projected onto the road surface to the end of the slope where the slope meets the road surface," may vary depending on the environment of the shooting site (presence or absence of sidewalks, gutters, and guardrails). In this embodiment, the slope angle is approximately 45 degrees to 73.3 degrees, and the "distance W from the shooting reference position to the object," i.e., the "distance W from the position where the shooting reference position of the shooting device installed on the moving body (vehicle) is projected onto the road surface to the end of the slope where the slope meets the road surface," is 1.75 m to 3.75 m. It is assumed that shooting will be performed without changing the Scheimpflug angle set for each of the shooting devices 7C to 7E (cameras 3 to 5) (0.7 degrees for shooting devices 7C and 7D (cameras 3 and 4), and 1.1 degrees for shooting device 7E (camera 5)).

However, in this embodiment, for the imaging devices 7C and 7D (cameras 3 and 4), in-focus imaging is possible even when the Scheimpflug angle is 0.7 degrees. However, for the imaging device 7E (camera 5) with a Scheimpflug angle of 1.1 degrees, when the "distance W from the imaging reference position to the object," that is, the "distance W from the position where the imaging reference position of the imaging device installed on the moving body (vehicle) is projected onto the road surface to the end of the slope where the slope meets the road surface," approaches 1.75 m (when getting too close to the slope), it may become difficult to capture in focus. To solve this problem, the imaging device 7E (camera 5) is used as a "position-variable imaging unit" and its installation position on the moving body 6 is moved in the direction in which the distance W increases (the direction moving away from the slope (end of the slope) on the Z-axis), so that in-focus imaging can be achieved even with the imaging device 7E (camera 5) with a Scheimpflug angle of 1.1 degrees.

In this sense, the theoretical Scheimpflug angle of the photographing device 7E (camera 5) is changed in a pseudo manner by adjusting the installation position of the photographing device 7E (camera 5) as a position-variable photographing unit in the direction (Z-axis direction) intersecting with the moving direction (X-axis direction) relative to the moving body 6. That is, the set Scheimpflug angle of the photographing device 7E (camera 5) itself is the same (although it does not have an angle change unit), but the theoretical Scheimpflug angle is "pseudo" changed by adjusting the slide position of the photographing device 7E (camera 5). In other words, by adjusting the installation position of the photographing device 7E (camera 5) as a position-variable photographing unit in the direction (Z-axis direction) intersecting with the moving direction (X-axis direction) relative to the moving body 6 and lowering the elevation angle of the photographing device 7E (camera 5), the "focus plane angle" of the photographing device 7E (camera 5) as a position-variable photographing unit can be changed (adjusted) without changing the set Scheimpflug angle of the photographing device 7E (camera 5) itself.

Here, the "focus plane" of the imaging device means a plane that includes the focus position (point) on the optical axis of the lens of the imaging device. When the set Scheimpflug angle is 0°, the imaging surface (object-side principal plane) of the imaging device (image sensor) and the focus plane are parallel, and when the set Scheimpflug angle is other than 0°, the focus plane is angled with respect to a plane parallel to the object-side principal plane according to the set Scheimpflug angle. In this embodiment, the angle of the focus plane with respect to this plane parallel to the object-side principal plane is defined as the "focus plane angle." And if the focusing distance changes, the angle of the focus plane changes even when the Scheimpflug angle set in the imaging device itself is fixed. Therefore, as described above, by adjusting the installation position of the image capturing device 7E (camera 5) as the installation position variable image capturing unit in the direction (Z axis direction) intersecting with the moving direction (X axis direction) relative to the moving body 6 and lowering the elevation angle of the image capturing device 7E (camera 5), it is possible to change (adjust) the "focal plane angle" of the image capturing device 7E (camera 5) as the installation position variable image capturing unit while keeping the Scheimpflug angle of the image capturing device 7E (camera 5) itself fixed. Note that in FIG. 11, the "focal plane angle" of the image capturing device 7E (camera 5) as the installation position variable image capturing unit is depicted with the symbol β. Also, in FIG. 11 and FIG. 12, the camera and the image capturing lens (lens barrel) with the Scheimpflug angle set are depicted as schematics, and the "lens object side principal point position" is depicted there. The "lens object side principal point position" is the intersection point of the optical axis and the object side principal plane.

In FIG. 11, the plane that passes through the lens image-side principal point position and is perpendicular to the optical axis of the photographing lens is the "image-side principal plane", and the plane that passes through the lens object-side principal point position and is perpendicular to the optical axis of the photographing lens is the "object-side principal plane". The "image-side principal plane" and the "object-side principal plane" are perpendicular to the optical axis of the photographing lens. When the Scheimpflug angle is not set (when the Scheimpflug angle is 0°), the optical axis of the photographing lens is perpendicular to the "imaging surface of the image sensor", so the "imaging surface of the image sensor" and the "object-side principal plane" are parallel. On the other hand, when the Scheimpflug angle is set, the optical axis of the photographing lens is no longer perpendicular to the "imaging surface of the image sensor", so the "imaging surface of the image sensor" and the "object-side principal plane" are no longer parallel. When the Scheimpflug angle is not set (when the Scheimpflug angle is 0°), the "focus plane" depicted in FIG. 11 is parallel to the "imaging surface of the image sensor", but when the Scheimpflug angle is set, it is significantly tilted with respect to the "imaging surface of the image sensor". In this way, the focal range on the slope is wider by being closer to the angle of the slope than when the Scheimpflug angle is not set.

In this way, according to this embodiment, when photographing a slope (photographing area of the object) within a predetermined inclination angle range (e.g., 45 degrees to 73.3 degrees) within the expected distance (e.g., 1.75 m to 3.75 m) from the toe of the slope (reference position of the object) to the photographing reference position, it is possible to photograph the slope without changing the Scheimpflug angle set on the camera. More specifically, when the "distance from the photographing reference position to the object" is close to within the predetermined range, by changing the installation position of the camera photographing the photographing target area located at a high position (e.g., the second target area) so that the "distance from the photographing reference position to the object" is increased, it is possible to obtain a focused image by photographing the photographing target area located at a high position (e.g., the second target area) without changing the Scheimpflug angle of the camera photographing the photographing target area located at a high position (e.g., the second target area).

Next, using the concept (keyword) of "tolerance of deviation from the theoretical value of the Scheimpflug angle," it will be explained that for the imaging devices 7C and 7D (cameras 3 and 4), in-focus photography is possible even when shooting with a Scheimpflug angle of 0.7 degrees, but for the imaging device 7E (camera 5) with a Scheimpflug angle of 1.1 degrees, when the "distance W from the imaging reference position to the subject," in other words, "the distance W from the position where the imaging reference position of the imaging device installed on a moving body (vehicle) is projected onto the road surface to the end of the slope where the slope meets the road surface," becomes close to 1.75 m (too close to the slope), and it may become difficult to shoot in focus.

Figures 11, 12, and 13 are first, second, and third conceptual diagrams showing an example of the allowable deviation from the theoretical Scheimpflug angle value.

In Fig. 11, the lens direction α (°) is determined based on the lens position and the inclination angle of the slope, in accordance with the criteria in Table 1 below. The direction is determined taking into consideration the ability to observe slopes with a height of 15 m or more, and the shooting range of the lower camera.

Sim, calculate the object distance under the conditions. The lens direction is set to 15 m or near there, and the object distance is calculated for a total of six levels, including two levels of each inclination angle (73.3°, 45°) and three levels of distance W from the shooting reference position to the object (1.75 m, 2.75 m, 3.75 m). The object distance _Z0 is calculated based on the following formulas (A), (B), (C), and (D) when the parameters θ, A, α, C, W, H, and γ in FIG. 11 are defined as follows:

θ: Angle between the optical axis and the slope [°]
A: Angle of inclination of the slope [°]
α: Lens orientation [°]
C: Distance between the end of the lens and the edge of the lens [m]
W: Distance from the shooting reference position to the object [m]
H: Lens height [m]
γ: Angle between the line connecting the principal point of the object and the toe of the slope and the line perpendicular to the ground [°]
D: Height of the target area [m]
_Z0 : Object distance [m]

11 and 12, the Scheimpflug angle θ' of the camera is calculated based on the following formula (E): In Fig. 12, the inclination angle A of the slope is set to about 45° to 73.3°, the distance W from the shooting reference position to the target object is set to 1.75 m to 3.75 m, the lens height H is set to 2.3 m, and the movement amount (slide amount in the Z-axis direction) of the lens for shooting near 15 m (camera 5) is set to 1 m.

θ': Scheimpflug angle [°]
f: 15m focal length of lens for shooting near field (lens focal length of camera 5) [m]
_Z0 : Object distance [m]
θ: Angle between the optical axis and the slope [°]

In this embodiment, the "lens object side principal point position" is the shooting reference position of the imaging device. Therefore, the lens height H [m] above means the height from the road surface (XZ plane) on which the moving body (vehicle) moves to the "lens object side principal point position." In other words, the lens height H [m] means the height from the road surface (XZ plane) on which the moving body (vehicle) moves to the "shooting reference position."

In this manner, the theoretical value of the Scheimpflug angle θ' when the slope to be inspected is photographed is obtained. That is, by substituting the object distance _Z0 obtained by the above formulas (A) to (D) into formula (E) for obtaining the theoretical value of the Scheimpflug angle, the theoretical value of the Scheimpflug angle θ' when the slope to be inspected is photographed can be obtained.

Here, the more the actual Scheimpflug angle that is set deviates from the theoretical value, the narrower the depth of field (see FIG. 12) defined by the "meander allowance before -" and the "meander allowance after +" becomes, and the higher the possibility that an unfocused part will be captured in the captured image. As an example, if the actual Scheimpflug angle that is set is within a predetermined range (±0.2°) from the theoretical value, it can be determined that the depth of field is sufficient for performing a search, but if the actual Scheimpflug angle that is set is outside the predetermined range from the theoretical value (exceeding ±0.2°), the higher the possibility that an unfocused part will be captured in the captured image. Here, the predetermined range of "±0.2°" is merely an example, and various design changes are possible. For example, the predetermined range that is the criterion for determining whether or not the depth of field required for inspection is within the range can be set appropriately according to the image resolution and inspection accuracy (required specifications) at the time of inspection.

If the actual Scheimpflug angle that is set falls outside the specified range from the theoretical value (if it exceeds ±0.2° as is), the photographing device 7E (camera 5) as the position-adjustable photographing unit is moved in the Z-axis direction (horizontal direction) (away from the slope) to adjust the actual Scheimpflug angle that is set so that it falls within the specified range from the theoretical value (±0.2°).

Figure 13(a) shows the theoretical value of the Scheimpflug angle when the Z-axis position of the image capture device 7E (camera 5) as a position-adjustable image capture unit is fixed, and Figure 13(b) shows the theoretical value (improvement) of the Scheimpflug angle when the Z-axis position of the image capture device 7E (camera 5) as a position-adjustable image capture unit is adjusted. In Figures 13(a) and (b), the horizontal axis represents the slope angle and the vertical axis represents the theoretical value of the Scheimpflug angle, and the theoretical value of the Scheimpflug angle is plotted against the slope angle. Figures 13(a) and (b) also plot the theoretical values of the Scheimpflug angle for cameras 2, 4, and 5 (CAM2, CAM4, CAM5) under two conditions where the distance W from the image capture reference position to the object is 1.75 m and 3.75 m.

The theoretical range of the Scheimpflug angle for CAM2, which photographs the middle of the slope, is within a range of 0.37° from -0.27° to 0.1°, and since the difference from the theoretical value is small even when the Scheimpflug angle is set to 0°, focus shifts are unlikely to occur at the top and bottom of the shooting range even if the shooting conditions (slope or distance W from the shooting reference position to the subject) change. Similarly, the theoretical range of the Scheimpflug angle for CAM4 is within a range of 0.39° from -0.69° to -0.3°, and since the difference from the theoretical value is small even when the Scheimpflug angle is set to -0.7°, there are no focus problems. Similarly, there are no focus problems with CAM1 and CAM3.

However, because CAM5 is looking at a high place, there is a large variation in distance depending on the shooting conditions (inclination and the distance W from the shooting reference position to the subject), and because the focal length of the lens is large, the theoretical value of the Scheimpflug angle is in a wide range of 0.87°, from -1.82° to -0.95°, resulting in problems with focusing.

As a result, by attaching a 1m sliding mechanism in the horizontal direction (Z-axis direction) as shown in Figure 13(b), when the distance to the end of the slope is close (distance W from the shooting reference position to the target object is small) of 1.75m, the CAM5 can be moved away with the sliding mechanism, so that the effective distance to the end of the slope (distance W from the shooting reference position to the target object) is only between 2.75m and 3.75m, narrowing the range of the theoretical value of the Scheimpflug angle (see the bold frame in Figure 13(b)). As a result, when the distance to the end of the slope is close (distance W from the shooting reference position to the target object is small) between 1.75m and 3.75m, it is possible to eliminate the problem of being unable to focus in the slope inclination range of 45° to 73.3°. In other words, it is possible to obtain an image that is in focus over a wide area of the slope.

If it is detected that the position of the camera installed while photographing the slope exceeds the allowable range based on the theoretical Scheimpflug angle (when the driving line needs to be changed, or the photographing distance changes due to the shape of the slope, etc.), a warning may be issued to indicate that further photographs will not be able to capture images suitable for inspection, photographing may be stopped, or an instruction may be given to retake photographs due to the detected error.

In the imaging method of this embodiment, the imaging device captures an area to be imaged of an object by dividing the area to be imaged into a plurality of imaging areas along the direction of movement of the moving object. The information processing method of this embodiment includes a generating step of generating an image of the area to be imaged by stitching together the images of the plurality of imaging areas captured by the imaging method described above. The information processing method of this embodiment also includes an evaluation step of evaluating the image of the area to be imaged obtained by stitching together the images of the plurality of imaging areas. The information processing method of this embodiment also includes a display step of displaying the image of the area to be imaged obtained by stitching together the images of the plurality of imaging areas. Below, one aspect of these information processing methods will be described with reference to Figs. 14 to 17.

Figure 14 is a sequence diagram showing an example of display processing in a status inspection system.

Below, the sequence between the evaluation device 3 and the data management device 5 will be explained, but the same applies to the sequences between the data acquisition device 9, the terminal device 1100, and the terminal device 1200 and the data management device 5.

When the user of the evaluation device 3 specifies a folder, the reception unit of the evaluation device 3 accepts the selection of the target data (step S91). Alternatively, the user of the evaluation device 3 may select an arbitrary position in the map information managed by the map data management unit of the evaluation device 3, and the reception unit of the evaluation device 3 may accept the selection of position information in the map information.

Next, the communication unit of the evaluation device 3 transmits a request for an input/output screen related to the target data selected in step S91 to the data management device 5, and the communication unit of the data management device 5 receives the request transmitted from the evaluation device 3 (step S92). This request includes the folder name selected in step S91. Alternatively, this request may include location information in the map information.

Next, the storage/reading unit of the data management device 5 searches the processing data management DB 5003 using the folder name included in the request received in step S92 as a search key, thereby reading out image data associated with the folder name included in the request. Alternatively, the storage/reading unit searches the acquisition data management DB using the location information included in the request received in step S92 as a search key, thereby reading out image data associated with the location information included in the request.

The generating unit of the data management device 5 generates an input/output screen including the image data based on the image data read by the storing/reading unit (step S93). This input/output screen is a screen that accepts an instruction operation to generate an image showing a specific position in a luminance image showing a slope.

The communication unit of the data management device 5 transmits input/output screen information related to the input/output screen generated in step S93 to the evaluation device 3, and the communication unit of the evaluation device 3 receives the input/output screen information transmitted from the data management device 5 (step S94). Step S94 is an example of a generation reception screen transmission step.

Next, the display control unit of the evaluation device 3 displays the input/output screen received in step S94 on the display (step S95). The reception unit of the evaluation device 3 receives a predetermined input operation by the user on the displayed input/output screen. This input operation includes an instruction operation to generate an image showing a specific position in a luminance image showing a slope. Step S95 is an example of a reception step.

The communication unit of the evaluation device 3 transmits input information related to the input operation received by the reception unit to the data management device 5, and the communication unit of the data management device 5 receives the input information transmitted from the evaluation device 3 (step S96). This input information includes instruction information that instructs the generation of an image showing a specific position in the luminance image showing the slope.

The generating section of the data management device 5 generates a display image using the image data read by the storing and reading section in step S93 based on the received input information (step S97). This display image includes a surface display image including a surface image showing the surface of the slope and a surface position image showing a specific position in the surface image, and a cross-section display image including a cross-section image showing the cross-section of the slope and a cross-section position image showing a specific position in the cross-section image. Step S97 is an example of an image generating step.

The communication unit of the data management device 5 transmits the display image generated in step S97 to the evaluation device 3, and the communication unit of the evaluation device 3 receives the display image transmitted from the data management device 5 (step S98). Step S98 is an example of a display image transmission step.

The display control unit of the evaluation device 3 displays the display image received in step S98 on the display (step S99). Step S99 is an example of a display step.

FIG. 14 shows the sequence of the display process between the evaluation device 3 and the data management device 5, but the evaluation device 3 may execute the display process independently.

In this case, steps S92, 94, 96, and 98 relating to data transmission and reception are omitted, and the evaluation device 3 can perform the same display processing as in FIG. 14 by independently executing steps S91, 93, 95, 97, and 99. The data acquisition device 9, terminal device 1100, and terminal device 1200 can also independently execute display processing like the evaluation device 3.

Generation of a surface display image based on an operation of specifying a specific position Fig. 15 is an explanatory diagram of an operation on a display screen of a state inspection system. Fig. 15 shows an input/output screen 2000 displayed on the display of the evaluation device 3 in step S95 of the sequence diagram shown in Fig. 14, but the same is true for the input/output screen 2000 displayed on the respective displays of the data acquisition device 9, the terminal device 1100, and the terminal device 1200.

The display control unit of the evaluation device 3 displays an input/output screen 2000 including a designation reception screen 2010 that receives a designation operation for designating a specific position in a luminance image showing a slope, and a generation reception screen 2020 that receives an instruction operation for instructing the generation of an image showing a specific position on the slope.

The display control unit displays a surface image 2100 showing the surface of the slope on the specification reception screen 2010, and also displays a pointer 2300 operated by a pointing device on the surface image 2100.

The surface image 2100 is a luminance image read out from the captured image data in step S92 of FIG. 14, and the display control unit displays the surface image 2100 in association with the captured images 1 and 2 shown in FIG. 5 and the X-axis direction and Y-axis direction shown in the captured image data 7A shown in FIG. 6(a).

The display control unit displays the generation reception screen 2020, which includes a specified position confirmation button 2400, a deformation confirmation button 2410, a deformation sign confirmation button 2420, a front view analysis button 2430, a front view comparison button 2440, a cross-sectional view analysis button 2450, and a cross-sectional view comparison button 2460. The deformation confirmation button 2410, the deformation sign confirmation button 2420, the front view analysis button 2430, the front view comparison button 2440, the cross-sectional view analysis button 2450, and the cross-sectional view comparison button 2460 are buttons that instruct the generation of an image showing a specific position on the slope, with the position of a part in the surface image 2100 or the cross-sectional image 2200 that satisfies a specified condition as the specific position.

The designated position confirmation button 2400 is a button that instructs the user to confirm a specific position on the slope specified on the designation acceptance screen 2010 and generate an image showing the specific position on the slope. The designated position confirmation button 2400 may determine not only a specific position specified on the designation acceptance screen 2010, but also a specific position that has been specified by a judgment unit or the like and displayed on the designation acceptance screen 2010.

The Deformation Confirmation button 2410 is a button that instructs the system to generate an image showing a specific position on the slope, with a position indicating a deformation of the slope set as the specific position, and the Deformation Sign Confirmation button 2420 is a button that instructs the system to generate an image showing a specific position on the slope, with a position indicating a deformation of the slope set as the specific position.

The front view analysis button 2430 is a button that instructs the system to generate an image showing a specific position on the slope by specifying a portion obtained by analyzing the surface image 2100 as the specific position, and the front view comparison button 2440 is a button that instructs the system to generate an image showing a specific position on the slope by specifying a portion obtained by comparing the surface image 2100 with another image as the specific position.

The cross-sectional view analysis button 2450 is a button that instructs the system to generate an image showing a specific position on the slope by specifying a portion obtained by analyzing the cross-sectional image (described later) as the specific position, and the cross-sectional view comparison button 2460 is a button that instructs the system to generate an image showing a specific position on the slope by specifying a portion obtained by comparing the cross-sectional image with another image as the specific position.

FIG. 16 is a flowchart showing an example of the information processing method. FIG. 16(a) shows the processing in the evaluation device 3, and FIG. 16(b) shows the processing in the data management device 5.

When a specific position on the surface image 2100 is pointed to by the pointer 2300, the reception unit of the evaluation device 3 receives the pointing operation (step S101), and when the designated position confirmation button 2400 is operated, the reception unit receives the operation (step S102).

Next, the judgment unit of the evaluation device 3 detects the XY coordinates of the pointed position in the surface image 2100 as a specific position (step S103). This specific position may indicate a point in the XY coordinates, or may indicate an area.

Next, the communication unit of the evaluation device 3 transmits input information related to the input operation received by the reception unit 32 to the data management device 5 (step S104). This input information includes designation information for designating a specific position in XY coordinates based on a pointing operation using the pointer 2300, and instruction information for instructing the generation of an image showing the specific position on the slope based on the operation of the designated position confirmation button 2400.

The communication unit of the data management device 5 receives the input information sent from the evaluation device 3, and the generation unit uses the image data shown in FIG. 6(a) based on the instruction information and specification information contained in the received input information to generate a surface position image that overlaps with the XY coordinates of the specific position by superimposing it on the surface image to generate a surface display image (step S105). The surface position image does not necessarily have to completely match the XY coordinates of the specific position, as long as it overlaps with the XY coordinates of the specific position.

Then, the generating unit generates a cross-sectional image corresponding to the X-coordinate of the specific position using the image data shown in FIG. 6(a) and the distance measurement data shown in FIG. 6(b) (step S106). If the distance measurement data shown in FIG. 6(b) does not include the X-coordinate of the specific position, the generating unit generates a cross-sectional image based on data in the vicinity of the X-coordinate of the specific position included in the distance measurement data shown in FIG. 6(b).

In step S106, the generating unit generates a cross-sectional image of a cross-section including the Z-axis direction and the vertical direction shown in FIG. 5, but it may also generate a cross-sectional image of a cross-section including the Z-axis direction and a direction inclined from the vertical direction, or a cross-sectional image of a cross-section including a direction inclined from the Z-axis direction.

The generation unit generates a cross-sectional position image that overlaps with the Y coordinate of the specific position by superimposing it on the edge line of the cross-sectional image, and generates a cross-sectional display image (step S107).

The communication unit transmits the surface display image generated in step S105 and the cross-sectional display image generated in step S107 to the evaluation device 3 (step S108).

Then, as shown in steps S98 and S99 of FIG. 14, the communication unit of the evaluation device 3 receives the surface display image and cross-sectional display image transmitted from the data management device 5, and the display control unit of the evaluation device 3 displays the received surface display image and cross-sectional display image on the display.

FIG. 17 is a diagram showing an example of a display screen after the processing shown in FIG. 14. FIG. 17 shows an input/output screen 2000 that is displayed on the display of the evaluation device 3 in step S99 of the sequence diagram shown in FIG. 14.

The display contents of the generation reception screen 2020 are the same as those in FIG. 15, but the display contents of the specification reception screen 2010 are different from those in FIG. 15.

The display control unit of the evaluation device 3 causes the specification reception screen 2010 to display a surface display image 2150 including a surface image 2100 showing the surface of the slope and a surface position image 2110 showing a specific position in the surface image 2100, and a cross-section display image 2250 including a cross-section image 2200 showing the cross-section of the slope and a cross-section position image 2210 showing a specific position in the cross-section image 2200.

The display control unit displays the cross-sectional image 2200 in association with the Y-axis direction and Z-axis direction shown in FIG. 5.

In this way, in the imaging method and information processing method of this embodiment, the length from the position where the imaging reference position of the imaging device installed on the moving body is projected onto the XZ plane along which the moving body moves to the position on the XZ plane where the XZ plane intersects with the object is defined as the distance from the imaging reference position to the object, and the length on the Z axis from the XZ plane to the imaging target area of the object is defined as the height of the imaging target area. Then, in the installation position setting step, the installation position is adjusted in the Z axis direction that intersects with the movement direction according to at least one of the distance from the imaging reference position to the object and the height of the imaging target area. This makes it possible to obtain an image that is in focus over a wide area of the object (e.g. a slope).

　The invention disclosed herein has been described in detail above, but it is clear to those skilled in the art that the invention disclosed herein is not limited to the embodiments described herein. The invention disclosed herein can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the claims. Therefore, the description of the disclosure is intended as an illustrative example and does not impose any limiting meaning on the invention disclosed herein.

For example, in the above embodiment, five photographing devices (cameras) are prepared as the multiple photographing devices (cameras), and only one photographing device (camera) that photographs the uppermost part of the slope (object including the slope) that is the target is designated as a "variable installation position photographing unit (variable installation position camera)", and the remaining four photographing devices (cameras) are designated as "fixed installation position photographing units (fixed installation position cameras)". However, there is a degree of freedom in the number of photographing devices (cameras), and there is also a degree of freedom in how to arrange the "variable installation position photographing units (variable installation position cameras)" and the "fixed installation position photographing units (fixed installation position cameras)". For example, it is also possible to have only one "variable installation position photographing unit (variable installation position camera)" as the photographing device (camera), and to have six or more photographing devices (cameras) and two or more of them as "variable installation position photographing units (variable installation position cameras)".

1 Condition inspection system 6 Mobile body (vehicle)
7. Camera
7A Photographing device (camera 1, fixed position photographing unit, first photographing device)
7B Imaging device (camera 2, fixed position imaging unit, first imaging device)
7C Photographing device (camera 3, fixed position photographing unit, first photographing device)
7D Photographing device (camera 4, fixed position photographing unit, first photographing device)
7E Photographing device (camera 5, variable installation position photographing unit, second photographing device)
60 Mobile system (photography system)

Patent No. 6498488

Claims (9)

1. A method for photographing an object while moving using a photographing device installed on a moving body, comprising:
   The photographing device photographs a photographing target area of the object located in a direction intersecting a moving direction,
   an installation position setting step of setting an installation position of the photographing device in the moving body;
   Using an XYZ Cartesian coordinate system in which the moving direction of the moving body is the X axis, the direction intersecting the moving direction is the Z axis, and the direction perpendicular to the X and Z axes is the Y axis, the length from the position where the shooting reference position of the shooting device installed on the moving body is projected onto the XZ plane along which the moving body moves to the position on the XZ plane where the XZ plane intersects with the object is defined as the distance from the shooting reference position to the object, and the length on the Y axis from the XZ plane to the shooting target area of the object is defined as the height of the shooting target area,
   The installation position setting step includes:
   adjusting the installation position in the Z-axis direction intersecting with the movement direction according to at least one of a distance from the shooting reference position to the object and a height of the shooting target area.
2. The photographing method according to claim 1, characterized in that the distance from the photographing reference position to the object and the height of the photographing target area are parameters for calculating a theoretical Scheimpflug angle for focusing on the photographing target area, and the installation position is adjusted according to a difference between the theoretical Scheimpflug angle and a set Scheimpflug angle that is preset in the photographing device.
3. The imaging target area includes a first target area and a second target area located above the first target area in the Y-axis direction,
   the photographing device includes a first photographing device that photographs the first target area and a second photographing device that photographs the second target area;
   3. The photographing method according to claim 1, wherein the installation position setting step includes adjusting the installation position of the second photographing device.
4. The photographing method according to claim 3, characterized in that a set Scheimpflug angle is preset in the second photographing device, in which an optical axis of a photographing lens in the second photographing device is tilted with respect to an axis perpendicular to an imaging surface of the second photographing device.
5. The object is a slope,
   the moving object is a vehicle,
   the XZ plane on which the moving body moves is a road surface on which the vehicle moves,
   3. The photographing method according to claim 1, wherein the distance from the photographing reference position to the object is the distance from a position where the photographing reference position of the photographing device installed on the moving body is projected onto the road surface to the toe of the slope where the slope meets the road surface.
6. 3. The method according to claim 1, wherein the image capturing device captures an image of a target area of the object by dividing the target area into a plurality of image capturing areas along a moving direction of the moving body.
7. An information processing method comprising: a generating step of generating a captured image of the subject area by stitching together the captured images of the plurality of areas captured by the imaging method according to claim 6.
8. The information processing method according to claim 7 , further comprising an evaluation step of evaluating the captured image of the target area obtained by joining together the captured images of the plurality of shooting areas.
9. The information processing method according to claim 7 , further comprising a display step of displaying a captured image of the subject area obtained by stitching together the captured images of the plurality of shooting areas.

Images