JP2017142613A - Information processing device, information processing system, information processing method and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To connect information corresponding to a plurality of images including overlapping areas with high accuracy.SOLUTION: An information processing device includes: an acquisition part for acquiring stereo picked-up images of at least two or more frames having an overlapping imaging area of prescribed width imaged by a stereo camera, and three-dimensional information data of two or more frames having three-dimensional information calculated the respective picked-up images; a feature point extraction part for extracting feature points from the three-dimensional information data and the picked-up images; a weighting determination part for determining weighting to the three-dimensional information data and the picked-up images on the basis of the number of extracted feature points which is equal to or more than a prescribed threshold; and an image processing part for performing connection processing for connecting two frames of at least either the three-dimensional information data or the picked-up images so as to overlap overlapping imaged areas on the basis of the three-dimensional information data of the two frames and the picked-up images of the two frames, and the weighting determined by the weighting determination part.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing system, an information processing method, and an information processing program.

  For safety management, it is required to inspect the road (road surface) unevenness and whether the white line has disappeared. Conventionally, in order to confirm the state of the road, visual confirmation or confirmation using a laser device has been performed, but the judgment criteria are likely to vary by visual inspection, which is very expensive when using a laser device. A dedicated vehicle was needed. In view of this, a stereo camera is installed in a vehicle, and a road state is detected from a luminance image or a parallax image taken by the stereo camera.

  For example, in Patent Document 1, distance data of an object is calculated based on the parallax of the object displayed on an image data pair obtained by capturing a scene in front of the host vehicle with a stereo camera, and the image data and A stereo-type vehicle exterior monitoring device that recognizes a traveling environment based on distance data is disclosed.

  However, when a region that cannot be captured at once is captured in multiple times and a plurality of captured images are connected to generate a captured image of the entire region, the captured images may not be connected accurately. .

  The present invention has been made in view of the above, and is an information processing apparatus, an information processing system, an information processing method, and an information processing program capable of accurately joining information corresponding to a plurality of images including overlapping regions. The purpose is to provide.

  In order to solve the above-described problems and achieve the object, the present invention provides at least two frames or more of stereo captured images having overlapping imaging regions of a predetermined width captured by a stereo camera, and each of the captured images. An acquisition unit that acquires two or more frames of three-dimensional information data having the calculated three-dimensional information; a feature point extraction unit that extracts feature points from the three-dimensional information data and the captured image acquired by the acquisition unit; A weight determining unit for determining weights for the three-dimensional information data and the captured image based on the number of feature points equal to or greater than a predetermined threshold extracted from the three-dimensional information data and the captured image by the feature point extracting unit; The two-dimensional three-dimensional information data acquired by the acquisition unit, the two frames of the captured image, and the weight determination unit are determined. Based on the weighting, with a, an image processing unit that performs boundary processing of connecting to the imaging regions overlap to the overlapping at least one of two frames of the three-dimensional information data and the captured image.

  According to the present invention, there is an effect that information corresponding to a plurality of images including overlapping regions can be joined together with high accuracy.

FIG. 1 is a block diagram illustrating a configuration example of an information processing system according to the embodiment. FIG. 2 is a diagram schematically showing a state in which the in-vehicle device is installed in the vehicle and picks up an image. FIG. 3 is a diagram illustrating the installation positions of the first stereo camera and the second stereo camera. FIG. 4 is a diagram schematically illustrating a luminance image and a parallax image acquired by the acquisition unit. FIG. 5 is a diagram schematically illustrating connection processing performed by the image processing unit. FIG. 6 is a diagram schematically illustrating a first example of additional processing performed by the image processing unit in connection processing. FIG. 7 is a diagram schematically illustrating a second example of the addition process performed by the image processing unit in the connection process. FIG. 8 is a diagram illustrating a plurality of image frames that are continuously acquired over time. FIG. 9 is a diagram illustrating parameters used for the linkage process. FIG. 10 is a diagram illustrating a first example of processing performed by the information processing apparatus. FIG. 11 is a diagram illustrating a second example of processing performed by the information processing apparatus. FIG. 12 is a diagram illustrating a parameter adjustment method.

  Hereinafter, an information processing system according to an embodiment will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram illustrating a configuration example of an information processing system 10 according to the embodiment. As illustrated in FIG. 1, the information processing system 10 includes, for example, an in-vehicle device 20 and an information processing device 30.

  The in-vehicle device 20 includes, for example, a first stereo camera 200, a second stereo camera 202, an inclination sensor (attitude sensor) 204, a GPS (position sensor) 206, an acceleration sensor (vehicle speed sensor) 208, a parallax image generation unit 210, and a first communication. It has a section 212 and operates by being loaded (installed) on a vehicle 100 or the like to be described later.

  The first stereo camera 200 includes, for example, two lenses and two image sensors, and simultaneously captures two luminance images (captured images corresponding to the right eye and the left eye). Similar to the first stereo camera 200, the second stereo camera 202 includes, for example, two lenses and two image sensors, and simultaneously captures two luminance images (captured images corresponding to the right eye and the left eye). That is, the in-vehicle device 20 captures four luminance images at the same time using the first stereo camera 200 and the second stereo camera 202. The captured images captured by the first stereo camera 200 and the second stereo camera 202 are not limited to luminance images, and may be captured images such as a polarized image and a spectral image in which other pixel values are arranged.

  For example, as illustrated in FIG. 2, the in-vehicle device 20 captures an area continuing in the traveling direction of the vehicle 100 as different frames (captured images) at different times so as to include overlapping imaging areas (image overlapping portions) having a predetermined width. . The image overlapping portion is, for example, 10 to 30% of the width of the captured image. In FIG. 2, frames that are imaged at different times by any one of a total of four imaging elements included in the first stereo camera 200 and the second stereo camera 202 are schematically illustrated.

  More specifically, the first stereo camera 200 and the second stereo camera 202 are arranged side by side behind the vehicle 100 as shown in FIG. 3, for example, and simultaneously image all regions in the width direction of the road from above. The frame rate of the first stereo camera 200 and the second stereo camera 202 is set to 30 fps, for example. In this case, the moving distance of the vehicle 100 in the traveling direction at 1/30 s is 0.37 m @ 40 km / h, 0.46 m @ 50 km / h, or the like. In this case, if the width of the captured image in the traveling direction is 0.6 m, the image overlap portion is 0.23 m or 0.14 m. Further, the frame rate of the first stereo camera 200 and the second stereo camera 202 may be set according to the moving speed of the vehicle 100.

  The in-vehicle device 20 is not limited to having two stereo cameras, and three or more stereo cameras may be provided in order to efficiently capture an area in the width direction of the road. Also, any of the stereo cameras may be set so that the height from the road is different from other stereo cameras. The in-vehicle device 20 may be provided with a projection device that projects a texture at the timing when the first stereo camera 200 and the second stereo camera 202 capture images.

  The inclination sensor 204 (FIG. 1) detects the acceleration in the direction of gravity and the like, and detects the attitude (inclination) of the in-vehicle device 20 that changes according to the inclination of the road. The GPS 206 functions as a position sensor that detects the position of the in-vehicle device 20. The acceleration sensor 208 detects acceleration in movement of the in-vehicle device 20 (vehicle 100). The acceleration sensor 208 also has a function as a position sensor that detects the position of the in-vehicle device 20 from the acceleration.

  Further, the frame rate captured by the first stereo camera 200 and the second stereo camera 202 may be set according to the acceleration (or moving speed) detected by the acceleration sensor 208. For example, when the speed of the vehicle 100 is 1 m / s and the imaging range of the first stereo camera 200 is 50 cm in the traveling direction, the first stereo camera 200 captures images three times or more per second. Note that when the moving speed of the vehicle 100 is 0, the first stereo camera 200 and the second stereo camera 202 do not perform the imaging process and reduce the total data amount.

  The parallax image generation unit 210 generates parallax images using luminance images captured by the first stereo camera 200 and the second stereo camera 202, respectively. The parallax image generation unit 210 may be configured by hardware, or a part or all of the parallax image generation unit 210 may be configured by software. The first communication unit 212 is, for example, a wireless communication device that transmits information output from each unit configuring the in-vehicle device 20 to the information processing device 30.

  The information processing apparatus 30 is a server including, for example, a CPU and a storage device, and includes a second communication unit 300, an acquisition unit 302, a feature point extraction unit 304, a weight determination unit 306, an image processing unit 308, and a processing control unit 310. . However, the information processing device 30 may be configured integrally with the in-vehicle device 20.

  The second communication unit 300 is, for example, a wireless communication device that receives information transmitted by the first communication unit 212. The acquisition unit 302 includes a luminance image acquisition unit 312, a parallax image acquisition unit 314, and a three-dimensional information conversion unit 316, and acquires information transmitted by the first communication unit 212 via the second communication unit 300.

  More specifically, the luminance image acquisition unit 312 acquires each frame of luminance images captured by the first stereo camera 200 and the second stereo camera 202. The parallax image acquisition unit 314 acquires the parallax image generated by the parallax image generation unit 210. The three-dimensional information conversion unit 316 indicates the state of the road (such as unevenness) using the parallax image acquired by the parallax image acquisition unit 314 using information detected by at least one of the inclination sensor 204, the GPS 206, and the acceleration sensor 208. Convert to 3D data. This three-dimensional data is sometimes referred to as point group data because points (pixels) having XYZ coordinates form a group. The point cloud data is, for example, three-dimensional information data, and is an image (third order) on the XY plane with different colors or the like according to values in the Z direction (distance from the stereo camera or road surface height information). Correspond to the original information image). That is, the three-dimensional information conversion unit 316 generates a three-dimensional information image indicating the three-dimensional information on a plane. In addition, it can be said that a parallax image is also an image (three-dimensional information image) containing three-dimensional information.

  The feature point extraction unit 304 uses, for example, a Harris operator from the luminance image (captured image) captured by the first stereo camera 200 and the second stereo camera 202 and the 3D information image generated by the conversion by the 3D information conversion unit 316. Feature points are extracted by

  The weighting determination unit 306 determines the weighting for the three-dimensional information image and the captured image based on the number of feature points that are equal to or larger than a predetermined threshold extracted from the three-dimensional information image and the captured image by the feature point extraction unit 304.

  The image processing unit 308 includes an interpolation processing unit 318, and is based on the two-frame three-dimensional information image (or parallax image) acquired by the acquisition unit 302 and the two-frame captured image, and the weighting determined by the weighting determination unit 306. Then, a linkage process is performed in which the imaging regions that overlap at least two frames of at least one of the three-dimensional information image (or parallax image) and the captured image are overlapped. Since the interpolation processing unit 318 does not always have the pixels in the image overlapping portion of the two frames captured at the same location, the interpolation processing can be performed to identify pixel overlap in units of subpixels. To do. In addition, the image processing unit 308 may be configured to perform the joining process performed on one of the two frames of the captured image or the three-dimensional information image also on the other two frames.

  In addition, the image processing unit 308 can continuously connect the images corresponding to the entire area captured by the first stereo camera 200 and the second stereo camera 202 by performing the connection process continuously for each frame. . Further, the image processing unit 308 stops the linkage process when the feature points of the images at three frames or more, that is, at the times T1, T2, and T3 match. Further, when it is determined from the detection result of the acceleration sensor 208 that the vehicle 100 has stopped (definite acceleration) and accelerated again from the detection result of the acceleration sensor 208, the image processing unit 308 is configured not to perform the linkage process. Further, the image processing unit 308 may be configured to change the processing frequency of the connection processing according to the moving speed of the in-vehicle device 20.

  The processing control unit 310 controls processing performed by each unit configuring the information processing apparatus 30. The acquisition unit 302, the feature point extraction unit 304, the weight determination unit 306, the image processing unit 308, and the processing control unit 310 may be configured by hardware, or part or all may be configured by software. .

  Next, the operation of the information processing system 10 will be described with reference to the drawings. FIG. 4 is a diagram schematically illustrating a luminance image and a parallax image acquired by the acquisition unit 302. First, the first stereo camera 200 captures the luminance image A1 and the luminance image B1 at time T1. The parallax image generation unit 210 generates the parallax image 1 from the luminance image A1 and the luminance image B1. Furthermore, the first stereo camera 200 captures the luminance image A2 and the luminance image B2 at time T2. The parallax image generation unit 210 generates the parallax image 2 from the luminance image A2 and the luminance image B2.

  The luminance image captured at time T1 and the luminance image captured at time T2 include an image overlap portion in which the same region (road) is captured. The luminance image acquisition unit 312 acquires the parallax image 1 and the parallax image 2. In addition, the parallax image acquisition unit 314 acquires, for example, a region of the luminance image A1 where the photographing range overlaps with the luminance image B1 as the overlapping portion luminance image 1. In addition, the parallax image acquisition unit 314 acquires, for example, a region of the luminance image A2 where the photographing range overlaps with the luminance image B2 as the overlapping portion luminance image 2.

  FIG. 5 is a diagram schematically illustrating a connection process performed by the image processing unit 308. The image processing unit 308 performs processing for connecting the overlapping portion luminance image 1 and the overlapping portion luminance image 2 and processing for connecting the parallax image 1 and the parallax image 2. It should be noted that the luminance image A1 and the luminance image A2 do not necessarily have pixels (pixels) showing the same location (area). Therefore, the image processing unit 308 specifies the overlap in units of subpixels by the interpolation processing unit 318 performing interpolation processing or the like.

  FIG. 6 is a diagram schematically illustrating a first example of addition processing performed by the image processing unit 308 in connection processing. FIG. 6 shows a case where the vehicle 100 traveling on a road having almost no parallax (unevenness) moves laterally shifted between frames. In the luminance image A1 and the luminance image A2, the white line on the road has high contrast and feature points are likely to appear. On the other hand, the white line of the road has only a slight step in shape, and it is difficult to obtain feature points due to parallax.

  In the luminance image, since the luminance of the white line is high, it is easy to accurately connect the luminance image A1 and the luminance image A2 with the feature points. Therefore, when the luminance image has more feature points equal to or greater than the predetermined threshold than the parallax image as illustrated in FIG. 6, the weighting determination unit 306 determines to weight the luminance image information. The image processing unit 308 performs a joining process based on the weight determined by the weight determining unit 306. In other words, the image processing unit 308 performs processing for connecting the corresponding parallax images using the method of connecting the luminance images. For example, if the image processing unit 308 can identify the region where the lower 5 pixels of the parallax image 1 and the upper 5 pixels of the parallax image 2 overlap from the luminance image, the image processing unit 308 connects the parallax image 1 and the parallax image 2 under the conditions.

  FIG. 7 is a diagram schematically illustrating a second example of the addition process performed by the image processing unit 308 in the connection process. FIG. 7 shows a case where the vehicle 100 traveling on a road having almost no parallax (unevenness) has a rotational motion between frames. When the vehicle 100 rotates (bends), the image processing unit 308 checks the coincidence of pixels while rotating the parallax image 2 with respect to the parallax image 1, and the degree of coincidence is the highest at which angle of rotation. It is determined whether it is high, and the parallax image 1 and the parallax image 2 are connected.

  In this case, if the parallax is small, it is difficult to find a matching area in the parallax image. Therefore, the weighting determination unit 306 determines to weight the information on the luminance image A1 and the luminance image A2 having many feature points that are equal to or greater than a predetermined threshold. The image processing unit 308 performs a joining process based on the weight determined by the weight determining unit 306. In other words, the image processing unit 308 performs processing for connecting the corresponding parallax images using the method of connecting the luminance images.

  Further, the image processing unit 308 may perform object recognition by machine learning or the like regarding the rotation angle, and detect only the angle from the change in the angle. The addition processing shown in FIG. 7 is effective when a linear object such as a white line is included.

  On the other hand, features such as roadsides are difficult to appear in the luminance image. Therefore, the weight determination unit 306 determines to give weight to the parallax image when performing processing for connecting the luminance images in which the frames are taken.

  As described above, the information processing apparatus 30 performs the linkage process using the luminance image and the parallax image acquired from the in-vehicle apparatus 20. Then, as illustrated in FIG. 8, the information processing device 30 continuously acquires a plurality of image frames from the in-vehicle device 20 as time elapses, and connects the luminance image and the point cloud data (three-dimensional information image). Process.

  Here, the image processing unit 308 includes information (X, Y, θz: rotation angle around the Z axis) obtained from the luminance image shown in FIG. 9, an inclination sensor (attitude sensor) 204, and a GPS (position sensor) 206. And the connection processing is performed using each information obtained from the acceleration sensor (vehicle speed sensor) 208.

  FIG. 10 is a diagram illustrating a first example of processing performed by the information processing apparatus 30. As shown in FIG. 10, the feature point extraction unit 304 performs feature point extraction processing using point group data (Cn, Cn + 1: see FIG. 8) for two adjacent frames (S100), and luminance for two adjacent frames. Feature point extraction processing is performed using images (Dn, Dn + 1: see FIG. 8) (S102). The feature point extraction unit 304 outputs, for example, the feature points as position and intensity information on the image plane.

  The weight determination unit 306 determines a weight W for each feature point output from the feature point extraction unit 304 by invalidating a feature point having an intensity equal to or less than a predetermined threshold (weight output process: S104). The threshold value for luminance is Th_b, and the threshold value for point cloud data is Th_s. Here, each threshold value is set so that three or more feature points are secured.

  A feature point having a luminance equal to or higher than a predetermined threshold is Cb (n_Cb) (Px, Py, I), and a feature point of the point cloud data is Cs (n_Cs) (Px, Py, I). Px and Py indicate positions in the image plane, and I indicates intensity. (N_Cb) and (n_Cs) indicate the number of feature points having an intensity equal to or greater than a predetermined threshold.

  There are the following methods for calculating the weight W.

  When the weight W is determined according to the ratio of the number of feature points equal to or greater than a predetermined threshold, the weight W is calculated by the following equation 1.

  W = n_Cb / (n_Cb + n_Cs) (1)

  Further, when the weight W is determined according to the ratio of the sum of the intensity of the feature points, the weight W is calculated by the following equation 2.

W = Σ (I_Cb) / {Σ (I_Cb) + Σ (I_Cs)} (2)
Here, I_Cb and I_Cs indicate the strength of each feature point.

  Further, when the number of feature points is limited to, for example, 3 and the weight W is determined according to the ratio of the top three intensities, the weight W is calculated by the following equation 3.

  W = Σ (I_Cb) / Σ (I_Cb + I_Cs) (3)

  Further, when the weight W is determined according to the ratio of the product of “total distance between feature points” and “intensity” obtained from the “position” of each feature point, the weight W is calculated by the following equation 4.

W = A_b / (A_b + A_s) (4)
However, the sum of distances between feature points is D_b and D_s, and the intensity of each feature point is I_Cb (n) and I_Cs (n). n)). Here, the greater the distance between feature points, the greater the accuracy in calculating θx, θy, and θz.

  Then, the weight determination unit 306 performs a comparison process using the determined weight W, provides W> Th_W (weight threshold), and determines which of the luminance image and the point cloud data is to be weighted (S106). Since the weight W can take a value within the range of 0 <W <1, Th_W is set to 0.5, for example.

  Next, the image processing unit 308 performs matching processing of feature points of adjacent frames (S108), and performs relative parameter calculation processing (joining processing) (S110). Here, the image processing unit 308 performs matching processing using, for example, SIFT feature values.

  Then, the image processing unit 308 performs parameter adjustment using the weight W in the connection process. For example, the image processing unit 308 generates an intermediate numerical value between the luminance image and the point cloud data shown in the following expression 5 and uses it for the connection process.

  p_s ′ = (W · p_s + (1−W) · p_b) / 2 (5)

  Note that p_s is a parameter of any of the point group data shown in FIG. 9, and p_b is a parameter of the luminance image.

  FIG. 11 is a diagram illustrating a second example of processing performed by the information processing apparatus 30. As shown in FIG. 11, the feature point extraction unit 304 performs feature point extraction processing using point group data (Cn, Cn + 1: see FIG. 8) for two adjacent frames (S100), and luminance for two adjacent frames. Feature point extraction processing is performed using images (Dn, Dn + 1: see FIG. 8) (S102). The feature point extraction unit 304 outputs, for example, the feature points to the image processing unit 308 and the weight determination unit 306 as position and intensity information on the image plane.

  The image processing unit 308 performs a feature point matching process (S200) using only point cloud data and a feature point matching process (S202) using only a luminance image, and a point cloud using only point cloud data. Data connection processing (S204) and luminance image connection processing using only luminance images (S206) are performed.

  The weight determination unit 306 determines the weight W (weight output processing: S208). Then, the image processing unit 308 uses the weight W determined by the weight determination unit 306 to adjust the result of the linkage processing executed in the processing of S204 and S206 as illustrated in FIG. 12 (relative parameter adjustment processing: S210). ).

  As described above, the information processing system 10 accurately generates and outputs continuous three-dimensional shape data (three-dimensional information image) and luminance images of a road unit distance (eg, several hundred meters). The three-dimensional shape data can indicate the state of road flatness (unevenness in the traveling direction), rutting (unevenness in the width direction of the road), cracks, and white lines. The luminance image can show road cracks and the like.

DESCRIPTION OF SYMBOLS 10 Information processing system 20 In-vehicle apparatus 30 Information processing apparatus 100 Vehicle 200 First stereo camera 202 Second stereo camera 204 Inclination sensor 206 GPS
208 acceleration sensor 210 parallax image generation unit 212 first communication unit 300 second communication unit 302 acquisition unit 304 feature point extraction unit 306 weight determination unit 308 image processing unit 310 processing control unit 312 luminance image acquisition unit 314 parallax image acquisition unit 316 tertiary Original information conversion unit 318 Interpolation processing unit

JP 2001-043495 A

Claims (10)

Acquire at least two frames or more of stereo imaged images having a predetermined width overlapping imaging area imaged by a stereo camera, and two or more frames of 3D information data having 3D information calculated from each of the imaged images. An acquisition unit to
A feature point extraction unit that extracts feature points from the three-dimensional information data and the captured image acquired by the acquisition unit;
A weight determining unit for determining weights for the three-dimensional information data and the captured image based on the number of feature points equal to or greater than a predetermined threshold extracted from the three-dimensional information data and the captured image by the feature point extracting unit; ,
Two of at least one of the three-dimensional information data and the captured image based on the two frames of the three-dimensional information data acquired by the acquisition unit, the two frames of the captured image, and the weighting determined by the weight determination unit. An image processing unit for performing a linking process for connecting frames so that the overlapping imaging areas overlap; and
An information processing apparatus comprising: The image processing unit
The information processing apparatus according to claim 1, wherein the joining process performed on one of the two frames of the captured image or the three-dimensional information data is performed on the other two frames. The image processing unit
The information processing apparatus according to claim 1, wherein the joining process is performed based on a change in an angle indicating a moving speed and a posture of the stereo camera. The image processing unit
The information processing apparatus according to any one of claims 1 to 3, wherein a processing frequency of the connection process is changed according to a moving speed of the stereo camera. The stereo camera made movable at a predetermined speed;
An information processing apparatus according to any one of claims 1 to 4,
An information processing system comprising: The stereo camera
The information processing system according to claim 5, wherein a frame rate is set according to the moving speed. The image processing unit
The information processing system according to claim 5 or 6, wherein when the feature points extracted by the feature point extraction unit match in a predetermined number of frames, the joining process is stopped. The image processing unit
8. The connection process according to claim 5, wherein when the movement of the stereo camera is stopped after being decelerated and then accelerated again, the joining process is not performed for a period during which the stereo camera was stopped. Information processing system described in 1. Acquire at least two frames or more of stereo imaged images having a predetermined width overlapping imaging area imaged by a stereo camera, and two or more frames of 3D information data having 3D information calculated from each of the imaged images. And a process of
Extracting a feature point from the acquired three-dimensional information data and the captured image;
Determining weights for the three-dimensional information data and the captured image based on the number of feature points equal to or greater than a predetermined threshold extracted from the three-dimensional information data and the captured image;
Based on the acquired two frames of the three-dimensional information data, two frames of the captured image, and the determined weighting, the overlapping imaging region overlaps at least two frames of the three-dimensional information data and the captured image. A step of performing a bridging process to connect,
An information processing method including: Acquire at least two frames or more of stereo imaged images having a predetermined width overlapping imaging area imaged by a stereo camera, and two or more frames of 3D information data having 3D information calculated from each of the imaged images. And steps to
Extracting feature points from the acquired three-dimensional information data and the captured image;
Determining weights for the three-dimensional information data and the captured image based on the number of feature points equal to or greater than a predetermined threshold extracted from the three-dimensional information data and the captured image;
Based on the acquired two frames of the three-dimensional information data, two frames of the captured image, and the determined weighting, the overlapping imaging region overlaps at least two frames of the three-dimensional information data and the captured image. The step of performing the bridging process to connect,
Processing program for causing a computer to execute.