JP2003333590A - System for generating image at site - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system for generating images at a site for grasping a three-dimensional shape of a construction site and properly informing the operator of a construction machine about a relation between a planned frame and a present completed amount, a traveling available range, and a dangerous range or the like. <P>SOLUTION: A plurality of camera poles 1 each provided with a stereo camera 3 comprising a plurality of cameras 2 are placed respectively at different places, and each camera 2 photographs a construction work site 5. Each stereo camera 3 applies stereo processing to a photographed image to obtain a distance image. A host device 4 acquiring the distance image from each stereo camera 3 integrates the distance images to produce three-dimensional model data of the construction work site 5. The operator of the construction machine 6 designates an optional virtual view point 9-1, 9-2, 9-3, or 9-4 to the host device 4 from a driver's seat. The host device 4 draws an image viewing the construction work site 5 from the designated virtual view point on the basis of the three-dimensional model data and transmits the image to the construction machine 6, which displays the image in front of the driver's seat. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to image processing for providing an operator, such as an occupant of a work vehicle such as a construction machine, a site worker, or a supervisor with an image of a site observed from an arbitrary virtual viewpoint. Regarding technology.

[0002]

2. Description of the Related Art For example, at a construction site of civil engineering,
Currently, operators of construction machines (hereinafter abbreviated as construction machines) use tautons (such as cruciforms and other prescribed shapes that stand on the ground, are made of wooden sticks, etc., to indicate the finished shape of the land, etc. However, it is not possible to accurately grasp from the operator's seat of the construction machine what the actual work is like for the siding. Therefore, work efficiency is poor because a worker who guides the construction machine operator is introduced or the construction machine operator descends from the seat and checks the relationship between the present condition and the tension.

In an automobile, for example, as disclosed in Japanese Patent Laid-Open No. 2001-339716, an image taken by a camera mounted on the vehicle is used to dynamically change the viewpoint position virtually, and a display device in the vehicle is used to change the viewpoint according to the driving situation. There is disclosed a vehicle periphery monitoring device and method capable of displaying an optimal composite image of the vehicle periphery. It is expected that the efficiency of construction work will be improved by applying this virtual viewpoint technology to construction machinery.

[0004]

[Problems to be Solved by the Invention]
Even if the one described in No. 01-339716 is applied to the construction machine as it is, the image from the camera mounted on the construction machine is used, so it is only possible to understand how the site area around the construction machine is viewed from the viewpoint centering on the construction machine. You can only do it. Moreover, only the situation of the site within the range where the construction machine moves can be grasped, and the situation of the whole site cannot be grasped accurately. Therefore, it is necessary to accurately inform the relevant parties of the current situation of the entire site that the construction machine operator, site workers, supervisors, etc. want to know, such as the relationship between the tension and the current work model, the movable range and the dangerous range. Is difficult.

Therefore, an object of the present invention is to provide a system for grasping the current situation of the entire site and accurately informing the persons concerned.

[0006]

An on-site image generation system according to an embodiment of the present invention includes an image input unit for inputting a plurality of images of a site viewed from different predetermined places on the site, and a desired virtual image. A viewpoint specifying means for specifying a viewpoint,
An image generating unit that generates an image of the site viewed from the designated virtual viewpoint using a plurality of images of the site.

In a preferred embodiment, the image generating means further comprises a three-dimensional modeling means for generating the three-dimensional model data of the site.

[0008] In a preferred embodiment, a plurality of stereo cameras arranged at different places where the scene can be observed are provided,
The image generation means generates an image of the site viewed from the designated virtual viewpoint, using the images captured by the plurality of stereo cameras.

In a preferred embodiment, each of the stereo cameras is a pole type having a plurality of cameras arranged in a line in the vertical direction.

In a preferred embodiment, the image input means inputs a plurality of moving images of the scene, and the image generating means uses the plurality of input moving images to perform the designation. A moving image in which the scene is viewed from a virtual viewpoint is generated.

In a preferred embodiment, further, a specific object detecting means for detecting a three-dimensional position of a specific object such as a marking on the site is further provided, and the image generating means generates an image looking at the site. At this time, based on the detected three-dimensional position of the specific object, a display for showing the specific object is included in the generated image.

In a preferred embodiment, the viewpoint designating means and the display means are mounted on a work vehicle working on the site.

In a preferred embodiment, the viewpoint designating means registers a viewpoint registration means for registering one or more designated virtual viewpoints, and a desired virtual viewpoint from the registered one or more virtual viewpoints. The image generating means has a viewpoint selecting means for selecting, and when the virtual viewpoint is switched from a certain value to another value by the viewpoint selecting means, the image generating means responds to this immediately when generating an image. Switch the virtual viewpoint.

[0014]

BEST MODE FOR CARRYING OUT THE INVENTION A site image generation system for a work vehicle according to an embodiment to which the present invention is applied will be described below with reference to the drawings.

FIG. 1 is a diagram showing the arrangement of cameras installed at a construction work site in the on-site image generation system according to the first embodiment of the present invention.

The present system has a plurality (here, three) of camera poles 1, and each camera pole 1 is arranged at a different position in order to photograph the construction work site 5. Each camera pole 1 can be fixed to the ground, or can be mounted on a vehicle or the like to be movable. The position at which each camera pole 1 is arranged is arranged at a position where a blind spot cannot be made in order to photograph the entire construction work site 5, and its three-dimensional coordinates are measured in advance.
The number of camera poles 1 does not necessarily have to be three.

Each camera pole 1 includes a plurality of (five in this case) moving image cameras 2 arranged at different predetermined relative positions.
It has a stereo camera 3 which is a combination of (for example, a video camera). 5 included in one stereo camera 3
The cameras 2 in a row are arranged in a line in the height direction, but this does not necessarily have to be the case. For example, the cameras 2 may be arranged in a row of 5's on the dice or other arrangements. Well, the number of cameras 2 does not have to be 5, and may be more or less than 5 as long as it is 2 or more. Then, all the cameras 2 of each camera pole 1 simultaneously photograph the construction work site 5 of the construction work site. As will be described later, distance images are independently generated from the five moving images from each camera pole 1, and a dynamic three-dimensional model of the construction work site 5 is synthesized from these distance images. Become.

The construction work site 5 is provided with stakes 7 at a plurality of points, and a construction machine (hereinafter referred to as a construction machine) 6 performs a predetermined work according to the stakes 7. Here, in the figure, the dotted line 8 connecting the stakes 7 is a virtual design line that represents the finished shape of the land indicated by the stakes 7. Since the operator of the construction machine 6 will perform work so that the land shape matches the design line 8 indicated by the stake 7, it is necessary to check the relation between the stake 7 and the actual topography at any time during the work. . Further, in order to proceed with the work safely and efficiently, the operator needs to observe the topography of the construction work site 5 at any time to grasp the dangerous area and the travelable area.

In order to make that possible, as will be described in detail later, the operator's cab of the construction machine 6 is equipped with an information terminal having an operating device such as a joystick and a display device. Then, the operator can specify an arbitrary virtual viewpoint (including the position of the viewpoint and the direction of the line of sight, which will be collectively referred to as a viewpoint parameter) by operating the operating device on the driver's seat. It is also possible to register a plurality of viewpoint parameters and select a desired viewpoint parameter from among them. A host computer installed (or mounted on the construction machine 6) in a place different from the construction machine 6 generates dynamic three-dimensional model data of the construction work site 5 using the moving image from the stereo camera 3. Then, using the dynamic three-dimensional model data, a moving image when the construction work site 5 is viewed with the viewpoint parameter desired by the operator is created, and the moving image is sent to the information terminal in the cab of the construction machine 6. It is sent and displayed on the display device. The operator confirms the construction state by looking at the operation image.

Each of the construction machines 6 has a specific shape (here, a spherical shape) markers 61, 62 at a predetermined location (here, a predetermined position on the roof of the driver's seat) visible from the stereo camera 3.
A plurality (here, two) are provided. Markers 61 and 62
Are colored in different colors, for example, and can be identified by image processing from an image captured by the camera 2. This marker pair is used when detecting the position and direction of the construction machine 6, as described later.

Next, the functional configuration of this system will be described with reference to FIGS.

FIG. 2 shows a schematic operation flow of this system.

As shown in FIG. 2, in step 100, topographical information (distance image) of the construction work site 5 is collected by the plurality of stereo cameras 3. In step 110, topographical information from the plurality of stereo cameras 3 is combined to generate dynamic three-dimensional model data of the construction work site 5. At step 120, a viewpoint parameter desired by the operator is designated from among a plurality of registered viewpoint parameters. In step 130, a moving image of the construction work site 5 viewed from the specified viewpoint parameter is generated from the dynamic three-dimensional model data. In step 140, the moving image is displayed on the display device in the construction machine driver's cab.

The designation of the viewpoint parameter is performed by operating an operating device such as a joystick in the operator's cab of the construction machine. As described above, the viewpoint parameter is not only selected from the plurality of registered viewpoint parameters. You can also move freely to specify the viewpoint parameters you like. Usually, at the start of the work, in step 150, the operator appropriately changes the viewpoint parameter and observes the image with the viewpoint parameter, so that it is possible to conveniently grasp the relation between the tent and the land or the state of the terrain. Search for different viewpoint parameters (viewpoint position and line-of-sight direction). Then, in step 160, each of the appropriate viewpoint parameters found is registered in the in-vehicle information terminal or the host computer. Multiple viewpoint parameters can be registered by this method. For example, as illustrated in FIG. 1, a viewpoint parameter 9 when viewed horizontally from the direction orthogonal to the height of the virtual design line 8 indicated by the tension 7 (that is, a sectional view of the site 5 is displayed) 9. It is possible to find and register -1, 9-2, 9-3, the viewpoint parameter 9-4 vertically viewed from above (that is, a plan view of the site 5 is displayed), and the like. Once one or more viewpoint parameters are registered, the above steps 120 to 140 will be performed thereafter.
Thus, the operator simply selects the registered viewpoint parameter, and the system immediately generates an image of the construction work site 5 with the selected viewpoint parameter without manually changing and adjusting the viewpoint parameter. And display it on the display device. You can freely change, delete, or add the registered viewpoint parameters.

FIG. 3 is a diagram showing the overall functional configuration of this system. This system includes a stereo camera 3 installed on each camera pole 1, a host device 4 that obtains a three-dimensional model of a construction work site 5 by integrating distance images synthesized by the stereo cameras 3, and a construction machine 6. The information terminal 200 mounted on the driver's seat is provided.

Each stereo camera 3 includes five cameras 2 and an image processing unit 31 for generating a distance moving image by a stereo image processing method using the luminance moving images captured by the five cameras 2. The wireless transmission / reception unit 32 wirelessly transmits the processing result of the image processing unit 31 to the host device 4. (In this embodiment, each stereo camera 3 generates a distance moving image and transmits the distance moving image to the host device 4, and the host device 4 generates a three-dimensional model of the construction site 5 from a plurality of collected distance moving images. However, it does not have to be the case that another method, for example, each stereo camera 3 does not perform complicated image processing, and the luminance moving image of the five cameras 2 is used as a host image. The data is transmitted to the device 4, and the host device 4 generates a range moving image for each stereo camera 3 by the stereo image processing method.
A three-dimensional model may be generated from those distance moving images. Since various variations can be adopted for the sharing of processing between the stereo camera 3 and the host device 4 as described above, the following description is merely one example. )

FIG. 4 shows a detailed functional configuration of the image processing section 31 for each stereo camera 3.

The image processing section 31 includes a construction machine position recognition section 33.
A stereo processing unit 34 and a coordinate conversion processing unit 35.

Here, in order to perform a stereo image process to obtain a range image, 5 included in one stereo camera 3
One of the two cameras 2 is the reference camera 2a, and the remaining four are the comparison cameras 2b. And the reference camera 2a
And each of the comparison cameras 2b form one camera pair. That is, four camera pairs are configured in this embodiment.

The stereo processing unit 34 includes a LoG (Laplacian of Gaussian) processing unit 341 for removing noise and enhancing edges of images captured by the reference camera 2a and the comparison camera 2b, and each camera pair. For each, pattern matching is performed between the image from the reference camera 2a (hereinafter referred to as the reference image) and the image from each comparison camera 2b (hereinafter referred to as the comparison image) to obtain the SAD (Sum of Absolute Distance). An SAD processing unit 342 for calculating, an SSAD (Sum of SAD) processing unit 343 for totaling SADs of four camera pairs, and a distance image combining unit 344 for combining distance images are provided.

When each LoG processing unit 341 performs LoG processing on the image from each camera 2, the difference in image quality between different cameras can be reduced. As a result, it is possible to make the pattern matching process with high accuracy between successive images of different cameras.

Each of the SAD processing section 342 and the SSAD processing section 343 performs processing in the following procedure, for example,
A so-called corresponding point search is performed.

(1) Select one pixel as a target of distance measurement from the reference image. A certain distance is assumed as the distance to the object in the construction work site corresponding to this target pixel. Then, the position of the pixel (hereinafter referred to as “corresponding candidate pixel”) in each comparison image corresponding to the coordinate point in the construction work site space separated by the assumed distance is obtained.

(2) A matching window (pixel block) (hereinafter referred to as "reference window") having a size of, for example, 3 pixels × 3 pixels centered on the position of the target pixel is set in the reference image. Within each comparison image, a matching window (hereinafter referred to as “comparison window”) of the same size centered on the position of the corresponding candidate pixel is set.

(3) For each camera pair, the pixel value of each pixel in the reference window on the standard image and the pixel value of each corresponding pixel in the comparison window on the comparative image are compared, and “Dissimilarity” is calculated. As the “degree of difference”, the absolute value of the difference between pixel values, the square of the difference between pixel values, or the like is used.

(4) The "difference" is obtained for all the pixels in the reference window, and the "difference" is summed up for all the pixels in the reference window (hereinafter referred to as window addition) to obtain "difference obtained by window addition". Degree ”for each camera pair. This “window added difference” is “SA
D ".

The SSAD processing unit 343 acquires the “SAD” of each camera pair from each SAD processing unit 342, and sums the “SAD” of the four camera pairs that have been collected to obtain the “final sum of dissimilarity”. Is asked. This "final total of dissimilarity" is "SSAD".

Each SAD processing unit 342 further assumes a plurality of distances as the distance to the object in the construction work site corresponding to the target pixel, and each of these distances is " “SAD” is obtained, and in response to this, the SSAD processing unit 343 performs the above-described processing to obtain “SSAD” for each distance. Then, the SSAD processing unit 343 estimates one assumed distance at which “SSAD” is the minimum as the distance to the object corresponding to the target pixel, and outputs the target pixel and the estimated distance to the distance image synthesis unit 344.

Here, the minimum "SSAD" means that the image pattern in the reference window of the standard image is
This means that the comparison image is most similar to the image pattern in the comparison window.

Then, all pixels in the reference image are used as target pixels to perform the above-described processing to estimate the distance, and the distance image synthesizing unit 344 accumulates this to obtain a distance image corresponding to the reference image. To be

In the present embodiment, in order to grasp the movement of the construction machine 6 in real time, each camera 2 is basically a moving image camera (for example, a video camera), and each output from the moving image camera 2. Corresponding point search is performed for each frame image at the time point. As a result, the image output from the stereo processing unit 34 is also a moving image in which the frame image at each time point is a distance image.

By performing the above corresponding point search for each stereo camera 3, a range image is obtained for each stereo camera 3, but the three-dimensional coordinate value of the target area 5 obtained from the range image of each stereo camera 3 ( That is, the x and y coordinate values of the pixel in the image and the distance value (z coordinate value) indicated by the pixel value)
Is expressed in a camera coordinate system based on the location and direction of the stereo camera 3. Therefore, the coordinate conversion processing unit 35 performs a process of converting the coordinate value of each camera coordinate system obtained from the distance image of each stereo camera 3 into a predetermined common reference coordinate system.

As a specific device for performing the coordinate conversion process, for example, the device disclosed in JP-A-2002-32744 can be applied.

The construction machine position recognition unit 33 includes a marker coordinate detection unit 331 that detects a marker attached to the construction machine in the image from the reference camera, and a marker distance determination unit that determines the distance from the camera pole 1 to the marker. 332.

The marker coordinate detector 331 receives a reference image from one of the five cameras 2, for example, the reference camera 2a. Since the markers 61 and 62 attached to the construction machine 6 are shown as circles in the image,
It is extracted by pattern recognition, and each marker 61, 6 is extracted.
2. Determine the x, y coordinate values on the two reference images.

The marker distance determining unit 332 corresponds to the x and y coordinates of the markers 61 and 62 detected by the marker coordinate detecting unit 331 from the distance image corresponding to the reference image combined by the distance image combining unit 344. The z coordinate value is obtained, and the distance from the camera pole 1 to the markers 61 and 62 is obtained.

The marker distance determination unit 332 notifies the coordinate conversion processing unit 35 of the coordinate values of the markers 61 and 62 on the reference image and the distances from the camera pole 1 to the markers 61 and 62. The coordinate conversion processing unit 35, similarly to the conversion of the distance image, three-dimensional coordinate values indicating the position of each marker (x and y coordinate values of pixels in the image and distance value to the marker (z coordinate value)).
Is converted into a predetermined common reference coordinate system.

The coordinate conversion processing unit 35 transfers the three-dimensional coordinate values of each marker and the distance image to the wireless transmission / reception unit 32, and the wireless transmission / reception unit 32 transmits them to the host device 4. (Note that as a modification, the coordinate conversion processing unit 35 is provided in the host device 4,
You may make it transmit the marker coordinate value and distance image of the camera coordinate system before coordinate conversion from each stereo camera 3 to the host device 4. )

Next, returning to FIG. 3, the information terminal 200 of the host device 4 and the construction machine 6 will be described.

The host device 4 is composed of, for example, a general-purpose computer system, and individual components or functions in the host device 4 described below are realized by executing a computer program, for example.

The host device 4 has, as its internal functions, a radio transmitter / receiver 41 for receiving a radio signal from the stereo camera 3 and an individual construction machine based on the three-dimensional coordinate values of the markers from the plurality of stereo cameras 3. Of a construction machine, which detects the position and direction of the construction machine, a terrain generation section 43 which statistically processes distance images from a plurality of stereo cameras 3 to generate three-dimensional model data of the terrain of a construction work site, GIS data including position and direction and 3D model data of topography
And a GIS data storage unit 44 for storing as (Geographic Information System) data. Furthermore,
The host device 4 uses the terrain 3 in the GIS data storage unit 44.
An operator-desired one is used by using the tension detection unit 45 that detects a tension portion from the three-dimensional model data, the data indicating the position and direction of the construction machine in the GIS data storage unit 44, and the three-dimensional model data of the terrain. The rendering unit 46, which creates an image of the construction work site 5 with the viewpoint parameter, wirelessly communicates with the information terminal 200 of the construction machine, receives information such as designation of the viewpoint parameter from the information terminal 200, and the rendering unit 46. The wireless transmission / reception unit 47 that transmits the image from the information terminal 200 to the information terminal 200.

The construction machine information terminal 200 is also composed of, for example, a general-purpose computer system. The information terminal 200 includes a viewpoint operating unit 201 for the operator to change, register, and select viewpoint parameters, and a marking specification unit 202 for the operator to specify the portion to be marked from the displayed construction work site image. And a display device 203 for displaying a construction work site image from the host device 4, and wireless communication with the host device 4, and the viewpoint operating unit 201 provides information about viewpoint parameters and the marking designation unit 20.
The wireless transmission / reception unit 204 that transmits the information for specifying the stitching from the host device 4 to the host device 4 and receives the construction work site image from the host device 4.

FIG. 5 shows the detailed construction of the construction machine detection section 42 and the terrain generation section 43 of the host device 4.

As shown in FIG. 5, the construction machine detection unit 42 has three
A construction machine position determination unit 421 that identifies a pair of markers corresponding to each construction machine based on the positions of the markers estimated by the three stereo cameras 3 and determines the position and direction of the construction machine from the position of the marker pair. A construction machine recognition unit 422 that recognizes a three-dimensional model of each construction machine from the three-dimensional model of the terrain, and a construction machine pattern storage unit 423 that stores a characteristic pattern of the three-dimensional model of each construction machine.

The construction machine position determining unit 421 uses a predetermined statistical process (for example, an average of the coordinate values from the three stereo cameras 3) on the three-dimensional coordinate values of the markers estimated by the three stereo cameras 3, for example. The majority of the coordinate values from the three stereo cameras 3 is calculated, or the most reliable value among the coordinate values from the three stereo cameras 3 is selected. The specific position coordinate value. Then, by classifying the plurality of fixed-position markers into a pair of markers having a predetermined mutual positional relationship, the pair of markers 61 and 62 attached to each construction machine are detected, and The representative position of the pair of markers 61, 62 is set as the position of the construction machine, and the direction of the construction machine is determined based on the arrangement direction of the markers 61, 62 of the pair. To determine this position and direction,
Do this for all construction equipment (that is, markers for all pairs).

The construction machine recognizing unit 422 is used by the construction machine position determining unit 42.
Created by the distance image integration processing unit 431, which will be described later, using the position and direction of each construction machine from 1 and the three-dimensional characteristic patterns of construction machines of various models accumulated in the construction machine pattern storage unit 423. A 3D model portion corresponding to each construction machine is extracted from the 3D model data of the formed terrain. For example, from the three-dimensional model data of the terrain, a three-dimensional model portion of an area estimated to be an individual construction machine determined by the position and direction of each construction machine from the construction machine position determining unit 421 is extracted, and the extraction By comparing the 3D shape features of the 3D model part with the 3D feature patterns of the various types of construction machines stored in the construction machine pattern storage unit 423, the 3D model part of each construction machine can be obtained. More accurate indexing and identification of the model of the construction machine. The position and direction of each construction machine and the three-dimensional model obtained in this way are G elements as elements of GIS data.
It is stored in the IS data storage unit 44. (Note that the extraction processing of the three-dimensional model data of the construction machine by the construction machine recognition unit 422 may be omitted. In this case, the construction machine-related data included in the GIS data is the data from the construction machine position determination unit 421. Only the position and direction of the machine.)

The terrain generation unit 43 statistically processes the distance images from the three stereo cameras 3 and integrates the distance image integration processing unit 431, the processing result of the distance image integration processing unit 431, and the construction machine recognition unit 422 described above. And a terrain determination unit 431 that determines a three-dimensional model of the terrain at the construction work site based on the processing result of 1.

The distance image integration processing unit 431 performs a predetermined statistical process on the distance images from the three stereo cameras 3 (for example, majority decision of the coordinate values from the three stereo cameras 3, or three). The most reliable value among the coordinate values from the stereo camera 3 is selected, and the final three-dimensional coordinate value of the terrain (including construction machinery) (that is, three-dimensional model data of the terrain) is applied. ).

A more specific method of the above statistical processing is the method disclosed in Japanese Patent Laid-Open No. 2002-32744, that is, voxel data in three-dimensional space based on distance images from a plurality of stereo cameras. Can be applied to apply the method of integration.

The terrain determining unit 432 determines the 3D model of the construction machine determined by the construction machine recognizing unit 422 from the 3D model data of the terrain (including the construction machine and the siding) from the distance image integration processing unit 431. Is removed to generate three-dimensional model data of pure terrain (including stakes) that does not include construction machinery at the construction work site. This three-dimensional model data of pure terrain is used as an element of GIS data and stored in the GIS data storage unit 4
Stored in 4. (Note that, as described above, the construction machine recognition unit 4
When 22 is omitted, the terrain determination unit 432 is also omitted. In that case, the three-dimensional model data of the terrain including the construction machine from the distance image integration processing unit 431 is stored in the GIS data storage unit 44. )

Referring again to FIG. The rendering unit 46 in the host device 4 uses the GI in the GIS data storage unit 44.
Information terminal 20 for construction equipment using S data (terrain that changes in real time and dynamic three-dimensional model data of construction equipment)
A moving image of the construction work site 5 is drawn with the viewpoint parameter specified by the viewpoint operation unit 201 of 0. Where G
Since the position and direction of the construction machine are known in the IS data, the construction machine and the terrain other than the construction machine can be displayed in different colors in the moving image. Especially for construction machinery 3
When the dimensional model data is extracted, it is possible to clearly display the construction machine from other terrain. Since the three-dimensional model of the terrain and the position and direction of the construction machine and the three-dimensional model are dynamic ones that follow the movement of the moving image captured by the camera, the image display based on those data also moves. You can Furthermore, since the function of the marking detection unit 45 described below can identify the marking portion from the three-dimensional model data of the terrain in the GIS data, the marking portion can be identified in the moving image. Can also be displayed separately from other topography.

FIG. 6 shows a detailed functional configuration of the stitch detection unit 45.

As shown in FIG. 6, the tension detection unit 45 is
A three-dimensional topography is specified based on the marking position input unit 451 for inputting the marking position on the display image by the operator and the viewpoint parameter of the displayed image, and the specified marking position and the viewpoint parameter. A tension position determining unit 452 for calculating the tension position from the model data,
A three-dimensional model of a stake by identifying a stencil pattern storage unit 453 storing various three-dimensional shape characteristic patterns of a stab and a three-dimensional model data of the terrain from the three-dimensional model data of the terrain. A sizing recognition unit 454 that stores data in the GIS data storage unit 44 as an element of GIS data is provided.

The position specification of the marking on the display image is input from the marking specifying unit 202 of the information terminal 200 of the construction machine to the marking specifying input unit 451. That is, the operator of the construction machine usually finds a stake in the construction work site image displayed on the display device 203 before starting the work, and designates the stake portion in the display image. Designate by operating the unit 202 (for example, by placing the cursor on the sticking portion on the display screen and pressing the designation button). Then, the coordinate values in the designated marking portion display image are input to the marking designation input unit 451 as the marking position designation information. At the same time, the viewpoint parameter used by the rendering unit 46 when creating the display image is also input to the stitching specification input unit 451. The marking specification input unit 451 estimates a rough three-dimensional coordinate value of the marking based on the coordinate value of the marking portion in the display image and the viewpoint parameter of the display image.

The marking position determining unit 452 uses the GI to determine the approximate three-dimensional coordinate value of the marking from the marking specifying input unit 451.
By matching with the coordinate values of the three-dimensional model data of the terrain from the S data storage unit 44, the three-dimensional coordinate values estimated to be the sticking in the three-dimensional model data of the terrain are calculated.

The signature recognition unit 454 extracts from the 3D model data of the terrain the 3D model data of the area near the 3D coordinate values estimated to be the signature determined by the signature position determination unit 452. 3 of the extracted neighborhood area
By comparing the shape features of the dimensional model data with various shape feature patterns of the straits stored in the stencil pattern storage unit 453, the portion corresponding to the stencil in the three-dimensional terrain model data can be identified. , And stores the identified three-dimensional model data (three-dimensional coordinate values) in the GIS data storage unit 44. When generating the image, the rendering unit 46 uses not only the three-dimensional model data of the terrain but also the three-dimensional model data of the terrain to generate an image in which the territory and other terrain are clearly distinguished by color coding or the like. You can do it. Once installed, the three-dimensional model data of the three-dimensional model (identified at a certain point) is usually used, as it usually remains immobile even if the work progresses unless it is destroyed or intentionally moved. The three-dimensional coordinate value) is continuously used for each frame generation of the moving image by the rendering unit 46 after that point. Further, by doing so, even if the tine is destroyed, the tine before the rupture can be displayed on the display image, so that the construction work can be correctly continued.

It should be noted that by omitting the processing of indexing the 3D model data of the marking by the marking recognition unit 454 described above, the 3D position of the marking estimated by the marking position determining unit 452 is converted into GIS data. It may be stored as an element, and when the image is rendered, a mark or the like may be displayed at a position on the image corresponding to the three-dimensional position of the strut. Further, in addition to or instead of the processing of detecting the stitch and displaying it on the display image as described above, the operator draws the virtual design line 8 indicated by the stitch on the display image, The design line may be displayed on the display image. Further, instead of having the operator specify the stencil or the virtual design line as described above, it is possible to detect the stencil and set the virtual design line completely automatically by using the feature point extraction and pattern recognition techniques. Although possible, it is considered that the above-mentioned method of borrowing an operator's hand is practical because the burden on the system can be reduced by utilizing a very high recognition ability of human beings.

FIG. 7 shows examples of display images viewed from different virtual viewpoints.

As shown in FIG. 7, for example, the construction machine terminal 200
A plurality of viewpoint selection buttons “viewpoint 1” to “viewpoint 4” provided on the display selection unit 211 provided on the viewpoint operation unit 201
It is possible to previously assign viewpoint parameters of preset virtual viewpoints such as a guideman's viewpoint, an operator's viewpoint, an aerial viewpoint, and a sideways horizontal viewpoint. And
By pressing any button of "viewpoint 1" to "viewpoint 4" and selecting a desired viewpoint, the images 231 to 234 of the construction site viewed from those virtual viewpoints can be selectively displayed on the display device 203. it can.

The above-described embodiments of the present invention are examples for explaining the present invention, and the scope of the present invention is not limited only to those embodiments. Those skilled in the art can implement the present invention in various other modes without departing from the gist of the present invention.

For example, in the above-described embodiment, the position and orientation of the construction machine are detected by using the markers installed on the construction machine. However, this is done by pattern matching with the construction machine image or the construction machine. It may be replaced with the detection using the color information specific to the.

Further, in the above embodiment, the distance images are combined for each stereo camera and the integration processing is performed by the host device. However, all the images taken by the cameras are transmitted to the host device, and the host device Can do all the processing consistently.

Further, the finally output data is GI.
The data need not be S data, and may be data in other formats.

The source of the image does not necessarily have to be the moving image camera as in the above-mentioned embodiment, and may be a still image camera or an image acquired from the network.

Further, in the above-described embodiment, the three-dimensional model data of the work site is once created from the images of the work site captured by a plurality of cameras, and based on the three-dimensional model data, from an arbitrary viewpoint. An image of the work site is generated. However, since the three-dimensional position of the camera is determined, the relationship between the image of the work site taken by those cameras and the image of the work site viewed from an arbitrary viewpoint is geometrically determined. It is also possible to generate and display an image viewed from an arbitrary viewpoint directly from the photographed image (that is, without creating three-dimensional model data) using such a relationship. Also in this case, regarding the display of the marking or the virtual design line, for example, the operator designates on the display image as in the above-described embodiment, and the three-dimensional coordinate of the marking or the virtual design line is divided based on the designation. It is possible to adopt a method in which the drawing position is determined, and the position of the taut or the position of the virtual design line on the image from the arbitrary viewpoint is determined and displayed based on the three-dimensional coordinates.

[Brief description of drawings]

FIG. 1 is a diagram showing the arrangement of cameras in a work site image generation system according to an embodiment of the present invention.

FIG. 2 is a diagram showing a schematic operation flow of the work site image generation system according to the embodiment of the present invention.

FIG. 3 is a diagram showing a functional configuration of a work site image generation system according to the embodiment of the present invention.

FIG. 4 is a diagram illustrating a detailed functional configuration of an image processing unit.

FIG. 5 is a diagram showing a detailed functional configuration of a portion for generating a three-dimensional model of a host device.

FIG. 6 is a diagram showing a functional configuration of a stitching detection unit of a host device.

FIG. 7 is a diagram showing an example of a display image viewed from different virtual viewpoints.

[Explanation of symbols]

1 ... camera pole, 2 ... camera, 3 ... stereo camera,
4 ... Host device, 5 ... Construction work site, 6 ... Construction machine, 7
… Collecting, 8… Virtual design line

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hiroshi Miwa             1200 Manda, Hiratsuka-shi, Kanagawa Made by Komatsu Ltd.             Seisakusho Institute (72) Inventor Hiroyoshi Yamaguchi             1200 Manda, Hiratsuka-shi, Kanagawa Made by Komatsu Ltd.             Seisakusho Institute F-term (reference) 5B050 AA03 BA04 BA09 BA11 DA07                       EA27 FA02 FA06                 5B057 AA20 CA12 CB13 CD14 CH01                       DA16                 5C054 CE15 CF03 DA07 EH00 FE02                       HA00 HA05

Claims (3)

[Claims] 1. An image input unit for inputting a plurality of images of a site viewed from a predetermined different place on the site, a viewpoint designating unit for designating a desired virtual viewpoint, and a plurality of images of the site, An image generating system for a scene, comprising: an image generating unit that generates an image of the scene viewed from the designated virtual viewpoint. 2. The work site image generation system according to claim 1, further comprising: a three-dimensional modeling unit configured to generate three-dimensional model data of the work site. 3. A step of inputting a plurality of images of a scene viewed from a predetermined different place of the scene, a step of designating a desired virtual viewpoint, and a plurality of images of the scene being used for the designation. Generating an image of the site viewed from a virtual viewpoint.