JP5575571B2 - Image generation system - Google Patents

Description

  The present invention relates to an image generation system that generates an output image for presentation to a crane operator based on an input image captured by an imaging device attached to a crane.

  Conventionally, a guideline for converting an image captured by the camera 1 attached to the upper swing body of a mobile crane into a bird's eye view image and projecting a trajectory drawn by the tip of the boom of the mobile crane when turning to the ground. A camera system that makes it easy to check the surrounding state of the swivel body by drawing the image on the bird's eye view image is known (for example, see Patent Document 1).

  A traveling mark drawn on the wall surface of the building that supports the girder runway included in the image captured by the camera fixedly attached to the girder of the overhead crane traveling in the X-axis direction. XY coordinate value representing the position of the trolley based on the traverse mark drawn on the surface of the trolley that traverses in the axial direction, and the image of the hook hanging from the hoisting device installed on the trolley, and the A crane position detection system that calculates vertical coordinate values representing the position of a hook and presents the calculated value to an operator is known (see, for example, Patent Document 2).

  Also known is a container collision prevention device for preventing collision between containers while detecting the bottom edge of the handling container and the ceiling edge of the stack container based on the output of a two-dimensional laser sensor attached to the trolley of the yard crane. (For example, refer to Patent Document 3).

JP 2008-312004 A Japanese Patent No. 4209532 Japanese Patent No. 4295591

  However, the camera system described in Patent Document 1 is for an operator who operates a crane while visually recognizing a hook that rotates with a revolving body from the horizontal direction (an operator in a cockpit located close to the ground). It is intended to present in advance the turning trajectory of the hook when the turning body turns, and can provide useful image information to an operator who operates the crane while looking down the hook obliquely from above. Can not.

  Moreover, the crane position detection system described in Patent Document 2 only presents various coordinate values to the operator without displaying an image captured by the camera. Because it only controls the rotation speed of the hoisting device according to the output of the two-dimensional laser sensor without imaging the image, it is useful for the operator who operates the crane while looking down from the upper side of the hook or spreader. Image information cannot be presented.

  In view of the above-described points, an object of the present invention is to provide an image generation system capable of presenting useful image information to an operator who operates a crane while looking down at the periphery of a hook from vertically above.

In order to achieve the above-described object, an image generation system according to an embodiment of the present invention is an image generation system that supports the operation of a crane that operates while looking down the periphery of a suspension tool from a diagonally upward direction, and a hoisting device includes: an imaging device for imaging a rolled up or wound lowering the load-lifting vertically from obliquely above, performs viewpoint conversion processing on the image which the imaging device imaged, and an output image generation means for generating an output image viewed the sling from the horizontal direction with Ru. An image generation system according to another embodiment of the present invention is an image generation system that supports operation of a crane, and the image pickup device that picks up the lifting tool that the hoisting device winds up or lowers from vertically above, Output image generating means for performing viewpoint conversion processing on the image captured by the imaging device and generating an output image of the suspension viewed from the horizontal direction, and the crane includes a girder that travels along the wall of the building; An overhead crane having a trolley traversing on the girder, and the imaging device and the hoisting device are attached to the trolley and traverse along with the trolley. Further, an image generation system according to still another embodiment of the present invention is an image generation system that supports crane operation, and an imaging device that captures a lifting tool that is hoisted or lowered by a hoisting device from vertically above, Output image generating means for performing viewpoint conversion processing on the image captured by the imaging device and generating an output image of the suspension viewed from the horizontal direction, and the crane includes a girder that travels along the wall of the building; An overhead crane having a trolley traversing on the girder, wherein the hoisting device is attached to the trolley and traverses along with the trolley, and the imaging device travels along the traversing direction of the trolley. Several are installed.

  With the above-described means, the present invention can provide an image generation system that can present useful image information to an operator who operates the crane while looking down around the hook.

1 is a block diagram schematically showing a configuration example of an image generation system according to a first embodiment of the present invention. It is a figure which shows the structural example of the overhead crane by which the image generation system of FIG. 1 is mounted. It is FIG. (1) which shows an example of the space model on which an input image is projected. It is a figure which shows the positional relationship between the space model of FIG. 3, and an overhead crane. It is an example of the input image which the camera imaged. It is a figure for demonstrating matching with the coordinate on an input image plane, and the coordinate on a space model. It is a figure for demonstrating matching with the coordinate on an input image plane, the coordinate on a space model, and the coordinate on an output image plane. 6 is an example (part 1) of an output image based on the space model of FIG. 3 and the input image of FIG. 6 is an example (part 2) of an output image based on the spatial model of FIG. 3 and the input image of FIG. It is FIG. (2) which shows an example of the space model on which an input image is projected. It is an example of the output image based on the space model of FIG. 10, and the input image of FIG. It is a flowchart which shows the flow of an output image generation process. It is a block diagram which shows roughly the structural example of the image generation system which concerns on 2nd Example of this invention. It is a block diagram which shows roughly the structural example of the image generation system which concerns on 3rd Example of this invention. FIG. 10 is a diagram (No. 1) for describing a process of generating a virtual looking-down image by viewpoint conversion; FIG. 10 is a diagram (No. 2) for describing a process of generating a virtual looking-down image by viewpoint conversion; FIG. 10 is a diagram (No. 1) for describing a process of generating a virtual looking-down image and a virtual horizontal image; FIG. 10 is a diagram (No. 2) for describing a process of generating a virtual looking-down image and a virtual horizontal image. It is a block diagram which shows roughly the structural example of the image generation system which concerns on 4th Example of this invention. It is a figure which shows the structural example of the overhead crane by which the image generation system of FIG. 19 is mounted. It is FIG. (1) for demonstrating the scale drawing process by a scale drawing means. It is FIG. (2) for demonstrating the scale drawing process by a scale drawing means. It is FIG. (3) for demonstrating the scale drawing process by a scale drawing means. It is FIG. (1) which shows the example of the output image containing the several image from which a characteristic differs. It is FIG. (2) which shows the example of the output image containing the several image from which a characteristic differs.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically showing a configuration example of an image generation system 100 according to the first embodiment of the present invention.

  The image generation system 100 is, for example, a system that generates an output image based on an input image captured by the camera 2 mounted on the overhead crane 60 and presents the output image to the driver. , An input unit 3, a storage unit 4, and a display unit 5.

  FIG. 2 is a diagram illustrating a configuration example of the overhead crane 60 on which the image generation system 100 is mounted. FIG. 2A illustrates a top view of the overhead crane 60, and FIG. 2B illustrates FIG. Fig. 2C is a front view of the overhead crane 60 viewed from the IIB-IIB direction, and Fig. 2C is a side view of the overhead crane 60 viewed from the IIC-IIC direction of Fig. 2B. 2A to 2C is an imaging range of the camera 2.

  The overhead crane 60 has wheels 62L (including a front wheel 62LF and a rear wheel 62LB) and 62R (a front wheel 62RF (not shown) and an illustration shown) on runways 61L and 61R installed on the left wall WL and the right wall WR of the building, respectively. The rear wheel 62RB is included) and the girder 63 travels in the X-axis direction.

  The overhead crane 60 has wheels 65L (including front wheels 65LF and rear wheels 65LB not shown) and 65R (including front wheels 65RF and rear wheels 65RB) on runways 64F and 64B installed on the girder 63. And a trolley 66 that traverses in the Y-axis direction.

  The overhead crane 60 has a hoisting device 67 attached to a trolley 66, and the hoisting device 67 winds a wire rope 68 around a drum (not shown) using a motor (not shown). The hook 69 is lowered and moved up and down in the Z-axis direction.

  In this embodiment, the overhead crane 60 is a club trolley type overhead crane, but may be a rope trolley type overhead crane or a hoist type overhead crane.

  Next, each component of the image generation system 100 will be described.

  The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), etc. A program corresponding to each of the attaching means 10 and the output image generating means 11 is stored in the ROM or NVRAM, and the CPU executes processing corresponding to each means while using the RAM as a temporary storage area.

  The camera 2 is a device for acquiring an input image that reflects the periphery of the hook 69. For example, the camera 2 includes an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and is vertically downward from the trolley 66. After extending, it is attached to the tip of the bracket BR extending in the X-axis direction, traverses in the Y-axis direction together with the trolley 66, and travels in the X-axis direction together with the girder 63 (see FIGS. 2A and 2C). .)

  The camera 2 is arranged so as to pick up an image of the hook 69 from above in the vertical direction, and even when the hook 69 enters a recess (a space surrounded by other objects or walls), the hook 69 is moved. The optical axis is arranged so as to form a sufficient depression angle for reliable imaging.

  As the depression angle formed by the optical axis of the camera 2 increases, even when the hook 69 enters a narrower recess, the hook 69 can capture an image without being blocked by the surrounding objects, The smaller the depression angle formed by the optical axis, the clearer the later-described viewpoint conversion image (as if the hook 69 was viewed from the horizontal direction) can be presented more clearly (the optical axis of the camera 2 and the virtual axis). This is because the angle difference from the optical axis of the camera is reduced.)

  The camera 2 may be one in which the operator can dynamically adjust the direction of the optical axis, and the direction of the optical axis is directed toward the hook 69 while detecting the position of the hook 69 by a known technique. As described above, the hook 69 may be automatically tracked.

  In addition, the camera 2 may be attached to the side surface of the girder 63 (in this case, it does not traverse with the trolley 66), and a wide-angle lens or a fish-eye lens may be attached so that a wide range can be imaged.

  In addition, the camera 2 acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1. In addition, when the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image obtained by correcting apparent distortion and tilt caused by using these lenses is transmitted to the control unit 1. Although it is output, an input image in which the apparent distortion or tilt is not corrected may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

  The input unit 3 is a device that allows an operator to input various types of information to the image generation system 100. Examples of the input unit 3 include a touch panel, a button switch, a pointing device, and a keyboard.

  The storage unit 4 is a device for storing various types of information, and is, for example, a hard disk, an optical disk, or a semiconductor memory.

  The display unit 5 is a device for displaying image information. For example, the display unit 5 is a liquid crystal display or a projector installed in the operation room 70 and displays various images output from the control unit 1.

  Further, the image generation system 100 generates a processing target image based on the input image, and performs an image conversion process on the processing target image, thereby intuitively determining the positional relationship and the sense of distance between the hook 69 and the suspended load. An output image that can be grasped is generated, and the output image is presented to the driver.

  The “processing target image” is an image that is generated based on the input image and is a target of image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, and the like). This is an image suitable for image conversion processing, obtained by projecting an input image from a camera that captures image 69 from above onto a spatial model. The processing target image may be used as an output image as it is without performing an image conversion process.

  The “spatial model” is an input image projection target composed of one or a plurality of planes or curved surfaces.

  FIG. 3 is a diagram illustrating an example of a spatial model MD onto which an input image is projected. The spatial model MD is formed by two orthogonal planes (horizontal plane R1 and vertical plane R2). For convenience of explanation, the horizontal plane R1 has a grid pattern formed by a plurality of straight lines drawn at equal intervals in the vertical and horizontal directions, and the vertical plane R2 is arranged at equal intervals in the vertical and horizontal directions. It is assumed that a dot pattern formed by dots (circles) to be formed is included, but these patterns are not actually displayed on the image.

  FIG. 4 is a diagram showing a positional relationship between the space model MD and the overhead crane 60, and FIG. 4 (A) is a floor vertically below the overhead crane 60 shown in the top view of FIG. 2 (A). FIG. 4B shows a state in which the horizontal plane R1 of the space model MD is arranged on the plane FF, and FIG. The state which has arrange | positioned the vertical surface R2 of model MD is shown.

  FIG. 5 is a diagram illustrating an example of an input image captured by the camera 2, and shows a state where the hook 69 suspended by the wire rope 68 is looked down obliquely from above. The horizontal plane R1 and the vertical plane R2 of the space model MD are displayed for convenience of explanation, and do not exist in the actual image.

  Moreover, FIG. 5 shows that the dot pattern of the vertical surface R2 is smaller and the interval is narrower as it approaches the floor surface FF (horizontal plane R1).

  Next, various units included in the control unit 1 will be described.

  The coordinate associating means 10 is a means for associating coordinates on the input image plane where the input image captured by the camera 2 is located with coordinates on the space model MD, for example, preset or input Various parameters related to the camera 2 such as the optical center, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method, etc., which are input via the unit 3, and a predetermined input image plane And the coordinates on the input image plane and the coordinates on the space model MD are associated with each other based on the positional relationship between the space model MD and the corresponding relationship in the input image / space model correspondence map 40 of the storage unit 4. Remember.

  The output image generation means 11 is preset or input via the input unit 3, such as the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, and the projection method. Based on the various parameters, the coordinates on the spatial model MD are associated with the coordinates on the output image plane where the output image is located, the correspondence is stored in the spatial model / output image correspondence map 41 of the storage unit 4, and Referring to the input image / space model correspondence map 40, the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) and the value of each pixel in the input image are associated with each other. Generate an output image.

  Next, an example of specific processing by the coordinate association unit 10 and the output image generation unit 11 will be described.

  The coordinate association means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, a Hamilton quaternion.

  FIG. 6 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the space model. The input image plane R4 of the camera 2 is UVW orthogonal with the optical center C of the camera 2 as the origin. It is expressed as one plane in the coordinate system, and the space model MD is expressed as a three-dimensional surface in the XYZ orthogonal coordinate system.

  First, the coordinate association unit 10 converts the coordinates on the space model MD (coordinates on the XYZ coordinate system) to the coordinates on the input image plane R4 (coordinates on the UVW coordinate system), and therefore the origin of the XYZ coordinate system. O is moved parallel to the optical center C (the origin of the UVW coordinate system), then the X axis is the U axis, the Y axis is the V axis, and the Z axis is the -W axis (the sign "-" is in the opposite direction). This means that the XYZ coordinate system is rotated so that the UVW coordinate system coincides with the + W direction in front of the camera and the XYZ coordinate system in the −Z direction vertically below.

  When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system. Therefore, the coordinate association unit 10 uses an XYZ coordinate system in parallel with each of the plurality of UVW coordinate systems. It will be moved and rotated.

  In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 is the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis. Since it is realized by rotating to coincide with the axis, the coordinate matching means 10 can combine these two rotations into one rotation calculation by describing this transformation in Hamilton's quaternion. it can.

  By the way, the rotation for making one vector A coincide with another vector B corresponds to the process of rotating the vector A and the vector B by the angle formed by using the normal line of the surface extending between the vector A and the vector B as an axis. If the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

で表されることとなる。

It will be expressed as

  Further, the unit vector N of the normal line between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

で表されることとなる。

It will be expressed as

  Note that the quaternion has i, j, and k as imaginary units,

を満たす超複素数であり、本実施例において、四元数Ｑは、実成分をｔ、純虚成分をａ、ｂ、ｃとして、

In this embodiment, the quaternion Q is represented by t as a real component and a, b, and c as pure imaginary components.

で表されるものとし、四元数Ｑの共役四元数は、

The conjugate quaternion of the quaternion Q is

で表されるものとする。

It shall be represented by

  The quaternion Q can represent a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero), and t, a, b , C can also be used to express a rotational motion with an arbitrary vector as an axis.

  Further, the quaternion Q can be expressed as a single rotation operation by integrating a plurality of continuous rotation operations. For example, an arbitrary point S (sx, sy, sz) can be expressed as an arbitrary unit vector. A point D (ex, ey, ez) when rotated by an angle θ with C (l, m, n) as an axis can be expressed as follows.

ここで、本実施例において、Ｚ軸を−Ｗ軸に一致させる回転を表す四元数をＱｚとすると、ＸＹＺ座標系におけるＸ軸上の点Ｘは、点Ｘ'に移動させられるので、点Ｘ'は、

Here, in this embodiment, if the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X ′. X '

で表されることとなる。

It will be expressed as

  In this embodiment, if the quaternion representing the rotation that matches the line connecting the point X ′ on the X axis and the origin to the U axis is Qx, “the Z axis matches the −W axis, , A quaternion R representing "rotation to make the X axis coincide with the U axis"

で表されることとなる。

It will be expressed as

  As described above, the coordinate P ′ when an arbitrary coordinate P on the space model (XYZ coordinate system) is expressed by a coordinate on the input image plane (UVW coordinate system) is

で表されることとなり、座標対応付け手段１０は、以後、この演算を実行するだけで空間モデル（ＸＹＺ座標系）上の座標を入力画像平面（ＵＶＷ座標系）上の座標に変換することができる。なお、カメラ２の光軸が固定である場合には四元数Ｒ及びその共役四元数Ｒ^＊は不変であるが、カメラ２の光軸が調整される場合には、四元数Ｒ及びその共役四元数Ｒ^＊は再計算される必要がある点に留意すべきである。
四元数Ｒがカメラ２のそれぞれで不変であることから、
空間モデル（ＸＹＺ座標系）上の座標を入力画像平面（ＵＶＷ座標系）上の座標に変換した後、座標対応付け手段１０は、カメラ２の光学中心Ｃ（ＵＶＷ座標系上の座標）と空間モデル上の任意の座標ＰをＵＶＷ座標系で表した座標Ｐ'とを結ぶ線分ＣＰ'と、カメラ２の光軸Ｇとが形成する入射角αを算出する。

Then, the coordinate matching means 10 can convert the coordinates on the space model (XYZ coordinate system) into the coordinates on the input image plane (UVW coordinate system) by simply executing this calculation. it can. When the optical axis of the camera 2 is fixed, the quaternion R and its conjugate quaternion R ^* are unchanged, but when the optical axis of the camera 2 is adjusted, the quaternion R and Note that the conjugated quaternion R ^* needs to be recalculated.
Since the quaternion R is invariant for each camera 2,
After the coordinates on the space model (XYZ coordinate system) are converted to the coordinates on the input image plane (UVW coordinate system), the coordinate association means 10 determines the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the space. An incident angle α formed by a line segment CP ′ connecting an arbitrary coordinate P on the model with a coordinate P ′ represented in the UVW coordinate system and the optical axis G of the camera 2 is calculated.

  In addition, the coordinate association unit 10 includes an intersection E between the plane H and the optical axis G, and a coordinate P ′ in a plane H that is parallel to the input image plane R4 (for example, CCD plane) of the camera 2 and includes the coordinate P ′. Are calculated, and the deviation angle φ formed by the line segment EP ′ connecting the two and the U ′ axis in the plane H, and the length of the line segment EP ′.

  In the optical system of the camera, the image height h is usually a function of the incident angle α and the focal length f. Therefore, the coordinate matching means 10 performs normal projection (h = ftanα) and orthographic projection (h = fsinα). Image height by selecting an appropriate projection method such as stereo projection (h = 2 ftan (α / 2)), equisolid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc. Calculate h.

  Thereafter, the coordinate matching means 10 decomposes the calculated image height h into U and V components on the UV coordinate system by the declination φ, and is a numerical value corresponding to the pixel size per pixel of the input image plane R4. By dividing, the coordinates P (P ′) on the space model MD can be associated with the coordinates on the input image plane R4.

  If the pixel size per pixel in the U-axis direction of the input image plane R4 is aU and the pixel size per pixel in the V-axis direction of the input image plane R4 is aV, the coordinates P (P The coordinates (u, v) on the input image plane R4 corresponding to ') are

で表されることとなる。

It will be expressed as

  In this way, the coordinate association unit 10 associates the coordinates on the space model MD with the coordinates on one or a plurality of input image planes R4 existing for each camera, and coordinates on the space model MD, the camera identifier. And the coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

  Further, since the coordinate association unit 10 calculates the coordinate conversion using the quaternion, unlike the case where the coordinate conversion is calculated using the Euler angle, there is an advantage that no gimbal lock is generated. . However, the coordinate association unit 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may perform the coordinate conversion using the Euler angle.

  When one coordinate on the space model MD can be associated with a plurality of coordinates on the input image plane, the coordinate association means 10 determines the coordinates P (P ′) on the space model MD as the incident angle α. May be associated with coordinates on the input image plane relating to the camera with the smallest image size, may be associated with coordinates on the input image plane selected by the operator, or automatically selected by the image generation system 100 You may make it match with the coordinate on an input image plane.

  Next, the association between coordinates by the coordinate association means 10 will be described with reference to FIG.

  FIG. 7A is a diagram showing a correspondence relationship between coordinates on the input image plane R4 of the camera 2 that adopts normal projection (h = ftan α) as an example and coordinates on the space model MD. The attaching means 10 sets both coordinates so that each line segment connecting the coordinates on the input image plane R4 of the camera 2 and the coordinates on the spatial model MD corresponding to the coordinates passes through the optical center C of the camera 2. Associate.

  In the example of FIG. 7A, the coordinate association means 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the horizontal plane R1 of the spatial model MD, and on the input image plane R4 of the camera 2 Is associated with the coordinate L2 on the vertical plane R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

  In addition, when the camera 2 employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the coordinate association unit 10 uses each projection method. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the space model MD.

  Specifically, the coordinate association unit 10 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the space model MD. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

  FIG. 7B is a diagram illustrating a correspondence relationship between the coordinates on the space model MD and the coordinates on the output image plane R5 of the virtual camera 2V that adopts the normal projection (h = ftan α) as an example. The output image generation means 11 causes each line segment connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the spatial model MD corresponding to the coordinates to pass through the optical center CV of the virtual camera 2V. And associate both coordinates.

  In the example of FIG. 7B, the output image generation means 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates L1 on the horizontal plane R1 of the space model MD, and outputs the output image plane of the virtual camera 2V. The coordinate N2 on R5 is associated with the coordinate L2 on the vertical plane R2 of the space model MD. At this time, both the line segment L1-N1 and the line segment L2-N2 pass through the optical center CV of the virtual camera 2V.

  Note that when the virtual camera 2V employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the output image generation unit 11 uses each projection method. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates L1 and L2 on the space model MD.

  Specifically, the output image generation means 11 is a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), equal solid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the space model MD are associated with each other. In this case, the line segment L1-N1 and the line segment L2-N2 do not pass through the optical center CV of the virtual camera 2V.

  In this way, the output image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the space model MD, and associates the coordinates on the output image plane R5 with the coordinates on the space model MD to create a space. The value of each pixel in the output image is associated with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 stored in the model / output image correspondence map 41 and stored in the coordinate association means 10. To generate an output image.

  FIG. 8 is a view showing an example of an output image based on the input image of FIG. Note that the horizontal plane R1 and the vertical plane R2 of the spatial model MD are displayed for convenience of explanation as in the case of FIG. 5 and do not exist in the actual image.

  Further, FIG. 8 shows that the size of the dot pattern on the vertical surface R2 and the interval thereof are almost uniform regardless of the distance from the floor surface FF (horizontal surface R1), unlike the case of FIG.

  FIG. 9 is a diagram illustrating another example of the output image based on the input image of FIG. 5, and is different from the output image of FIG. 8 in that a scale indicating the position of the hook 69 is superimposed and displayed. In FIG. 9, the dot pattern on the vertical surface VS (vertical surface R2) is omitted for the sake of clarity.

  For example, the scale indicating the distance is displayed so as to be thicker in a region where the enlargement ratio by the viewpoint conversion becomes larger (in this case, closer to the floor surface FF), but is uniform regardless of the enlargement ratio by the viewpoint conversion. You may display by thickness. In addition, an arrow image AR for showing the level of the hook 69 in an easy-to-understand manner may be displayed.

  The scale value preferably represents the distance from the hoisting device 67 (the memory value is zero at the level of the hoisting device 67), but represents the distance from the tip of the hook 69 ( The value of the scale at the level of the hook 69 may be zero) or the distance from the floor FF (the value of the scale at the level of the floor FF becomes zero). .

  FIG. 10 is a diagram illustrating another example of a spatial model onto which an input image is projected. The spatial model MDA is similar to the spatial model MD in FIG. 3 by two orthogonal planes (horizontal plane R1A and vertical plane R2A). It is formed. The horizontal plane R1A has a grid pattern, the vertical plane R2A has a dot pattern, and the point that these patterns are not actually displayed on the image is the same as in the case of FIG.

  FIG. 11 is a diagram illustrating an example of an output image based on the spatial model MDA of FIG. 10 and the input image of FIG. 5. An image of the hook 69 suspended by the wire rope 68 viewed from the horizontal direction and the floor FF are shown. The state seen from diagonally combining with the image seen from the perpendicular upper part is shown.

  Note that the output image shown in FIG. 11 may be displayed with a scale indicating the position of the hook 69 superimposed, similarly to the output image shown in FIG.

  As described above, the image generation system 100 can present an image as if the hook 69 was captured from the horizontal direction (hereinafter referred to as “virtual horizontal image”) to the operator. The operator can easily recognize the distance to the surface FF, and can assist the operator in raising and lowering the hook 69.

  In addition, the image generation system 100 presents a portion of the input image where the hook 69 exists to the operator as a virtual horizontal image, while the other portion of the input image is displayed as if it is vertically above. The captured image can be presented to the operator, and the operator can easily recognize the relationship between the hook 69 and the object existing on the floor surface FF. The operation of raising and lowering can be supported.

  Next, a process in which the image generation system 100 generates an output image using an input image (hereinafter referred to as “output image generation process”) will be described with reference to FIG. FIG. 12 is a flowchart showing the flow of the output image generation processing. The camera 2 (input image plane R4), the space model MD (horizontal plane R1 and vertical plane R2), and the virtual camera 2V (output image plane R5). The arrangement is determined in advance.

  First, the control unit 1 causes the output image generation unit 11 to associate the coordinates on the output image plane R5 with the coordinates on the space model MD (step S1).

  Specifically, when generating the output image using the virtual camera 2V, the output image generation unit 11 calculates the coordinates on the space model MD from the coordinates on the output image plane R5 according to the adopted projection method. The correspondence relationship is stored in the space model / output image correspondence map 41.

  More specifically, the output image generation means 11 acquires the coordinates of the optical center CV of the virtual camera 2V when generating an output image using the virtual camera 2V that employs normal projection (h = ftanα). Then, a point that is a line segment extending from one coordinate on the output image plane R5 and that intersects the space model MD is calculated on the space model MD corresponding to the calculated point. The coordinates are derived as one coordinate on the space model MD corresponding to the one coordinate on the output image plane R5, and the corresponding relationship is stored in the space model / output image correspondence map 41.

  Thereafter, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the space model (step S2), and still associates all the coordinate values. If it is determined that there is not (NO in step S2), the process in step S1 is repeated.

  On the other hand, when the control unit 1 determines that all the coordinate values are associated (YES in step S2), each of the coordinates on the space model MD derived by the coordinate association unit 10 by the above-described processing. Are associated with coordinates on the input image plane R4 (step S3).

Specifically, the coordinate matching means 10 acquires the coordinates of the optical center C of the camera 2 that employs normal projection (h = ftanα), and from each of the coordinates on the spatial model MD derived by the above-described processing. A line segment extending through the optical center C intersects with the input image plane R4, and a coordinate on the input image plane R4 corresponding to the calculated point is set as one coordinate on the space model MD. Is derived as one coordinate on the input image plane R4, and the corresponding relationship is stored in the input image / space model correspondence map 40.

  Thereafter, the control unit 1 determines whether or not all the coordinates on the spatial model MD derived by the above-described process are associated with the coordinates on the input image plane R4 (step S4), and all the coordinates are still associated. If it is determined that it is not attached (NO in step S4), the process in step S3 is repeated.

  On the other hand, if the control unit 1 determines that all coordinates are associated (YES in step S4), the output image generation unit 11 causes the input image / space model correspondence map 40 and the space model / output image correspondence map 41 to be matched. , The correspondence between the coordinates on the input image plane R4 and the coordinates on the space model MD, and the correspondence between the coordinates on the space model MD and the coordinates on the output image plane R5 are traced, and the output image plane R5 A value (for example, a luminance value, a hue value, a saturation value, etc.) possessed by coordinates on the input image plane R4 corresponding to each of the above coordinates is acquired, and the acquired value is used as the corresponding output image plane R5. The values of the above coordinates are adopted (step S5). When a plurality of coordinates on one or a plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the output image generation unit 11 calculates a statistical value based on each value of the plurality of coordinates. (For example, an average value, a maximum value, a minimum value, an intermediate value, etc.) may be derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5.

  Thereafter, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6), and all the coordinate values are still associated with each other. If it is determined that there is not (NO in step S6), the process in step S5 is repeated.

  On the other hand, if it is determined that all coordinate values are associated (YES in step S6), the control unit 1 generates an output image and outputs the generated output image to the display unit 5 (step S7). This series of processes is terminated.

  With the above configuration, the image generation system 100 can generate an output image that allows the operator to intuitively grasp the positional relationship between the hook 69 suspended by the wire rope 68 and the floor surface FF. The output image can be presented to assist the operator in raising and lowering the hook 69.

  In addition, the image generation system 100 executes coordinate association from the output image plane R5 to the input image plane R4 via the spatial model MD, so that each coordinate on the output image plane R5 is displayed on the input image plane R4. Compared with the case where coordinates are associated in the order from the input image plane R4 to the output image plane R5 through the spatial model MD (in this case, the input Each coordinate on the image plane R4 can be made to correspond to one or more coordinates on the output image plane R5. However, a part of the coordinates on the output image plane R5 is any coordinate on the input image plane R4. In such a case, it is necessary to perform interpolation processing or the like on some of the coordinates on the output image plane R5.), It is possible to quickly generate a higher quality output image. That.

  Further, when changing the viewpoint of the output image, the image generation system 100 only needs to rewrite the space model / output image correspondence map 41 by changing the values of various parameters of the virtual camera 2V, thereby supporting the input image / space model. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the map 40.

  Next, a second embodiment of the present invention will be described with reference to FIG.

  FIG. 13 is a block diagram schematically showing a configuration example of an image generation system 200 according to the second embodiment of the present invention.

  The image generation system 200 is different from the image generation system 100 in FIG. 1 in that the control unit 1 includes the suspension position detection unit 12, but is common in other points. Therefore, the difference will be described in detail while omitting the description of the common points.

  The suspension position detecting means 12 is a means for detecting the vertical position of the suspension. For example, the amount of feeding of the wire rope 68 is derived based on the output of the rotation angle sensor that measures the rotation angle of the drum in the hoisting device 67. Based on the feed amount, the current vertical position of the hook 69 (for example, the height from the ground) is detected, and information regarding the detected vertical position of the hook 69 is output to the output image generation means 11. .

  Moreover, the lifting tool position detection means 12 is based on the output of the distance sensor (For example, it is a distance sensor using light, an electromagnetic wave, or an ultrasonic wave) attached to the winding apparatus 67. ) And the hook 69 may be derived, and the current vertical position of the hook 69 may be detected based on the distance.

  In the second embodiment, the output image generation means 11 determines the vertical position and optical axis direction of the optical center of the virtual camera based on the information about the vertical position of the hook 69 received from the suspension position detection means 12, and the hook 69. Are generated as an output image.

  Specifically, for example, the output image generation unit 11 makes the vertical position of the hook 69 at the current time the same as the vertical position of the optical center of the virtual camera, and directs the optical axis direction to the hook 69. The vertical position of the optical center of the virtual camera is changed according to the change of the vertical position of the virtual camera.

  With the above configuration, the image generation system 200 can always present an output image in which the hook 69 is viewed from the side to the operator regardless of a change in the vertical position of the hook 69, and the hook suspended by the wire rope 68. The operator can intuitively grasp the positional relationship between 69 and the floor surface FF.

  Next, a third embodiment of the present invention will be described with reference to FIGS.

  FIG. 14 is a block diagram schematically showing a configuration example of an image generation system 300 according to the third embodiment of the present invention.

  The image generation system 300 is different from the image generation system 100 of FIG. 1 in that it has three cameras 20, 21, and 22, and the control unit 1 has unevenness detection means 13. Common. Therefore, the difference will be described in detail while omitting the description of the common points.

  As shown in FIG. 15A, each of the camera 20 and the camera 22 is a camera that is attached to the girder 63 and images the floor surface FF under the overhead crane 60. The control unit uses the captured images as input images. 1 is output. The camera 20 and the camera 22 may be attached to the trolley 66 so as to sandwich the trolley 66 in the transverse direction of the trolley 66.

  Similarly to the camera 2 in the image generation system 100, the camera 21 is a camera that is attached to the trolley 66 and traverses along with the trolley 66, and outputs the captured image to the control unit 1 as an input image.

  The unevenness detecting means 13 is a means for detecting unevenness of the floor surface FF, for example, a distance sensor (for example, a distance using light, radio waves, or ultrasonic waves) that is attached to the trolley 66 and traverses along with the trolley 66. Sensor is a distance sensor that is mounted across the hoisting device 67 in the transverse direction and scans the floor surface FF while traversing along with the trolley 66. The detection result is output to the coordinate association means 10.

  Further, the unevenness detecting means 13 is data relating to the unevenness of the floor surface FF stored in advance in a ROM or the like (for example, data consisting of a combination of XY coordinates and the height of the floor surface FF with respect to a predetermined reference level). The unevenness of the floor surface FF is detected based on the XY coordinates of the position (for example, the current position of the trolley 66 obtained by a known means), and the detection result is output to the coordinate association means 10. It may be.

  In the third embodiment, the coordinate association means 10 determines whether the height of the surface below the hook 69 is different from the height of the surrounding floor surface FF based on the output of the unevenness detection means 13, When it is determined that the height of the surface is different from the height of the surrounding floor surface FF, at the level difference between the surface and the floor surface FF, the blind spot of the camera 21 and one of the camera 20 or the camera 22 are displayed. Assuming that there is a blind area, instead of the captured image of the camera 21 (an image obtained by capturing an image of the surface below the hook 69 vertically obliquely from above, hereinafter referred to as “looking-down image”), the camera 20 and While taking each captured image of the camera 22 as an input image, the coordinates on the input image plane of the camera 20 and the camera 22 are associated with the coordinates on the space model MD. The above description can also be applied to the case where the camera 21 is not present.

  In addition, the coordinate association unit 10 uses the captured images of the cameras 20 to 22 as input images instead of the captured images (looking-down images) of the camera 21 alone on the input image planes of the cameras 20 to 22. You may make it match a coordinate and the coordinate on space model MD.

  FIG. 15 is a diagram for explaining a process of generating a virtual looking-down image by viewpoint conversion. FIG. 15A shows a front view of the crane 60 similar to FIG. (B) shows the respective captured images of the cameras 20 to 22, and FIG. 15 (C) shows the relationship between the captured images of the camera 20 and the camera 22 and the spatial model MD, and combinations of these captured images and FIG. 15D shows an output image obtained as a result of the viewpoint conversion, and FIG. 15D shows a relationship between the captured images of the camera 20, the camera 21, and the camera 22 and the spatial model MD, a combination of the captured images, and a viewpoint conversion. The output image obtained as a result of is shown.

  FIG. 15A shows a state where the suspended load OBJ is placed vertically below the hook 69 and the worker HM is present beside the suspended load OBJ. In addition, the dashed-dotted line of a figure, a broken line, and a dashed-two dotted line show each imaging range of the cameras 20-22, respectively.

  In this case, as shown in FIG. 15B, neither the captured image IM1 of the camera 20 nor the captured image IM2 of the camera 21 can easily grasp the presence of the worker HM, and the image of the camera 22 is captured. Only the image IM3 can easily understand the presence of the worker HM.

  If the coordinate matching means 10 determines that the height of the top surface of the suspended load OBJ vertically below the hook 69 is higher than the height of the surrounding floor surface FF based on the output of the unevenness detecting means 13, FIG. ) On the region SR1 in the horizontal plane R1 of the space model MD (the region on the left side of the horizontal plane R1 including the intersection of the straight line passing through the wire rope 68 and the floor surface FF, and extending along the X axis) And the coordinates on the input image plane of the camera 20 and the coordinates on the area SR2 (the area on the right side of the horizontal plane R1) in the horizontal plane R1 of the spatial model MD and the coordinates on the input image plane of the camera 22 Are stored in the input image / space model correspondence map 40.

  The output image generation means 11 determines various parameters such as the optical center of the virtual camera, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, and projection method for generating a virtual looking down image. (For example, the parameters of the virtual camera are set to the same values as the parameters of the camera 21.) The coordinates on the output image plane and the coordinates on the space model MD are associated with each other, and the corresponding relationship is represented by the space model / output image. While storing in the correspondence map 41 and referring to the input image / spatial model correspondence map 40, values of each pixel in the output image (for example, a luminance value, a hue value, a saturation value, etc.) and two inputs An output image as shown in FIG. 15C is generated in association with the value of each pixel in the image.

  Alternatively, when the coordinate association unit 10 determines that the height of the top surface of the suspended load OBJ that is vertically below the hook 69 is higher than the height of the surrounding floor surface FF based on the output of the unevenness detection unit 13, FIG. As shown in (D), the coordinates on the area SR3 (the area on the left side of the horizontal plane R1) in the horizontal plane R1 of the spatial model MD are associated with the coordinates on the input image plane of the camera 20, and the horizontal plane R1 of the spatial model MD The coordinates on the area SR4 (the center area of the horizontal plane R1) in FIG. 5 are associated with the coordinates on the input image plane of the camera 21, and on the area SR5 (the area on the right side of the horizontal plane R1) in the horizontal plane R1 of the spatial model MD. The coordinates and coordinates on the input image plane of the camera 22 may be associated with each other, and the corresponding relationship may be stored in the input image / space model correspondence map 40. In this case, the output image generation means 11 generates an output image as shown in FIG.

  FIG. 16 is a diagram for explaining another process of generating a virtual looking-down image by viewpoint conversion, and FIG. 16 (A) shows a front view of the crane 60 similar to FIG. 2 (B), FIG. 16B shows captured images of the cameras 20 to 22, and FIG. 16C shows the relationship between the captured images of the camera 20 and the camera 22 and the space model MD and the captured images. FIG. 16D illustrates an output image obtained as a result of the combination and viewpoint conversion, and FIG. 16D illustrates the relationship between the captured images of the camera 20, the camera 21, and the camera 22 and the space model MD, and combinations of the captured images. The output image obtained as a result of viewpoint conversion is shown.

  FIG. 16A shows a state in which a groove GR is formed vertically below the hook 69 and the worker HM is present beside the groove GR. In addition, the dashed-dotted line of a figure, a broken line, and a dashed-two dotted line show each imaging range of the cameras 20-22, respectively.

  In this case, as shown in FIG. 16B, neither the captured image IM1 of the camera 20 nor the captured image IM2 of the camera 21 can easily grasp the presence of the worker HM, and the image of the camera 22 is captured. Only the image IM3 can easily understand the presence of the worker HM.

  If the coordinate matching means 10 determines that the height of the bottom surface of the groove GR vertically below the hook 69 is lower than the height of the surrounding floor surface FF based on the output of the unevenness detection means 13, in FIG. As shown, the coordinates on the area SR1 (the area on the left side of the horizontal plane R1) on the horizontal plane R1 of the spatial model MD are associated with the coordinates on the input image plane of the camera 22, and the area on the horizontal plane R1 of the spatial model MD The coordinates on SR2 (the area on the right side of the horizontal plane R1) and the coordinates on the input image plane of the camera 20 are associated with each other, and the corresponding relationship is stored in the input image / space model correspondence map 40. Note that this arrangement is opposite to the right and left as compared with the arrangement shown in FIG.

  The output image generation means 11 determines various parameters such as the optical center of the virtual camera, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, and projection method for generating a virtual looking down image. (For example, the parameters of the virtual camera are set to the same values as the parameters of the camera 21.) The coordinates on the output image plane and the coordinates on the space model MD are associated with each other, and the corresponding relationship is represented by the space model / output image. While storing in the correspondence map 41 and referring to the input image / spatial model correspondence map 40, values of each pixel in the output image (for example, a luminance value, a hue value, a saturation value, etc.) and two inputs An output image as shown in FIG. 16C is generated in association with the value of each pixel in the image.

  Alternatively, if the coordinate matching means 10 determines that the height of the bottom surface of the groove GR vertically below the hook 69 is lower than the height of the surrounding floor surface FF based on the output of the unevenness detection means 13, FIG. ), The coordinates on the area SR3 (the area on the left side of the horizontal plane R1) in the horizontal plane R1 of the spatial model MD are associated with the coordinates on the input image plane of the camera 20, and the area in the horizontal plane R1 of the spatial model MD The coordinates on SR4 (the center area of the horizontal plane R1) are associated with the coordinates on the input image plane of the camera 21, and the coordinates on the area SR5 (the area on the right side of the horizontal plane R1) in the horizontal plane R1 of the spatial model MD The coordinates on the input image plane of the camera 22 may be associated with each other, and the corresponding relationship may be stored in the input image / space model correspondence map 40. In this case, the output image generation means 11 generates an output image as shown in FIG.

  Note that the coordinate association unit 10 associates the coordinates on the space model MD with the coordinates on the input image planes of the plurality of cameras based on a predetermined correspondence relationship, regardless of the output of the unevenness detection unit 13. Or by switching the correspondence between the coordinates on the space model MD and the coordinates on the input image planes of the plurality of cameras according to the input from the operator via the input unit 3 May be stored in the input image / space model correspondence map 40. In this case, the unevenness detecting means 13 is omitted.

  FIG. 17 is a diagram for describing processing for generating a virtual looking-down image and a virtual horizontal image based on the captured images of the cameras 20 to 22 in FIG. 15.

  If the coordinate matching means 10 determines that the height of the top surface of the suspended load OBJ vertically below the hook 69 is higher than the height of the surrounding floor surface FF based on the output of the unevenness detecting means 13, the coordinate model means 10 The coordinates on the area SR3 in the horizontal plane R1 (the area on the left side of the horizontal plane R1) and the coordinates on the input image plane of the camera 20 are associated with each other on the area SR4 in the horizontal plane R1 of the spatial model MD (the central area of the horizontal plane R1). The coordinates are associated with the coordinates on the input image plane of the camera 21, and the coordinates on the area SR5 (the area on the right side of the horizontal plane R1) on the horizontal plane R1 of the spatial model MD are associated with the coordinates on the input image plane of the camera 22. The coordinates on the area SR6 (the area on the left side of the vertical plane R1) in the vertical plane R2 of the space model MD are associated with the coordinates on the input image plane of the camera 20, and the space model M The coordinates on the area SR7 (the center area of the vertical plane R1) on the vertical plane R2 and the coordinates on the input image plane of the camera 21 are associated with each other, and the area SR8 (vertical plane R1) on the vertical plane R2 of the space model MD And the coordinates on the input image plane of the camera 22 are stored in the input image / space model correspondence map 40.

  Further, the output image generation unit 11 determines a parameter of the first virtual camera for generating a virtual look-down image (for example, the parameter of the first virtual camera is set to the same value as the parameter of the camera 21). And determining the parameters of the second virtual camera for generating the virtual horizontal image, and each of the two output images (the virtual looking-down image and the virtual horizontal image). The coordinates on the output image plane corresponding to the coordinate on the space model MD are associated with each other, the corresponding relationship is stored in the space model / output image correspondence map 41, and the input image / space model correspondence map 40 is referred to. However, the values of the pixels in the two output images (for example, the brightness value, the hue value, the saturation value, etc.) and the values of the pixels in the three input images are associated with each other in FIG. It generates two output image as shown.

  FIG. 18 is a diagram for explaining processing for generating a virtual looking-down image and a virtual horizontal image based on the captured images of the cameras 20 to 22 in FIG. The point where the height of the bottom surface of a certain groove GR is lower than the height of the surrounding floor surface FF, the point where the coordinates on the region SR3 and the region SR6 are associated with the coordinate on the input image plane of the camera 22, and the region SR5 and the region SR8 Although different from the case of FIG. 17 in that the upper coordinates and the coordinates on the input image plane of the camera 20 are associated, the other points are common. Note that the hatched area in the virtual horizontal image of FIG. 18 is a computer graphic representing a cross section of the floor surface FF.

  Note that the image generation system 300 generates a virtual looking-down image using the camera 20 and the camera 22 attached with the hoisting device 67 sandwiched in the Y-axis direction (transverse direction). Alternatively or additionally, a virtual look-down image may be generated using two cameras attached with the hoisting device 67 sandwiched in the X-axis direction (traveling direction).

  Further, the image generation system 300 may generate a virtual look-down image using any two or three of the four cameras, and may use five or more cameras to generate a virtual view. A look-down image may be generated. In addition, these cameras do not necessarily have to be arranged on a straight line passing through the hoisting device 67, and may be arranged in any positional relationship as long as they are arranged so as to compensate for each other's blind spot.

  With the above configuration, the image generation system 300 can present a virtual looking-down image that eliminates blind spots and difficulty in viewing in the area where the work using the hook 69 is performed, and the operator raises and lowers the hook 69. Can support the operation.

  Next, a fourth embodiment of the present invention will be described with reference to FIGS.

  FIG. 19 is a block diagram schematically illustrating a configuration example of an image generation system 400 according to the fourth embodiment of the present invention. FIG. 20 illustrates three overhead cranes 60A to 60A on which the image generation system 400 is mounted. It is a figure which shows 60C. 20A shows a top view of the three overhead cranes 60A to 60C traveling on the pair of runways 61L and 61R, and FIG. 20B shows the XVIIB-XVIIB direction of FIG. 20A. The front view of overhead crane 60A seen from is shown.

  In addition to the camera 2 (one of the cameras 2A to 2C in FIG. 20A), the image generation system 400 mounted on each of the overhead cranes 60A to 60C includes two cameras 23 (FIG. A) and one of the cameras 23A to 23C in FIG. 20A and 24 (one of the cameras 24A to 24C in FIG. 20A), and the controller 1 draws a scale. Although it differs from the image generation system 100 of FIG. 1 in that it has the means 14, it is common in other points. Therefore, the difference will be described in detail while omitting the description of the common points. In the present embodiment, the image generation system 400 will be described as a representative of those mounted on the overhead crane 60B, but the same description shall be applied to those mounted on the overhead cranes 60A and 60C. .

  The camera 23B images the mark 25A (see FIG. 20B) drawn on the surface of the girder 63A (the surface facing the overhead crane 60B) in the overhead crane 60A adjacent to the rear of the overhead crane 60B. It is fixedly attached to the surface of the girder 63B (the surface facing the overhead crane 60A) of the overhead crane 60B, and preferably its optical axis is orthogonal to the surface of the girder 63A of the overhead crane 60A. To be installed.

  Similarly, the camera 24B images the mark 25C (not shown) drawn on the surface of the girder 63C (the surface facing the overhead crane 60B) in the overhead crane 60C adjacent to the front of the overhead crane 60B. It is fixedly attached to the surface of the girder 63B in the overhead crane 60B (the surface facing the overhead crane 60C), and preferably its optical axis is orthogonal to the surface of the girder 63C in the overhead crane 60C. To be installed.

  The mark 25A has a circular shape, and the optical axis of the camera 23B corresponds to the center point thereof, and the mark 25C similarly has a circular shape, and the optical axis of the camera 24B corresponds to the center point thereof.

  The scale drawing unit 14 is a unit for presenting the distance between the overhead cranes in the X-axis direction (traveling direction) in an easy-to-understand manner using the captured images of the camera 23 and the camera 24. Specifically, the scale drawing unit 14 A scale is superimposed on each captured image of the camera 23 and the camera 24 so that the distance between the overhead cranes is presented in an easy-to-understand manner.

  FIG. 21 is a diagram for explaining the scale drawing process by the scale drawing unit 14 and shows three images captured by the camera 23B and an output image on which the scale is superimposed and displayed by the scale drawing unit 14.

  FIG. 21 shows that in the three images captured by the camera 23B, the distance between the cranes between the overhead crane 60A and the overhead crane 60B gradually decreases in the order of the right image, the center image, and the left image, and the mark 25A. Indicates that the image of the mark 25A gradually approaches the camera 23B, the optical axis of the camera 23B always hits the center of the mark 25A, and the radius of the outer circumference circle centering on the image center point CC in the image of the mark 25A Indicates that gradually increases.

The scale drawing means 14 is an image of an outer circumference circle having an inter-crane distance L meter (variable value) and a radius α ₀ meter (fixed value) between the overhead crane 60A and the overhead crane 60B. With reference to a look-up table in which the correspondence relationship with the number of pixels (variable value) corresponding to the radius of the image of the outer circle centered on the image center point CC is stored in advance, for example, the distance between the cranes is 7 meters. Multiple combinations of inter-crane distance and outer circle radius, such as corresponding outer circle radius of 40 pixels, outer circle radius of 80 pixels corresponding to a crane distance of 3 meters, and outer circle radius of 120 pixels corresponding to a distance between cranes of 1 meter , And a scale is superimposed and displayed on the output image based on the read information.

When determining a plurality of combinations of the distance between the cranes and the outer circumference circle radius, if the actual outer circumference radius is α ₀ meters and the distance between the cranes is L meters, the optical center of the camera 23B and one point on the outer circumference circle are determined. The angle θ between the connecting line and the optical axis is

で表され、画像上の外周円半径をＸ_OBJピクセル、カメラ２３Ｂの画角をθ_cam、カメラ２３Ｂの画角θ_camに応じて決定される画像中心点ＣＣから撮像画像の端までの画素数をＸピクセルとすると、画像上の外周円半径Ｘ_OBJは、

The outer peripheral circle radius on the image is X _OBJ pixels, the angle of view of the camera 23B is θ _cam , and the number of pixels from the image center point CC determined according to the angle of view θ _cam of the camera 23B to the end of the captured image Is X pixel, the outer circumference radius X _OBJ on the image is

で表される。

It is represented by

As described above, since the outer peripheral circle radius α ₀ (meter), the angle of view θ _cam , and the number of pixels X (pixels) from the image center point CC to the end of the captured image are known, the inter-crane distance L (meter). Is determined, the outer peripheral circle radius X _OBJ (pixel) on the image is uniquely determined.

The image generation system 400, previously preparing a plurality of lookup tables corresponding to a plurality of angle theta _cam, when the angle theta _cam camera 23B having a zoom function is changed, the changed By selecting and referring to a look-up table corresponding to the angle of view θ _cam , the scale width displayed in a superimposed manner may be dynamically adjusted.

  The output image of FIG. 21 is an output image in which scales are superimposed and displayed, and scales including three circles indicated by broken lines and the distance between the cranes (meters) corresponding to each circle are superimposed and displayed. The radii of these three circles are 40 pixels, 80 pixels, and 120 pixels, respectively, from the inside, and represent distances between cranes of 7 meters, 3 meters, and 1 meter. Moreover, the output image of FIG. 21 represents that the distance between cranes at the present time is about 5 meters.

  FIG. 22 is a diagram for explaining another scale drawing process by the scale drawing unit 14. Similarly to FIG. 21, three images captured by the camera 23 </ b> B and an output in which the scale is superimposed and displayed by the scale drawing unit 14 are shown. And an image.

  FIG. 22 shows that a rectangular mark 25A1 is drawn on the surface of the girder 63A instead of the circular mark 25A, the optical axis of the camera 23B does not coincide with the center of the mark 25A1, and three images captured by the camera 23B. In this order, the distance between the cranes between the overhead crane 60A and the overhead crane 60B gradually decreases in the order of the right image, the center image, and the left image, the mark 25A1 approaches the camera 23B, and the image of the mark 25A1 is displayed. This is different from the case of FIG. 21 in that the distance between the right end of the image of the mark 25A1 and the image center point CC is gradually increased as it gradually increases.

The scale drawing means 14 has a distance L between the crane 60 C between the overhead crane 60 </ b> A and the overhead crane 60 </ b> B (variable value) and a distance α ₁ meter (fixed value) from a point on the surface of the girder 63 </ b> A where the optical axis of the camera 23 </ b> B hits. And the number of pixels (variable value) corresponding to the distance between the image center point CC and the image of the mark 25A1 in the image captured by the camera 23B. Is stored in advance, for example, a separation distance of 60 pixels corresponding to a distance between cranes of 7 meters, a separation distance of 120 pixels corresponding to a distance between cranes of 3 meters, and a separation corresponding to a distance between cranes of 1 meter. Read out multiple combinations of distances between cranes and separation distances, such as a distance of 180 pixels. Superimposed tick marks on an output image based on.

  A method for determining a plurality of combinations of the distance between the cranes and the separation distance is the same as the above-described method for determining a plurality of combinations of the distance between the cranes and the outer circumference circle radius.

  The output image in FIG. 22 is an output image in which scales are superimposed and scales including three vertical lines indicated by broken lines and the distance between the cranes (meters) corresponding to each vertical line are superimposed and displayed. The separation distances of the three vertical lines are 60 pixels, 120 pixels, and 180 pixels in order from the right, respectively, and represent distances between cranes of 7 meters, 3 meters, and 1 meter. Moreover, the output image of FIG. 22 represents that the distance between cranes at the present time is about 5 meters.

  FIG. 23 is a diagram for explaining yet another scale drawing process by the scale drawing unit 14. Like FIG. 21, three images captured by the camera 23 </ b> B and the scale drawing unit 14 are displayed in a superimposed manner. An output image is shown.

  FIG. 23 shows that the mark is not present on the surface of the girder 63A, and the three images captured by the camera 23B are the right side image, the center image, and the left side image in the order of the overhead crane 60A and the overhead crane 60B. This is different from the case of FIG. 21 in that the distance between the upper end of the image of the girder 63A and the image center point CC is gradually increased as the distance between the cranes gradually decreases.

The scale drawing means 14 has an inter-crane distance L meter (variable value) between the overhead crane 60A and the overhead crane 60B and a distance α ₂ meters (fixed value) from a point on the surface of the girder 63A where the optical axis of the camera 23B hits. The number of pixels (variable value) corresponding to the size of the separation distance between the image center point CC in the image captured by the camera 23B and the upper end of the image of the girder 63A. For example, a separation distance of 30 pixels corresponding to a distance between cranes of 7 meters, a separation distance of 70 pixels corresponding to a distance of 3 meters between cranes, and a distance between cranes of 1 Read out multiple combinations of distances between cranes and separation distances such as 110 pixel separations corresponding to meters, Superimposed tick marks on an output image based on the look out information.

  A method for determining a plurality of combinations of the distance between the cranes and the separation distance is the same as the above-described method for determining a plurality of combinations of the distance between the cranes and the outer circumference circle radius.

  The output image of FIG. 23 is an output image in which scales are displayed in a superimposed manner, and scales including three horizontal lines indicated by broken lines and the distance between the cranes (meters) corresponding to each horizontal line are displayed in a superimposed manner. The separation distances of these three horizontal lines are 30 pixels, 70 pixels, and 110 pixels, respectively, from the bottom, and represent distances between cranes of 7 meters, 3 meters, and 1 meter. Moreover, the output image of FIG. 23 represents that the distance between cranes at the present time is about 5 meters.

  24 and 25 are diagrams each showing an example of an output image including a plurality of images having different characteristics.

  FIG. 24 is an image that presents the distance to the rear next overhead crane on the left side, an image that presents the distance to the front next overhead crane on the right side, a virtual horizontal image on the upper side, and a virtual looking down image on the lower side. The output image which arrange | positions is shown.

  FIG. 25 shows a virtual looking-down image at the center, an image that presents the distance to the front overhead crane on the upper side, an image that presents the distance to the rear adjacent crane on the lower side, and a virtual horizontal image on the right side The output image which arrange | positions is shown.

  With the above configuration, the image generation system 400 intuitively grasps the distance to the adjacent overhead crane by simply displaying the scales superimposed on the captured image of the camera 23B without performing complicated image processing. The output image which can be made can be produced | generated, and the contact of adjacent overhead cranes can be prevented by showing the output image.

  Note that the image generation system 400 outputs or displays an alarm or a warning message when the distance to the adjacent overhead crane is less than a predetermined distance by a known means. Also good.

  The camera 23B is preferably installed so that its optical axis is orthogonal to the surface of the girder 63A in the overhead crane 60A, but the optical axis is inclined at a predetermined angle with respect to the surface. It may be installed. Even in this case, the scale drawing unit 14 takes into account the predetermined angle (for example, using a predetermined projection function), and the distance between the cranes between the overhead crane 60A and the overhead crane 60B is derived. A lookup table that stores the correspondence between the L meter (variable value) and the number of pixels from the image center point CC to a predetermined mark (variable value), and a plurality of the distance between the cranes and the number of pixels And a scale is superimposed and displayed on the output image based on the read information. The same applies to the case where a mark drawn on the surface inclined with respect to the optical axis of the camera 23B arranged so that the optical axis is orthogonal to the surface of the girder 63A. Further, the image generation system 400 can take a photographed image of the camera 23B so that the predetermined mark drawn on the surface inclined with respect to the optical axis of the camera 23B can be handled as if it were drawn on the surface orthogonal to the optical axis. You may make it perform image conversion to.

  The scale drawing process described above has been described with reference to an image captured by the camera 23B, but the same description applies to the camera 24B.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the image generation system 100 does not use the spatial model MD (via the spatial model MD while using the coordinates of the optical center C of the camera 2 and the coordinates of the optical center CV of the virtual camera 2V). Without indirectly associating the pixel group on the input image plane R4 and the pixel group on the output image plane R5), regardless of the coordinates of the optical center CV of the camera 2 and the coordinates of the optical center CV of the virtual camera 2V, The output image may be generated based on a predetermined direct correspondence between the pixel group on the input image plane R4 and the pixel group on the output image plane R5.

  In this case, the image generation system 100 converts a part of the pixel group on the input image plane R4 to a part of the pixel group on the output image plane R5 based on the predetermined direct correspondence (the spatial model MD is To generate a virtual horizontal image of the hook 69 by associating directly with another part of the pixel group on the input image plane R4 to another part of the pixel group on the output image plane R5 ( A virtual look-down image of the floor FF is generated by associating directly (without going through the space model MD), and the final output image is synthesized by synthesizing the virtual horizontal image and the virtual look-down image. You may make it produce | generate.

  In addition, as shown in FIG. 2, the camera 2 is attached to the tip of a bracket BR that extends vertically downward from the trolley 66 and then extends in the X-axis direction. Although the camera 2 is disposed on the vertical plane VS including the center line of the 60 girders 63, the camera 2 may be attached to the tip of a bracket extending in the Y-axis direction from the trolley 66. May be arranged on a vertical plane perpendicular to the center line of the girder 63 of the overhead crane 60. Note that the vertical surface R2 does not necessarily have a relationship parallel or orthogonal to the center line of the girder 63, and may be a surface that forms an arbitrary angle with the center line. Furthermore, the camera 2 may additionally be attached to the tip of one or a plurality of other brackets extending in a direction different from the bracket BR. This is because even if the hook 69 is hidden behind the obstacle and one camera cannot take an image of the hook 69, another camera can take an image of the hook 69 instead.

  The image generation system 100 also includes an input image captured by the camera 2 attached to the tip of the bracket BR extending from the trolley 66 in the X-axis direction, and a camera attached to the tip of the bracket extending from the trolley 66 in the Y-axis direction. An output image may be generated while selectively using a plurality of input images including the input image captured by the user. A plurality of output images are generated using each of the plurality of input images, and the plurality of output images are generated. These may be displayed side by side.

  Further, the camera 2 allows the operator to dynamically adjust the optical axis direction so that the operator can take an image of the hook 60 traversing with the trolley 66 and moving up and down in the vertical direction from the upper side of the vertical direction. It may be attached at a predetermined position. In this case, a plurality of cameras 2 are installed on the girder 63 at predetermined intervals along the transverse direction (Y-axis direction) of the trolley 66, and the distance in the transverse direction (Y-axis direction) with the hook 69 at the present time. A captured image of a camera that minimizes the image may be adopted as the input image, or an image obtained by combining a plurality of captured images from the plurality of cameras may be employed as the input image.

  In the above-described embodiment, the image generation system 100 is used to support the operation of the hook in the overhead crane 60. However, the image generation system 100 may be used to support the operation of the grab bucket of the unloader. , Cable cranes, or Telha hooks, or stacker crane forks, may be used to assist in operation, and jib crane hooks equipped with hoisting and turning devices may be used to assist in operation .

  The hook of the overhead crane 60 may be a lifting magnet, a vacuum lifter, a claw, or a spreader.

  In addition, the cameras 2, 20, 21, 22, 23A to C, and 24A to 24C that are individually described in each of the above-described embodiments, the suspension position detection unit 12, the unevenness detection unit 13, and the scale drawing unit 14 Can be introduced in any combination in any of the above-described embodiments.

  DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 3 ... Input part 4 ... Memory | storage part 5 ... Display part 10 ... Coordinate matching means 11 ... Output image generation means 12 ... Suspension tool Position detecting means 13 ... Concavity and convexity detecting means 14 ... Scale drawing means 20, 21, 22, 23A to C, 24A to C ... Camera 40 ... Input image / space model correspondence map 41 ... Space Model / output image correspondence map 60, 60A to 60C ... overhead crane 61 ... runway 62 ... wheels 63 ... girder 64 ... runway 65 ... wheels 66 ... trolley 67 ... -Hoisting device 68 ... Wire rope 69 ... Hook 70 ... Operation room 100, 200, 300, 400 ... Image generation system

Claims (5)

An image generation system that supports the operation of a crane that operates while looking down from the upper side of a vertical suspension ,
An imaging device for imaging the sling which hoisting device hoisting or lowering wind vertically from obliquely above,
An output image generation unit that performs viewpoint conversion processing on an image captured by the imaging device and generates an output image of the suspension viewed from a horizontal direction;
An image generation system comprising: An image generation system that supports crane operation,
An image pickup device for picking up a lifting device that the hoisting device winds up or down from vertically above;
An output image generation unit that performs viewpoint conversion processing on an image captured by the imaging device and generates an output image of the suspension viewed from a horizontal direction;
The crane is an overhead crane having a girder that travels along the wall of the building and a trolley that traverses the girder,
The imaging device and the hoisting device are attached to the trolley and traverse along with the trolley .
Images generation system. An image generation system that supports crane operation,
An image pickup device for picking up a lifting device that the hoisting device winds up or down from vertically above;
An output image generation unit that performs viewpoint conversion processing on an image captured by the imaging device and generates an output image of the suspension viewed from a horizontal direction;
The crane is an overhead crane having a girder that travels along the wall of the building and a trolley that traverses the girder,
The hoisting device is attached to the trolley and traverses with the trolley,
A plurality of the imaging devices are installed on the girder along the transverse direction of the trolley .
Images generation system. Further comprising a suspension position detecting means for detecting a vertical position of the suspension,
The output image generation means determines the vertical position of the optical center of the virtual camera employed in the viewpoint conversion process based on the vertical position of the suspension.
The image generation system according to claim 1, wherein the image generation system is an image generation system. The output image generation means performs a viewpoint conversion process on an image captured by the imaging device, and generates an output image when the suspension is viewed from a horizontal direction and an output image when the floor is viewed from above.
Image generation system according to any one of claims 1 to 3, characterized in that.

Images