JP6266836B1 - Determination apparatus, determination system, program, and recording medium for supporting crane operation - Google Patents

Abstract

It aims at realizing the means for reducing the burden of the operator of a crane, without using many sensors. In order to achieve this object, the present invention proposes a determination system (1) as an embodiment. The determination system (1) recognizes the image picked up by the image pickup device (11) and the image pickup device (11) which are arranged near the tip of the jib (922) of the crane (92) and continuously take an image downward. The movement state of objects such as workers and heavy machinery in the grab (924) and cargo hold (91) in the image is specified, and in the near future based on the specified movement state, the grab (924) and the object are on a plane. A determination device (12) for determining whether or not there is a possibility of contact or proximity. When it is determined that there is a possibility of contact or proximity between the grab (924) and the object, the determination device (12) notifies the operator of the crane (92).

Description

  The present invention relates to a determination apparatus for supporting crane operation.

  When operating a crane capable of turning and raising and lowering the jib, the operator must be careful not to pass the suspended load directly above the object such as an operator or heavy machinery in case the suspended load falls. Care must be taken.

  Techniques for reducing the burden on the crane operator described above have been proposed. For example, in Patent Document 1, an operator's helmet is recognized from an image of a crane peripheral area captured from above to below a suspended load by a camera, and the recognized helmet image is displayed on the crane peripheral area image. A safety confirmation device configured as described above has been proposed.

  The safety confirmation device described in Patent Literature 1 sets a dangerous area centered on a suspended load position calculated based on the length, elevation angle, and direction of a jib (boom), and is a worker's helmet recognized in the dangerous area. Is displayed in a different color from other helmets, etc., to notify the crane operator that there is an operator in the danger area. The safety confirmation device described in Patent Literature 1 changes the size of the dangerous area according to the suspended load amount identified from the wire tension and the suspended load height identified from the wire feed length or the like. Moreover, the safety confirmation device described in Patent Document 1 corrects the position of the dangerous area based on the wind direction and the wind speed measured by the sensor. Furthermore, the safety confirmation device described in Patent Literature 1 expands the danger area in the jib movement direction in accordance with the turning of the jib.

JP 2010-241548 A

  According to the safety confirmation device described in Patent Document 1, the operator of the crane can easily know the worker in the danger area, and can perform an operation for avoiding danger without paying close attention. it can. However, the safety confirmation device described in Patent Document 1 uses many types of information such as jib length, elevation angle, bearing, wire tension, wire feed length, wind direction, wind speed, and jib turning information. Is done. Therefore, in order to realize the safety confirmation device described in Patent Document 1, it is necessary to introduce many sensors.

  An object of this invention is to implement | achieve the means for reducing the burden of the operator of a crane, without using many sensors.

In order to solve the above-described problem, the present invention has a predetermined appearance feature in an image captured by an imaging device that is installed on a crane and performs imaging from above to below the suspended load of the crane. Image recognition means for recognizing an object and the suspended load; time of image capture for each of images captured at a plurality of different times by the imaging device; and the object recognized by the image recognition means Based on the position of the object and the suspended load in the image, the position of the object and the suspended load at each of a plurality of times within a period from the current time until the predetermined time elapses, based on the estimated position, whether the suspended load and the object approaches from a predetermined distance in the period until the elapsed time determined in advance from the current time Suggest determination device and a determination means for the first embodiment.

According to the present invention, in the first aspect described above, the determination unit recognizes the image capturing time and the image recognizing unit regarding each of images captured at a plurality of different times by the imaging device. Based on the position of the suspended load in the image, an approximate curve of the past movement path of the suspended load is specified, and based on the specified approximate curve, until the predetermined time elapses from the present time The configuration of estimating the moving route of the suspended load during the period is proposed as a second mode.

In the first or second aspect of the present invention, the crane is disposed on a ship, and the determination unit is configured to move the suspended load recognized by the image recognition unit. A configuration in which a vibration component included in a locus is specified and the determination is performed using the specified vibration component is proposed as a third aspect.

Further, according to the present invention, in any one of the first to third aspects, the determination unit is predetermined according to a moving speed in the image of the suspended load recognized by the image recognition unit. A configuration in which the distance is changed is proposed as a fourth aspect.

Further, according to the present invention, in any one of the first to fourth aspects, the determination unit includes the time of capturing the image regarding each of images captured at a plurality of different times by the imaging device, Based on the position of the feature point of the suspended load recognized by the image recognition means, the state of rotation of the suspended load with the rope hanging the suspended load as an axis is identified, and the identified state of rotation is A configuration in which the above determination is performed is proposed as a fifth aspect.

In addition, the present invention is an object having a predetermined appearance feature in an image captured by an imaging device that is installed on a crane disposed on a ship and performs imaging from above to below the suspended load of the crane. An image recognition means for recognizing an object, the suspended load, and a plurality of parts of the structure of the ship; and the image recognition means for recognizing one of images taken at a plurality of different times by the imaging device. The plurality of portions recognized by the image recognition means in other images different from the one image among the images taken at different times by the imaging device and the positions of the plurality of portions Based on the position of the object recognized by the image recognition means in the one image and the image recognition means in the other image by the image recognition means. And the position of the suspended object recognized by the image recognizing means in the other image and the position of the suspended object recognized by the image recognizing means in the one image. For each of images taken at a plurality of different times by the imaging device, and the images of the object and the suspended load recognized by the image recognition means A determination device comprising: determination means for determining whether or not the object and the suspended load are closer than a predetermined distance in a period until a predetermined time elapses from a current time based on a position in Proposed as the seventh aspect.

According to the present invention, in any one of the first to sixth aspects, the display instructs the display device to display an image obtained by adding a graphic representing the determination result to the image captured by the imaging device. The configuration of instruction means is proposed as a seventh aspect.

Further, the present invention provides an image pickup device that is installed in a crane and picks up an image from above the crane's suspended load, and image data that is connected to the image pickup device and represents an image picked up by the image pickup device. A determination system including the determination device according to any one of the first to seventh aspects described above that is received from the imaging device is proposed as an eighth aspect.

In addition, the present invention provides a computer having an object having a predetermined appearance characteristic in an image captured by an imaging device that is installed on a crane and performs imaging from above to below the suspended load of the crane. The process of recognizing a suspended load, the time of capturing the image for each of images captured at a plurality of different times by the imaging device, the object recognized in the recognition process, and the image of the suspended load A process for estimating the position of the object and the suspended load at each of a plurality of times within a period from the present time until the predetermined time elapses based on the position within the position, and based on the estimated position determines whether the suspended load the said object approaches from a predetermined distance in the period until the elapsed time determined in advance from the current time We propose a program for executing a physical as the ninth aspect.

In addition, the present invention provides a computer having an object having a predetermined appearance characteristic in an image captured by an imaging device that is installed on a crane and performs imaging from above to below the suspended load of the crane. The process of recognizing a suspended load, the time of capturing the image for each of images captured at a plurality of different times by the imaging device, the object recognized in the recognition process, and the image of the suspended load A process for estimating the position of the object and the suspended load at each of a plurality of times within a period from the present time until the predetermined time elapses based on the position within the position, and based on the estimated position determines whether the suspended load the said object approaches from a predetermined distance in the period until the elapsed time determined in advance from the current time Suggest recording medium that can persistently store read the computer program for executing a physical as tenth aspect.

According to the determination apparatus according to the first aspect of the present invention or the determination system according to the eighth aspect, when the current operation is continued using an image captured from the upper side to the lower side of the suspended load. It is determined whether the suspended load and the object are in contact with or close to each other. In addition, since it is determined whether the suspended load and the object are in contact with or close to each other based on the positions of the suspended load and the object at each of a plurality of times, for example, the object and the suspended load pass through the same position with a time difference. Even in such a case, an appropriate determination is made. As a result, the crane operator can easily ensure the safety of the jib movement.

According to the determination apparatus according to the second aspect of the present invention, even if the suspended load does not move linearly due to turning of the jib or the like, it is appropriate whether or not the suspended load and the object are in contact with or close to each other. Is determined.

According to the determination apparatus according to the third aspect of the present invention, in the determination as to whether the suspended load and the object are in contact with or in proximity to each other, the sway of the ship on which the crane is installed is considered. As a result, the operator of the crane can ensure safety in moving the jib without necessarily paying close attention to the state of the ship's shaking.

According to the determination apparatus according to the fourth aspect of the present invention, the moving speed of the suspended load is taken into consideration in determining whether the suspended load and the object are in contact with or in proximity to each other. As a result, the crane operator can operate the jib movement at high speed while ensuring safety.

According to the determination apparatus according to the fifth aspect of the present invention, the state of rotation of the suspended load is taken into account in determining whether the suspended load and the object are in contact with or close to each other. As a result, the operator of the crane can ensure safety in moving the jib without necessarily paying close attention to the state of rotation of the suspended load.

According to the determination apparatus according to the sixth aspect of the present invention, since the elements accompanying the change in the imaging position and the imaging direction included in the change in the position of the suspended load or the object in the image are removed, for example, the jib is at high speed Even if it moves and the imaging position changes greatly, it is properly determined whether the suspended load and the object are in contact with or close to each other.

According to the determination device according to the seventh aspect of the present invention, the operator of the crane can know whether there is a risk that the suspended load and the object are in contact with or close to each other by viewing the image displayed on the display device. it can.

According to the program according to the eighth aspect of the present invention or the recording medium according to the ninth aspect, the determination apparatus according to the first aspect of the present invention is realized by a computer.

The figure which showed typically the ship by which the determination system concerning one Embodiment and the said determination system are mounted. The figure which showed the basic composition of the computer used as hardware of the determination apparatus concerning one Embodiment. The figure which showed the function structure of the determination apparatus concerning one Embodiment. The figure which illustrated the data structure of the image table memorize | stored in the memory | storage means concerning one Embodiment. The figure which illustrated the data structure of the hanging load position table memorize | stored in the memory | storage means concerning one Embodiment, and an object position table. The figure which illustrated the data structure of the structure position table memorize | stored in the memory | storage means concerning one Embodiment. The figure which showed the flow of the process which the determination means concerning one Embodiment performs. The figure which showed the flow of the process which the determination means concerning one Embodiment performs. The figure which illustrated the image which a display device displays according to the directions of the display directions means concerning one embodiment. The figure for demonstrating the process of the determination means in one modification. The figure for demonstrating the process of the determination means in one modification. The figure for demonstrating the process of the determination means in one modification. The figure which illustrated the data structure of the hanging load position table used in one modification. The figure for demonstrating the process of the determination means in one modification. The figure for demonstrating the process of the determination means in one modification.

[Embodiment]
Below, the determination system 1 concerning one Embodiment of this invention is demonstrated. FIG. 1 is a diagram schematically showing a determination system 1 and a ship 9 on which the determination system 1 is mounted. The ship 9 is a bulk carrier and includes a plurality of cargo holds 91 and cranes 92 arranged according to the plurality of cargo holds 91. In FIG. 1, there are two cargo holds 91 and two cranes 92, but these numbers differ depending on the ship 9. The cargo hold 91 includes a main body 911 that constitutes a wall surface and a bottom surface, a hatch combing 912 provided around a hatch that is an opening above the main body 911, and a hatch cover 913 disposed on the hatch combing 912.

  The cargo hold 91 illustrated in FIG. 1 includes two hatch covers 913 arranged on the bow side and the stern side, and each hatch cover 913 slides on the hatch coaming 912 and is folded to cover the hatch. There can be two states: an open state that is not open and a closed state that is unfolded and covers the hatch. The hatch cover 913 of the cargo hold 91 illustrated on the right side of FIG. 1 is in an open state, and the hatch cover 913 of the cargo hold 91 illustrated on the left side is in a closed state.

  The crane 92 includes a crane post 921, a jib 922 extending outward from the crane post 921, a rope 923 hanging downward from the vicinity of the tip of the jib 922, and a grab 924 attached to the tip of the rope 923. In the present embodiment, the suspended load of the crane 92 is a grab 924 and a cargo gripped by the grab 924.

  An operation chamber 925 that is a space where an operator operates the crane 92 is provided on the upper part of the crane post 921. The operator operates an operation lever or the like disposed in the operation chamber 925 while visually checking the position and shape of cargo, workers, heavy machinery, etc. in the cargo hold 91 and the position of the grab 924 from the operation chamber 925. , Various operations such as turning the jib 922, raising and lowering, expanding and contracting, feeding and winding the rope 923, and opening and closing the grab 924 are performed.

  The determination system 1 is arranged according to each of the plurality of cranes 92. The determination system 1 is arranged in the vicinity of the tip of the jib 922 and is arranged in the operation room 925, and is arranged in the operation room 925, and is directed to the operator such as workers and heavy equipment in the cargo hold 91. A determination device 12 having a function of notifying the presence or absence of a danger of contact or proximity to the grab 924 is provided.

  The imaging device 11 is attached to the jib 922 via an attachment device that suspends the imaging device 11 so that the imaging direction (optical axis) is always vertically downward due to, for example, gravity. For example, the imaging device 11 sequentially transmits image data representing an image captured every elapse of a predetermined time to the determination device 12. In order to transmit and receive image data, the imaging device 11 and the determination device 12 are connected by communication. The communication connection between the imaging device 11 and the determination device 12 may be performed via a communication cable such as an optical fiber, or may be performed wirelessly.

  In the present embodiment, the determination device 12 is realized by a general computer performing processing according to a program. However, the determination device 12 may be configured as a dedicated device.

  FIG. 2 is a diagram illustrating a basic configuration of the computer 10 used as hardware of the determination device 12. The computer 10 includes a memory 101 that stores various data, a processor 102 that performs various data processing in accordance with a program stored in the memory 101, a communication IF 103 that is an IF (Interface) that performs data communication with the imaging device 11, a user A display device 104 such as a liquid crystal display for displaying an image, and an operation device 105 such as a touch panel for receiving a user operation.

  Instead of the display device 104 built in the computer 10, an external display device connected to the computer 10 may be used. Further, an external operation device connected to the computer 10 may be used instead of the operation device 105 built in the computer 10.

  FIG. 3 is a diagram illustrating a functional configuration of the determination device 12. When the computer 10 performs processing according to the program for the determination device 12, the determination device 12 having the functional configuration shown in FIG. 3 is realized. The functional configuration provided in the determination device 12 will be described below.

  The determination device 12 includes a storage unit 121 that stores various data, an image data reception unit 122 that receives image data from the imaging device 11, and an image represented by the image data received by the image data reception unit 122 from the imaging device 11. Recognized by the image recognition means 123 and the image recognition means 123 for recognizing the grab 924, the objects such as workers and heavy machinery in the cargo hold 91, and the characteristic portions of the structure constituting the ship 9 by known image recognition processing. When the current movement state of the grab 924 and the object is maintained, the determination means 124 for determining whether or not they are in contact with or close to each other on a plane in the near future, the result of the determination by the determination means 124, etc. Display instruction means 125 is provided for instructing the display device 104 to display an image.

  The storage means 121 stores in advance the suspended load feature data indicating the appearance characteristics of the grab 924, the object feature data indicating the appearance characteristics of the object regarding each of the objects such as workers and heavy machinery, and the ship 9 Structure feature data indicating the feature on the appearance of each of the plurality of portions constituting the portion is stored. These data are data used by the image recognition unit 123 to recognize the grab 924 or the like in the image captured by the imaging device 11.

  The storage unit 121 stores image data received from the imaging device 11 by the image data receiving unit 122. FIG. 4 is a diagram illustrating the data structure of an image table that is a table used by the storage unit 121 to store image data. The image data received by the image data receiving unit 122 from the imaging device 11 is accompanied by time data indicating the time at which the image represented by the image data was captured. The image table has a “time” field for storing time data and an “image” field for storing image data.

  The storage unit 121 stores data indicating the position in the image, such as a suspended load recognized by the image recognition unit 123. FIG. 5 shows a suspended load position table which is a table used by the storage unit 121 to store data indicating the position of the grab 924 (suspended load), and the storage unit 121 is a position of an object such as an operator or a heavy machine. It is the figure which illustrated the data structure of the target object position table which is a table used in order to accumulate | store the data which show etc. The object position table is generated according to each object recognized by the image recognition unit 123.

  The suspended load position table and the object position table store a “time” field that stores time data indicating the imaging time of an image used for recognition, and position data that indicates the position of the recognized suspended load or object. It has a “position” field and a “size” field for storing size data indicating the size of the recognized suspended load or object. The size data stored in the “size” field indicates, for example, the diameter of a circle circumscribing the recognized suspended load or the like in the image used for recognizing the suspended load or the like. The position data stored in the “position” field indicates, for example, the coordinates of the center position of a circle circumscribing the recognized suspended load or the like in the image used for recognizing the suspended load or the like.

  Further, the storage unit 121 stores data indicating the position and the like in the image of the characteristic part of the structure constituting the ship 9 recognized by the image recognition unit 123. FIG. 6 is a diagram illustrating a data structure of a structure position table, which is a table used by the storage unit 121 to store data indicating the position of the feature portion. The structure position table is generated according to each of a plurality of preselected feature portions. Hereinafter, as an example, it is assumed that the corner portion on the bow side and starboard side and the corner portion on the stern side and port side of the four corner portions of the hatch combing 912 of the cargo hold 91 are selected as the characteristic portions.

  The structure position table has a “time” field for storing time data indicating the imaging time of an image used for recognition, and a “position” field for storing position data indicating the position of the recognized feature portion. The position data stored in the “position” field indicates, for example, the coordinates of the recognized feature portion in the image used for recognition of the feature portion.

  Returning to FIG. 3, the description of the functional configuration of the determination device 12 is continued. As described above, the image data receiving unit 122 receives the image data sequentially transmitted from the imaging device 11. The image data received by the image data receiving means 122 is stored in the image table (FIG. 4).

  As described above, the image recognition unit 123 uses the suspended feature data to display the grab 924 in the image represented by the image data received from the imaging device 11 by the image data reception unit 122 by a known image recognition process. Using the data, an object such as an operator or a heavy machine in the cargo hold 91 is recognized, and a feature portion (two corners of the hatch combing 912) is recognized using the structure feature data. Data indicating the position of the grab 924 and the like recognized by the image recognition means 123 is stored in the suspended load position table (FIG. 5), the object position table (FIG. 5), or the structure position table (FIG. 6). .

  As described above, when the grab 924 recognized by the image recognition unit 123 and the current movement state of the object are maintained, the determination unit 124 determines whether or not they will contact or approach on a plane in the near future. judge. Note that “contact on a plane” means that the grab 924 passes just above the object even if it is not in contact in the three-dimensional space.

  FIG. 7A and FIG. 7B (hereinafter collectively referred to as “FIG. 7”) are diagrams showing a flow of processing performed by the determination unit 124 every elapse of a predetermined time. First, the determination unit 124 reads a record having the second most recent time indicated by the time data (hereinafter referred to as a record at time (k-1)) from the suspended load position table (FIG. 5), and sets the time (k-1). The coordinates of the grab 924 indicated by the position data of the record (the coordinates of the suspended load in the coordinate system of the image captured at time (k-1)) are the latest image (the image captured at time (k)). A process of converting to coordinates in the coordinate system is performed (step S101).

  The coordinate conversion in the process of step S101 is performed by converting the coordinates of a plurality of feature portions (two corner portions of hatch coaming 912) in the image captured at time (k-1) in the image captured at time (k). This is done by mapping corresponding to the coordinates of these feature portions. Specifically, the determination unit 124 reads the records of time (k) and time (k−1) from the structure position table (FIG. 6) corresponding to each of the plurality of feature portions, and the positions of the read records. The coordinates indicated by the position data of the record at time (k−1) read from the suspended load position table are converted by a known matrix calculation using the coordinates indicated by the data.

  Subsequently, the determination unit 124 reads the record at time (t) from the suspended load position table (FIG. 5), and starts from the coordinates (coordinates of the grab 924 at time (k-1)) obtained by the conversion in step S101. Then, a vector whose end point is the coordinate indicated by the position data of the record at the read time (t) (the coordinate of the grab 924 at time (k)) is specified (step S102). This vector indicates the moving direction and moving speed of the grab 924 at the present time. The determination unit 124 temporarily stores the vector data indicating the vector specified in step S102 in the storage unit 121.

  Subsequently, the determination unit 124 substitutes an initial value “1” into a counter i used for sequentially selecting the objects recognized by the image recognition unit 123 (step S103). Hereinafter, the number of objects recognized by the image recognition unit 123 is n.

  Subsequently, the determination unit 124 performs the same process as steps S101 and S102 on the i-th object among the objects recognized by the image recognition unit 123 (steps S104 and S105). Thereby, a vector indicating the moving direction and moving speed of the i-th object is specified. The determination unit 124 temporarily stores the vector data indicating the vector specified in step S105 in the storage unit 121.

  Subsequently, the determination unit 124 determines whether i = n (step S106). This determination is a determination as to whether or not the identification of the vector indicating the moving direction and moving speed has been completed for all the objects recognized by the image recognition means 123. When there is still an object for which the vector is not specified (step S106; “No”), the determination unit 124 increments the counter i by 1 (step S107), and returns the process to step S104.

  When the identification of the vectors regarding all the objects is completed (step S106; “Yes”), the determination unit 124 subsequently sets a distance threshold D corresponding to the moving speed of the grab 924 indicated by the vector identified in step S102. (Step S108). The threshold value D means a distance that can be considered that there is no risk that the grab 924 and the object are in contact with each other if the distance between the grab 924 and the object is more than that distance. As the moving speed of the grab 924 increases, there is a possibility that when the grab 924 or the cargo held by the grab 924 falls, the fallen object may reach far from the position of the grab 924 immediately before dropping. Accordingly, the determination unit 124 sets a larger threshold value D as the moving speed of the grab 924 increases. For example, the determination unit 124 calculates the threshold value D as a predetermined function (a monotonically increasing function of the moving speed of the grab 924) having the moving speed of the grab 924 as a variable.

  Subsequently, the determination unit 124 assigns an initial value “0” to a variable t indicating the elapsed time from the current time, and a flag f (1) indicating whether or not an alarm is required for contact or proximity between the grab 924 and the object. The initial value “0” is substituted for .about.f (n) (step S109). The flag f (i) is a flag corresponding to the i-th object.

  Subsequently, the determination unit 124 adds a predetermined time Δt to the variable t (step S110), and determines whether or not the variable t is larger than a predetermined time threshold T (step S111). The threshold value T is assumed that when the current moving state of the grab 924 and the target object (in this embodiment, the moving direction and moving speed) is maintained, they contact each other on the plane after the time has elapsed from the present time. This also means that the contact can surely be avoided if the operator of the crane 92 performs an appropriate operation for changing the moving state of the grab 924 before the passage of the time.

  When the variable t is equal to or smaller than the threshold T (step S111; “No”), the determination unit 124 assumes that the moving direction and the moving speed indicated by the vector specified in step S102 are maintained, and the time t has elapsed from the present time. The position of the grab 924 at that time (the coordinates in the coordinate system of the image captured at time (k)) is estimated (step S112).

  Subsequently, the determination unit 124 substitutes an initial value “1” for the counter i (step S113). Subsequently, regarding the i-th object, the determination unit 124 assumes that the same processing as in step S112, that is, the moving direction and the moving speed indicated by the vector specified in step S105 are maintained, and the time t from the present time is maintained. The position of the target object at the time when the time elapses (the coordinates in the coordinate system of the image captured at time (k)) is estimated (step S114).

  Subsequently, the determination device 12 determines whether the distance d between the position of the grab 924 estimated in step S112 and the position of the first object estimated in step S114 is less than the threshold D set in step S108. Is determined (step S115). When the distance d between the position of the grab 924 and the position of the first object at the time when the time t has elapsed from the present time is less than the threshold D (step S115; “Yes”), the determination unit 124 sets the flag f ( “1” indicating that a warning is required is substituted for i) (step S116). On the other hand, when the distance d between the position of the grab 924 and the position of the first object when the time t has elapsed from the current time is greater than or equal to the threshold value D (step S115; “No”), the determination unit 124 sets the flag No change is made to f (i).

  Subsequently, the determination unit 124 determines whether i = n (step S117). If i <n (step S117; “No”), the determination unit 124 increments the counter i by 1 (step S118), and returns the process to step S114. Thereafter, the processing after step S114 is repeated for another object, and the possibility of contact with or proximity to the grab 924 is determined. On the other hand, when i = n (step S117; “Yes”), the determination unit 124 returns the process to step S110. Thereafter, the possibility of contact or proximity between the grab 924 and each of the n objects at the time when the time t has passed is determined.

  If the variable t exceeds the threshold T in the determination in step S111 (step S111; “Yes”), the determination unit 124 temporarily stores the flags f (1) to f (n) at that time in the storage unit 121. To end the series of processing.

  Through the series of processes described above, vectors indicating the moving direction and moving speed of each of the grabs 924 and the n objects are specified, and the time of the threshold T elapses from the current time for each of the n objects. Until then, flags f (1) to f (n) indicating whether or not the grab 924 and the object are in contact or close to each other are obtained. The above is the description of the processing performed by the determination unit 124.

  Returning to FIG. 3, the description of the functional configuration of the determination device 12 is continued. As described above, the display instruction unit 125 instructs the display device 104 to display an image representing the determination result by the determination unit 124. FIG. 8 is a diagram illustrating an image displayed on the display device 104 in accordance with an instruction from the display instruction unit 125. The image in FIG. 8 is obtained by cutting out the area surrounded by the hatch coaming 912 from the latest image captured by the imaging device 11, the circle circumscribing each of the grab 924 and the target object, their moving direction, and It is the image which added the arrow which shows a moving speed.

The display instruction means 125 performs the following processing every time a predetermined time elapses, and generates an image that instructs the display device 104 to display. First, the display instruction unit 125 reads the following data.
(A) Image data stored in the latest record of the image table (FIG. 4) (b) Position data and size data stored in the latest record of the suspended load position table (FIG. 5) (c) Object Position data and size data stored in each latest record of the position table (FIG. 5) (d) Vector data (e) storage means 121 corresponding to grab 924 temporarily stored in storage means 121 Flags f (1) to f (n) temporarily stored in the vector data (f) storage means 121 corresponding to each of the objects temporarily stored in
(G) Position data stored in each latest record of the structure position table (FIG. 6)

  Subsequently, the display instruction means 125 is centered on the position indicated by the position data read in (b) above the image represented by the image data read in (a), and the size read in (b) above. A circle with the diameter indicated by the data is added. Further, the display instruction unit 125 adds an arrow having a direction and a length corresponding to the vector indicated by the vector data read in (d) so as to extend outward from the added circle. At this time, if all of the flags f (1) to f (n) read in (f) above indicate “0”, the display instruction unit 125 adds a blue circle and an arrow, for example, When “1” is indicated, the display instruction unit 125 adds, for example, a red circle and an arrow.

  Subsequently, for each of the plurality of objects, the display instruction unit 125 adds a circle having a diameter centered on the position indicated by the position data read in (c) above and the diameter indicated by the size data read in (c) above. Then, an arrow having a direction and length corresponding to the vector indicated by the vector data read in (e) is added so as to extend outward from the added circle. At that time, when the flag corresponding to the object to which the circle and the arrow are added among the flags f (1) to f (n) read in (f) above indicates “0”, the display instruction unit 125 is, for example, blue When a circle and an arrow are added and the flag corresponding to the object indicates “1”, the display instruction unit 125 adds, for example, a red circle and an arrow.

  Subsequently, the display instructing unit 125 includes two positions indicated by the position data read out in (g) above (the corner portion on the bow side and starboard side of the hatch coaming 912 and the image including the circle and the arrow as described above). Cut out a rectangular area whose diagonal is the line connecting the stern and port side corners).

  Subsequently, the display instruction unit 125 enlarges or reduces the cut-out rectangular area image so as to have a predetermined size. At that time, the display instruction means 125 adjusts the thickness of the circle or arrow line added to the image to be the same regardless of the enlargement or reduction ratio. The display instruction unit 125 outputs the image generated in this way to the display device 104 and instructs the display of the image. As a result, an image as illustrated in FIG. 8 is displayed on the display device 104.

  When the operator operates the crane 92, the operator sees the image (FIG. 8) displayed on the display device 104, and moves the grab 924 in the moving direction and moving speed, as well as the cargo hold 91 such as an operator and heavy equipment. The movement direction and the movement speed of each of the objects can be intuitively known, and whether or not the grab 924 may contact or approach each of the objects can be intuitively known. For example, the operator can freely operate the crane 92 while no red circle or arrow is displayed on the display device 104, so that work efficiency is improved. If a red circle or arrow is displayed on the display device 104, the contact speed or the proximity of the grab 924 and the object is avoided by reducing the moving speed of the jib 922 or changing the moving direction. be able to. As described above, according to the determination system 1 according to the above-described embodiment, the burden on the operator of the crane 92 is reduced, and the work efficiency is improved and the safety is ensured.

[Modification]
The above-described embodiments can be variously modified within the scope of the technical idea of the present invention. Examples of these modifications are shown below. These modifications may be combined as appropriate.

(1) In the above-described embodiment, the determination means 124 is a period during which the threshold T of time elapses from the present time for the moving direction and moving speed of the suspended load and the object at the present time (hereinafter referred to as “future period” for convenience). ), Estimating future positions of suspended loads and objects, assuming they are maintained. The method by which the determination unit 124 estimates the future position of the suspended load and the object is not limited to this.

  For example, the determination unit 124 specifies the acceleration in movement in addition to the moving direction and moving speed based on the temporal change of the past position of the suspended load and the object, and the specified acceleration is maintained in the future period. Assuming that, the future position of suspended loads and objects may be estimated.

  In addition, the determination unit 124 specifies an approximate curve of the movement path indicated by a change in the past position of the suspended load as information indicating the movement state of the suspended load by a known method, and the suspended load is based on the identified approximate curve. The movement path in the future period may be estimated. FIG. 9 is a diagram for explaining processing performed by the determination unit 124 in this modification. A solid curve AB shown in FIG. 9 indicates a path along which the grab 924 has moved in the past period up to the present time. Note that point B is the current position of the grab 924.

  The determination unit 124 specifies an approximate curve of the curve AB. A dashed curve AC shown in FIG. 9 shows an example of an approximate curve specified by the determination means 124. The movement path of the grab 924 is a component of movement along a circle centering on the axis of the crane post 921 accompanying the turning of the jib 922, and a circle centering on the axis of the crane post 921 accompanying undulation or expansion / contraction of the jib 922. It is determined by the combination of the component of movement along the radial direction of. Therefore, it is desirable to use a non-linear curve such as a polynomial curve as an approximate curve of the moving route of the crane's suspended load.

  The determination means 124 estimates the future position of the suspended load when the suspended load moves along the specified approximate curve in the future period, and uses the estimated position to make contact between the suspended load and the object or Make a proximity decision.

  Further, the determination unit 124 specifies the vibration component included in the movement trajectory in the image of the suspended load as information indicating the movement state of the suspended load, and uses the identified vibration component to contact the suspended load with the object or You may make a proximity determination.

  The solid curve AB shown in FIG. 9 was obtained by interpolating the coordinates of the grab 924 in each coordinate system of a plurality of images captured in the past into coordinates in the coordinate system of the latest image. It is a curve. FIG. 10 illustrates a plurality of images captured in the past, starting from a coordinate obtained by converting the coordinates of the center point of each image in the coordinate system of the plurality of images captured in the past into coordinates in the coordinate system of the latest image. It is the figure which showed the time-dependent change of the vector which made the end point what converted the coordinate of the grab 924 in each coordinate system into the coordinate in the coordinate system of the newest image. The directions of the vectors shown in FIG. 10 are approximately 1:30 and 7:30, and the lengths of these vectors periodically increase and decrease.

  In this modification, the determination unit 124 identifies a vector that changes with time as illustrated in FIG. 10, and represents the vibration direction of the grab 924 by specifying the direction of the identified vector and the period and amplitude of increase / decrease of the vector length. Identify as information. The determination means 124 estimates the future position of the suspended load, assuming that the suspended load will continue to vibrate in the direction, period and amplitude so identified during the future period, and using the position thus estimated, Judgment of contact or proximity between suspended load and object.

  A dashed curve AC shown in FIG. 11 is a curve obtained by adding the vibration component specified as described above to the dashed approximate curve AC shown in FIG. The determination means 124 estimates the future position of the suspended load when the suspended load moves along the dashed curve AC shown in FIG. 11, and uses the estimated position to determine the suspended load and the target. Judgment of contact or proximity to objects.

  Although the vectors illustrated in FIG. 10 have substantially the same direction, the direction of the vector changes at a substantially constant angular velocity as time passes when the vibrating surface of the suspended load rotates. Therefore, the determination means 124 specifies the rate of change (angular velocity) of the vibration direction in addition to the current vibration direction, period, and amplitude as information representing the vibration component included in the trajectory of the suspended load. It may be used to determine contact or proximity between the object and the object.

  Further, based on the position of the feature point of the suspended load recognized by the image recognizing unit 123, the determination unit 124 uses information indicating the state of rotation of the suspended load about the rope 923 as information indicating the moving state of the suspended load. The information indicating the rotation state of the specified suspended load may be specified, and the contact or proximity of the suspended load and the object may be determined.

  FIG. 12 is a diagram illustrating a data structure of a suspended load position table used in this modification. In this modified example, as the position of the suspended load recognized by the image recognition means 123, in addition to the center position, the position of the connection point between the rope 923 and the suspended load (connection point position) and the positions of the four corners of the suspended load (the first position) 1 corner position to 4th corner position) is specified.

  FIG. 13 shows the coordinates of the connecting point position and the coordinates of the first corner position stored in the suspended load position table (FIG. 12) converted to the coordinates in the coordinate system of the latest image, and the converted coordinates of the connecting point position. FIG. 6 is a diagram showing a change over time of a vector having a starting point as a starting point and the coordinates of the converted first corner position as an end point. The direction of the vector shown in FIG. 13 changes periodically between the 11 o'clock direction and the 1 o'clock direction.

  In this modified example, the determination unit 124 identifies a vector that changes with time as illustrated in FIG. 13, and the direction of change and the speed of change (angular velocity) of the specified vector are defined as a suspended load with the rope 923 as an axis. It is specified as information indicating the state of rotation. Note that, as illustrated in FIG. 13, when the direction of the vector periodically changes within a certain angle range, the determination unit 124 specifies the angle range and the period as information indicating the state of rotation. Further, when the vector direction changes in the same direction at a substantially constant angular velocity, the determination unit 124 specifies the rotation direction and velocity (angular velocity) as information indicating the rotation state.

  FIG. 14 is a diagram illustrating a state in which the suspended load moves with rotation about the rope 923 as an example in which the suspended load is not a glove 924 but a long member. In FIG. 14, the rectangle indicated by the solid line indicates the past position (posture) of the suspended load recognized by the image recognition means 123, and the curve indicated by the broken line indicates an approximate curve of the moving path at the center of the suspended load. . Further, the rectangle indicated by the broken line in FIG. 14 indicates that the center of the suspended load moves along the approximate curve in the future period, and the rotation state specified as described above is maintained in the future period. Shows the future position (posture) of the suspended load.

  The determination means 124 estimates the position (posture) of the suspended load indicated by the broken-line rectangle in FIG. 14, and determines the contact or proximity between the suspended load and the object based on the estimated position (posture) of the suspended load. Do.

(2) In the embodiment described above, the height of the suspended load is not considered. When the suspended load falls during movement, the higher the suspended load is, the farther the suspended load moves during the fall. Therefore, the determination means 124 may specify the height of the suspended load and use the identified suspended load height to determine contact or proximity between the suspended load and the object.

  The higher the suspended load, the larger the size of the suspended load with respect to the size of the structure of the ship 9 in the image. Thus, for example, the size of the suspended load relative to the distance between the two feature portions of the structure of the ship 9 recognized by the image recognition means 123 in the image by the image recognition means 123 (for example, the two corner portions of the hatch combing 912). A configuration may be employed in which the distance threshold D is set to be larger as the ratio is larger.

(3) The determination device 12 may include a sound generation instruction unit that instructs the sound generation device to generate a sound according to the determination result by the determination unit 124.

(4) The determination device 12 may include a control instruction unit that instructs the control device of the crane 92 to execute control according to the determination result by the determination unit 124. For example, when the determination unit 124 sets a distance threshold E having a value smaller than the threshold D in addition to the distance threshold D used in the above-described embodiment, the suspended load and the object are closer to each other than the threshold E distance. If estimated, a configuration in which the control instruction means instructs the control device of the crane 92 to control the speed of movement of the jib 922 may be employed.

(5) In the embodiment described above, the suspended load is a grab, but the form and type of the suspended load are not limited.

(6) The use of the determination system according to the present invention is not limited to a crane mounted on a ship. For example, the determination system according to the present invention may be employed in a crane used on the ground.

DESCRIPTION OF SYMBOLS 1 ... Determination system, 9 ... Ship, 10 ... Computer, 11 ... Imaging device, 12 ... Determination device, 91 ... Cargo hold, 92 ... Crane, 101 ... Memory, 102 ... Processor, 103 ... Communication IF, 104 ... Display device, DESCRIPTION OF SYMBOLS 105 ... Operation apparatus, 121 ... Memory | storage means, 122 ... Image data receiving means, 123 ... Image recognition means, 124 ... Determination means, 125 ... Display instruction means, 911 ... Main body, 912 ... Hatch combing, 913 ... Hatch cover, 921 ... Crane post, 922 ... Jib, 923 ... Rope, 924 ... Grab, 925 ... Operation room

Claims (10)

Image recognition means for recognizing an object having a predetermined appearance characteristic and the suspended load in an image captured by an imaging device that is installed on a crane and performs imaging from above to below the suspended load of the crane. When,
Based on the time at which the image was captured and the position of the object and the suspended load recognized by the image recognition means in each of the images captured at a plurality of different times by the imaging device, Estimating the position of the object and the suspended load at each of a plurality of times within a period from the current time until the predetermined time elapses, and based on the estimated position, the object and the suspended load are A determination device comprising: determination means for determining whether or not a predetermined distance is approached from a current time until a predetermined time elapses. The determination means is based on the time of imaging of the image and the position of the suspended load recognized by the image recognition means in each of images taken at a plurality of different times by the imaging device. Determining an approximate curve of a past movement path of the suspended load, and estimating the movement path of the suspended load in a period from the current time until the predetermined time elapses based on the identified approximate curve. The determination apparatus according to 1 . The crane is arranged in a ship;
The determination unit identifies the vibration component contained in the trajectory of movement in the suspended within load of images recognized by the image recognition means, to claim 1 or 2 makes the determination by using the specified vibration component The determination apparatus described. The determination means, the image according to the moving speed of the hanging in load of the image recognized by the recognition means, the determination device according to any one of claims 1 to 3 to change the distance said predetermined . The determination means relates to each of images taken at a plurality of different times by the imaging device, based on the time of imaging of the image and the position of the feature point of the suspended load recognized by the image recognition means, The determination according to any one of claims 1 to 4 , wherein a state of rotation of the suspended load about a rope that suspends the suspended load is specified, and the determination is performed using the specified state of rotation. apparatus. In an image captured by an imaging device that is installed in a crane disposed on a ship and performs imaging from above to below the suspended load of the crane, an object having a predetermined appearance characteristic and the suspended load Image recognition means for recognizing a plurality of parts of the ship structure;
The position of each of the plurality of portions recognized by the image recognition means in one of images taken at a plurality of different times by the imaging device and the image taken at a plurality of different times by the imaging device The object recognized by the image recognition means in the one image based on the position of each of the plurality of portions recognized by the image recognition means in another image different from the one image And the position of the object recognized by the image recognition means in the other image, and the position of the suspended load recognized by the image recognition means in the one image and the other image each of the above-identified the relationship between the position of the suspended load is recognized by the image recognition means, images captured at different times by the image pickup device in The time when the object and the suspended load are determined in advance from the current time based on the time when the image was captured and the position of the object and the suspended load recognized by the image recognition means in the image Determining means for determining whether or not the distance is closer than a predetermined distance in a period until the
A determination apparatus comprising: Determining device according to any one of claims 1 to 6 comprising display instruction means for instructing the display of an image with respect to the captured image obtained by adding a graphic representing the result of the determination by the imaging apparatus to the display device. An imaging device that is installed in a crane and performs imaging from above the suspended load of the crane toward the bottom;
A determination system comprising: the determination apparatus according to claim 1, wherein the determination apparatus is connected to the imaging apparatus and receives image data representing an image captured by the imaging apparatus from the imaging apparatus. On the computer,
A process of recognizing an object having a predetermined appearance feature and the suspended load in an image captured by an imaging device that is installed on a crane and performs imaging from above to below the suspended load of the crane;
For each of the images captured in a plurality of different times by the image pickup device, and the time of imaging of the image, based on the position of the recognized object to be recognized in the process and the suspended within load of the image, from the present A process of estimating the position of the object and the suspended load at each of a plurality of times within a period until the predetermined time elapses;
A program for executing , based on the estimated position, a process for determining whether or not the object and the suspended load are closer than a predetermined distance in a period until a predetermined time elapses from the present time. . On the computer,
A process of recognizing an object having a predetermined appearance feature and the suspended load in an image captured by an imaging device that is installed on a crane and performs imaging from above to below the suspended load of the crane;
For each of the images captured in a plurality of different times by the image pickup device, and the time of imaging of the image, based on the position of the recognized object to be recognized in the process and the suspended within load of the image, from the present A process of estimating the position of the object and the suspended load at each of a plurality of times within a period until the predetermined time elapses;
A program for executing , based on the estimated position, a process for determining whether or not the object and the suspended load are closer than a predetermined distance in a period until a predetermined time elapses from the present time. Is a recording medium that is stored continuously in a readable manner in the computer.

Images