JP6858104B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device and an information processing method for supporting cargo handling work of a forklift.

In cargo handling work using a forklift, the forklift driver visually checks the fork insertion position of the pallet on which the luggage is placed, or while checking the captured image acquired by the camera that captures the front of the forklift on the monitor, the fork difference. Positioning of the inlet and the fork was performed. For example, in Patent Document 1, the fork and the fork insertion port are positioned by using a display image in which a scale image representing at least a part of a fork or a load on the fork and an image taken by a camera are superimposed. Further, in Patent Document 2, the image data obtained by photographing the marks provided on the pallet on which the load stored in the storage portion of the shelf is placed and the marks provided on the shelves with a camera is subjected to pattern matching processing to obtain a pattern of the marks. The image is recognized to determine the position and size of the mark, and the fork is positioned based on this.

Japanese Patent No. 4666154 Japanese Patent No. 3921968

For example, in cargo handling work at a high place, it is difficult to visually accurately position the fork and the insertion port. Therefore, the pallet is lifted and loaded with the fork not sufficiently inserted into the pallet. Then, the load placed on the pallet may collapse.

In view of the above circumstances, an object of the present invention is to provide an information processing device and an information processing method capable of confirming the fork insertion amount.

In order to achieve the above object, the information processing apparatus according to one embodiment of the present invention includes an image acquisition unit and a display image signal generation unit.
The image acquisition unit acquires a photographed image of a pallet provided with a positioning mark or an object placed on the pallet, which is photographed by an imaging unit provided so as to be interlocked with the raising and lowering of the fork of the forklift.
The display image signal generation unit generates a display image signal that is a display image in which the captured image acquired from the image acquisition unit and a reference mark having an outer shape similar to the outer shape of the positioning mark are superimposed.

According to such a configuration, the fork can be positioned using the displayed image in which the captured image and the reference mark similar to the positioning mark are superimposed, so that the positioning mark and the reference mark can be displayed on the display image. By operating the forklift so that it almost matches, in addition to positioning the fork with respect to the insertion port, it is possible to perform positioning in the fork insertion direction while checking the fork insertion amount on the display image. As a result, it is possible to prevent the load from collapsing due to insufficient insertion of the fork into the pallet, and safe cargo handling work becomes possible.

Based on the position detection unit that detects the positions of the positioning mark and the reference mark in the display image and the position detection result by the position detection unit, the distance between the center of the positioning mark and the center of the reference mark, and Further, a determination unit for determining whether or not the distance between the outer shape of the positioning mark and the outer shape of the reference mark is within a predetermined range may be further provided.

The outer shape of the positioning mark and the reference mark may have a circular shape.

The circle detection unit for detecting the circular shape of the positioning mark is further provided, and the display image signal generation unit further includes the positioning mark of the display image when the circular shape of the positioning mark is detected by the circle detection unit. A display image signal that becomes a display image in which the outer shape of is highlighted may be generated.

The display image signal generation unit may generate a display image signal that is a display image in which a line segment connecting the center of the positioning mark and the center of the reference mark is further displayed on the display image.

The display image signal generation unit has an outer shape similar to the captured image, the reference mark, and the outer shape of the reference mark, is located inside the reference mark, and is centered at the same position as the center of the reference mark. A display image signal which is a display image in which the reference mark before the fork insertion and the reference mark having the above are superimposed may be generated.

The display image signal generation unit has an outer shape similar to the outer shape of the captured image, the reference mark, and the reference mark, is located so as to surround the reference mark, and is located at the same position as the center of the reference mark. A display image signal as a display image in which a reference mark for limiting the distance between the forklift and the pallet, which has a center, is superimposed may be generated.

In order to achieve the above object, the information processing method according to another embodiment of the present invention is performed on a pallet provided with a positioning mark or a pallet taken by a camera provided so as to be interlocked with the raising and lowering of a fork of a forklift. A photographed image of the mounted object to be mounted is acquired, and a display image signal is generated as a display image in which the photographed image and a reference mark having an outer shape similar to the outer shape of the positioning mark are superimposed. The positions of the positioning mark and the reference mark are detected, and based on the position detection result, it is determined whether or not the distance between the center of the positioning mark and the center of the reference mark is within a predetermined range. Based on the position detection result, it may be determined whether or not the distance between the outer shape of the positioning mark and the outer shape of the reference mark is within a predetermined range.

As described above, according to the present invention, it is possible to confirm the fork insertion amount, and in addition to positioning the fork with respect to the insertion port, positioning in the fork insertion direction can also be performed.

It is a side view which shows the state of cargo handling work using a forklift. It is a schematic front view which shows the appearance that the luggage was placed on the pallet. It is a partial schematic plan view and a partial schematic side view of a forklift which shows the state of inserting a fork into a pallet in cargo handling work using a forklift. It is a block diagram which shows the functional structure example of the information processing apparatus which concerns on one Embodiment of this invention. It is an example of the display image displayed on a monitor, and is the figure explaining the positional relationship between a positioning mark and a reference mark at the time of positioning. It is a flow chart which shows an example of the information processing method by the information processing apparatus. It is a figure which shows the position which provides the positioning mark which concerns on other embodiment of this invention. It is a figure which shows the shape of the reference mark which concerns on still another Embodiment of this invention, the reference mark before fork insertion, and the reference mark for restriction.

Hereinafter, the positioning of the fork of the forklift according to the embodiment of the present invention will be described with reference to the drawings. In the following figures, the X, Y, and Z-axis directions indicate three axial directions that are orthogonal to each other, and the Z-axis direction corresponds to the height direction.

In the present embodiment, the loading and unloading work is performed by inserting a fork into a loading pallet (hereinafter, simply referred to as a pallet) on which a load as a loading object is placed using a forklift and lifting the pallet. A positioning mark is affixed to the pallet or the luggage placed on the pallet at a predetermined position. The forklift is provided with a camera that moves up and down in conjunction with the raising and lowering of the fork.

In the information processing apparatus of the present embodiment, a photographed image of a pallet or a baggage including a positioning mark photographed by a camera is acquired, and a display image signal of a display image in which the photographed image and a reference mark are superimposed is generated.

The forklift driver drives the forklift while looking at the monitor on which the generated display image is displayed, positions the fork so that it is located at the pallet insertion port, and then inserts the fork from the insertion port. ..

By checking the positional relationship between the positioning mark and the reference mark displayed on the monitor, the driver can intuitively grasp the positional relationship between the fork and the insertion port and the amount of the fork inserted into the pallet. It has become. Hereinafter, a detailed description will be given.

Hereinafter, the state in which the fork is sufficiently inserted into the fork insertion area of the pallet is referred to as the normal position of the fork at the time of insertion. When the fork is fully inserted into the pallet, the load placed on the pallet does not collapse when the fork is inserted and the pallet is lifted by the forklift to perform the unloading work. A state in which the state that can be held above can be reliably maintained.

When the fork is in the normal position, typically, the side surface where the pallet insertion port is provided and the fork extending direction is located substantially orthogonal to the side surface where the pallet insertion port is provided. When the XZ plane is used, the central portion of the pair of forks in the X-axis direction and the central portion of the pallet in the X-axis direction substantially coincide with each other, and the forks are sufficiently inserted into the pallet.

The normal position of the fork at the time of insertion means that the fork is positioned in the XYZ direction with respect to the pallet, and when the fork is inserted to pick up and load the fork, the stable arrangement of the load on the pallet is maintained. Refers to the optimal position of the fork that can be made possible.

(First Embodiment)
FIG. 1 is a schematic side view showing a state of cargo handling work using a forklift. FIG. 2 is a schematic front view showing how the luggage is placed on the pallet. FIG. 3 shows a schematic plan view and a schematic side view for explaining the positional relationship between the pallet and the fork in a state where the fork is inserted into the pallet. FIG. 4 is a block diagram showing a functional configuration example of the information processing device mounted on the forklift shown in FIG.

<Pallet configuration>
The pallet 30 is a cargo handling platform on which the luggage 40 as a loading object is placed. As shown in FIGS. 1 to 3, the pallet 30 has a pair of upper plates 31 and lower plates 34 arranged apart from each other, and four rod-shaped pallets provided between the upper plate 31 and the lower plate 34. It has a digit 33 of. The upper plate 31 and the lower plate 34 each have a rectangular planar shape.

The four digits 33 are arranged parallel to each other and separated from each other. Of the four digits 33, the two digits 33 are provided along the two opposite sides of the rectangular upper plate 31 and the lower plate 34, respectively. The other two are arranged in a region sandwiched by a pair of digits 33 provided along the sides of the upper plate 31 and the lower plate 34.

Three spatial regions are formed in the pallet 30 by the upper plate 31, the lower plate 34, and the digits 33. When the longitudinal direction of the rod-shaped digit 33 is the Y-axis, the three spatial regions are located side by side in the X-axis direction. Two of the three spatial regions on both sides are fork insertion regions into which the fork 16 described later is inserted. The fork insertion region is a rectangular parallelepiped space having a longitudinal direction in the Y-axis direction on the drawing. The insertion port 32 of the fork 16 is formed by the upper plate 31, the lower plate 34, and the digit 33.

On the pair of side surfaces 37 of the pallet 30 facing each other, an insertion port 32 into which the fork 16 of the forklift 10 is inserted is provided. Two outlets 32 are arranged on each of the pair of side surfaces 37 of the pallet 30 facing each other. At the time of cargo handling work, the forklift 10 is arranged to face each other on one side surface 37 where the insertion port 32 of the pallet 30 on which the luggage 40 is placed is provided, and the pair of forks 16 of the forklift 10 are inserted into the insertion port 32 to load. The work of taking and storing is carried out.

A positioning mark 50 is provided on the pallet 30 or the luggage 40 placed on the pallet 30. When the positioning mark 50 is provided on the pallet 30, it is provided on the side surface 37 of the pallet 30 where the insertion port is provided.

Generally, the luggage 40 placed on the pallet 30 is formed by stacking a plurality of substantially rectangular parallelepiped loads, and the entire luggage 40 loaded on one pallet 30 has a substantially rectangular parallelepiped shape. When the positioning mark 50 is provided on the luggage 40, it is provided on the surface of the luggage 40 along the side surface of the pallet 30 on which the insertion port is provided.

Usually, the surface of the luggage 40 on which the positioning mark 50 is provided has a larger area than the side surface 37 of the pallet 30. Therefore, when the positioning mark 50 is provided on the luggage 40 side, the degree of freedom in the design range such as the size of the positioning mark 50 is larger than when the positioning mark 50 is provided on the side surface side of the pallet 30, and the lens selection range of the camera 22 can be increased. Can be wide.
In the present embodiment, a case where the positioning mark 50 is provided on the luggage 40 will be described as an example.

The positioning mark 50 is formed, for example, in the shape of a sticker and can be attached by an operator. The arrangement range of the positioning mark 50 is preset depending on the positional relationship with the pallet 30 and the installation location of the camera 22.

In this embodiment, the positioning mark 50 has a circular outer shape. The design drawn on the positioning mark 50 is not particularly limited. The reference mark 60, which will be described later, has a circular outer shape similar to the positioning mark 50. The reference mark 60 superimposed on the captured image is prepared in advance corresponding to the positioning mark 50 used, and is stored in a storage unit (not shown) of the information processing apparatus 70.

With the forklift 10 and the pallet 30 in the normal position, the area where the positioning mark 50 can be arranged is determined so that the camera 22 described later can take an image of the positioning mark 50. The positioning mark 50 is affixed within this displaceable area. The location where the positioning mark 50 is attached is determined based on the positional relationship between the camera 22 and the fork 16.

The camera 22 is configured to photograph a part of the luggage 40 including the positioning mark 50. A monocular camera or a stereo camera can be used as the camera 22, and a monocular camera is typically used, and the cost can be suppressed as compared with the case where the stereo camera is used.

The camera 22 can move in conjunction with the raising and lowering of the fork 16, and the distance between the camera 22 and the fork 16 does not change and is fixed. The details of the installation location of the camera 22 will be described later, but the camera 22 is installed above the fork 16 in the height direction (Z-axis direction). Further, the camera 22 is provided at the center of the pair of forklifts 10 in the X-axis direction.

In the present embodiment, since the camera 22 is provided at the central portion of the pair of forklifts 10 in the X-axis direction, the dispositionable area of the positioning mark 50 in the X-axis direction is determined to be near the central portion of the luggage 40 in the X-axis direction. Will be done. The operator may arrange the luggage 40 so that the positioning mark 50 is located around the central portion in the X-axis direction. For example, the two outlets 32 formed on the side surfaces 37 of the pallet 30 are provided on the side surface 37 in line symmetry with respect to the center line along the Z-axis direction, so that the central portions of the two outlets 32 are provided. The central portion in the X-axis direction can be determined by using the hit as a guide.

Further, the area where the positioning mark 50 can be arranged in the height direction (Z-axis direction) is determined by the distance from the upper surface of the pallet 30 based on the distance between the camera 22 and the fork 16. For example, when the distance between the camera 22 and the fork 16 is 30 cm or 30 cm, the positioning mark 50 may be arranged so as to be located between 20 cm and 40 cm from the upper surface of the pallet 30. In this way, the dispositionable area of the positioning mark 50 is determined by the position of the camera 22. The above numerical values are examples, and the present invention is not limited to these.

<Forklift configuration>
As shown in FIG. 1, the forklift 10 includes a vehicle body 1 having a driver's seat, a mast 15, a pair of forks 16, a backrest 18, a fork rail (not shown), and a camera 22 as an image pickup unit. The information processing device 70 and the monitor 21 as a display device are provided.

The fork 16 is used to lift a pallet or the like. The fork 16 is formed so as to extend in one direction and is fixed to the backrest 18.

The backrest 18 is a load receiving frame for preventing the load placed on the fork 16 from falling behind the mast 15. The backrest 18 is provided on a plane along the height direction, which is perpendicular to the extending direction of the fork 16.

A fork rail is a rail to which a fork is attached. The distance between the pair of forks can be adjusted by sliding it on the fork rail. The fork rail has a longitudinal direction in a direction orthogonal to the extending direction of the fork 16 in a plane parallel to the horizontal plane. The fork rail is provided between the opposite sides of the rectangular outer frame of the backrest 18.

The backrest 18 to which the fork 16 is fixed is configured to be able to move up and down by a lift chain (not shown). As a result, the fork 16 can also be raised and lowered.

The mast 15 acts like a rail required to move the backrest 18 with the fork 16 up and down. The mast 15 is provided in front of the vehicle body 1 which is in front of the driver when he / she gets in the driver's seat.

The camera 22 as an imaging unit has an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and images the front field of view of the forklift 10. The imaging unit 22 is typically a video camera such as a CCD camera.

The camera 22 is provided, for example, on the fork rail and is provided at the center of the fork rail having a longitudinal direction along the X-axis direction. The camera 22 is provided above the fork 16 in the height direction. The camera 22 is not limited to the example provided on the fork rail, and may be provided at a position where the camera 22 moves up and down in conjunction with the fork 16.

<Configuration of information processing device>
The information processing device 70 has hardware necessary for a computer such as a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disk Drive). When the CPU loads the program stored in the ROM or HDD into the RAM and executes it, the operation of each block of the information processing device 70, which will be described later, is controlled.

The program is installed in the information processing apparatus 70 via, for example, various storage media (internal memory). Alternatively, the program may be installed via the Internet or the like.

As the information processing device 70, for example, a PC (Personal Computer), a tablet terminal, or the like is used, but any other computer may be used.

The information processing device 70 includes an image acquisition unit 71, a display image signal generation unit 72, a circle detection unit 73, a position detection unit 74, a determination unit 75, an audio signal generation unit 76, and an output unit 77. To do.

The image acquisition unit 71 acquires a captured image captured by the camera 22.

FIG. 5 is an example of a display image displayed on the monitor 21 and is a diagram for explaining the positional relationship between the positioning mark and the reference mark at the time of positioning. When the side surface 37 provided with the insertion port 32 into which the fork 16 is inserted is the XZ surface, FIG. 5A is an example of a display image displayed on the monitor 21 when positioning the fork on the XZ surface. .. FIG. 5B is an example of a display image showing how the fork is positioned on the XZ plane. FIG. 5C is an example of a display image displayed on the monitor 21 when positioning the fork in the insertion direction (Y-axis direction) of the fork 16 into the pallet 30.

The display image signal generation unit 72 generates a display image signal of the display image displayed on the display unit 211 of the monitor 21. The display image signal generation unit 72 generates a display image signal which is a display image in which the captured image acquired by the image acquisition unit 71 and the reference mark 60 are superimposed.

Further, the display image signal generation unit 72 can generate a display image signal in which the circular outline 50B of the positioning mark 50 of the display image on which the captured image and the reference mark 60 are superimposed is highlighted. It is composed of. 5 (A) to 5 (C) show an example of the displayed image. The highlighting of the circular contour 50B of the positioning mark 50 is executed when the circle is detected by the circle detection unit 73, which will be described later.

Further, the display image signal generation unit 72 further superimposes and displays the line segment 55 connecting the center 50A of the positioning mark 50 and the center 60A of the reference mark 60 on the display image on which the captured image and the reference mark 60 are superimposed. It is configured to be able to generate a display image signal to be a display image. FIG. 5A shows an example of the displayed image. The display of the line segment 55 is executed based on the detection result by the position detection unit 74, which will be described later.

The display image signal generated by the display image signal generation unit 72 is transmitted to the output unit 77. In the present embodiment, the outer shape of each of the positioning mark 50 and the reference mark 60 has a circular shape. When the outer shape of the positioning mark 50 and the outer shape of the reference mark 60 substantially overlap on the monitor 21, the fork is in the normal position when the fork is inserted.

The circle detection unit 73 detects the circle of the positioning mark 50 in the captured image captured by the camera 22. When the circle detection unit 73 detects a circle on the captured image, the detection result is transmitted to the display image signal generation unit 72.

When the circular shape of the positioning mark is detected by the circle detection unit 73, the extending direction of the fork 16 is Y when the side surface of the pallet 30 on which the insertion port 32 into which the fork 16 is inserted is arranged is an XZ plane. It is considered that the positional relationship is almost parallel to the axial direction. Here, the detected circle includes not only a perfect circle but also a substantially perfect circle.

For example, when the side surface 37 of the pallet 30 on which the insertion port 32 is arranged is the XZ surface and the fork 16 is inserted diagonally to the side surface 37 of the pallet 30 instead of perpendicular to the X-axis direction, in other words, Y. When the fork 16 is inserted in a direction at an angle with respect to the axis, the outer shape of the positioning mark in the captured image becomes a distorted circular shape that is not a perfect circle.

In this way, even when the fork 16 is inserted diagonally rather than perpendicularly to the side surface 37 of the pallet 30, the fork 16 can be sufficiently inserted into the pallet 30 in that state, and the fork 16 is inserted. In that state, there may be no problem in loading and unloading work. Even if the fork 16 is inserted diagonally with respect to the side surface of the pallet 30 within a range where there is no problem in such cargo handling work, the circle detection is included so that the degree of the slant is small and does not affect the cargo handling work. Part 73 is configured to detect other than a perfect circle and a nearly perfect circle as a circle.

Based on the detection result of the circle detection unit 73, the display image signal generation unit 72 is a display image signal in which the circular outline of the positioning mark is highlighted among the display images in which the captured image and the reference mark are superimposed. To generate.

In highlighting, for example, the contour 50B is colored and displayed so that the position of the positioning mark 50 in the display image displayed on the monitor 21 can be immediately recognized. In the present embodiment, for example, the circular contour 50B of the positioning mark 50 is colored in red and displayed, and the circular contour of the reference mark 60 is displayed in green.

By highlighting the circular contour 50B of the positioning mark 50, the driver can intuitively grasp that the forklift 10 is facing the luggage 40 while looking at the monitor 21.

In addition to the notification method of highlighting the circular outline, the notification that the forklift 10 faces the luggage 40 may be indicated in characters and displayed as a display image.

In this way, the display image in which the circular outlines of the positioning mark 50 and the reference mark 60 are highlighted is displayed on the monitor 21, so that the driver can intuitively understand the positional relationship between the positioning mark 50 and the reference mark 60. Can be grasped. The coloring color at the time of highlighting is not limited to these, and the driver may be able to select an arbitrary color depending on the color of the pallet or the luggage on which the positioning mark is provided. Further, the highlighting is not limited to the colored display, and may be highlighted with a thick line.

The position detection unit 74 detects the positions of the reference mark 60 and the positioning mark 50 in the display image, and detects the coordinates and radii of the centers of the reference mark 60 and the positioning mark 50, respectively. The detection result is transmitted to the display image signal generation unit 72 and the determination unit 75.

The display image signal generation unit 72 adds the positioning mark 50 to the display image on which the captured image and the reference mark 60 are superimposed, based on the detection results of the center coordinates of the reference mark 60 and the positioning mark 50 from the position detection unit 74. A line segment 55 connecting the center 50A and the center 60A of the reference mark 60 is further superimposed to generate a display image signal as a display image. The line segment connecting the centers may be highlighted by, for example, being colored or highlighted by a thick line.

As a result, the driver can determine the positional relationship between the positioning mark 50 and the reference mark 60 from the length of the line segment 55 and the arrangement position of the line segment 55, and by extension, the insertion port 32 and the tip portion 16A of the fork 16. You can intuitively grasp the positional relationship of. Then, while checking the display image displayed on the display unit 211 of the monitor 21, the driver shortens the distance of the line segment 55 so that the center 50A of the positioning mark 50 and the center 60A of the reference mark 60 substantially coincide with each other. The forklift 10 is operated so as to be. As a result, the fork 16 is positioned on the XZ surface.

The determination unit 75 determines whether or not the distance between the centers of the reference mark 60 and the positioning mark 50 is within a predetermined range based on the detection result from the position detection unit 74. Further, it is determined whether or not the value of Rr, which is the difference between the radius value R of the reference mark 60 and the radius value r of the positioning mark 50, is within a predetermined range. The determination result is transmitted to the display image signal generation unit 71 and the audio signal generation unit 76.

When the distance between the centers of the reference mark 60 and the positioning mark 50 is within a predetermined range, the tip of the fork 16 and the insertion port 32 on the XZ surface are positioned.

When the difference between the radius value R of the reference mark 60 and the radius value r of the positioning mark 50 is within a predetermined range, the fork 16 is inserted into the pallet 30 at a normal position, and the fork 16 and the insertion port 32 in the XYZ direction are inserted. It is in a positioned state.

When the voice signal generation unit 76 receives the determination result from the determination unit 75 that the value of Rr, which is the difference between the radius value R of the reference mark and the radius value r of the positioning mark, is within a predetermined range, the voice signal generation unit 76 operates. A voice is generated to notify the person that the fork 16 is in the normal position, that is, that the positioning of the fork 16 is completed. The audio signal generated by the audio signal generation unit 76 is transmitted to the output unit 77.

The voice signal generation unit 76 generates a voice for notifying that the fork 16 is in the normal position, and also notifies that the circle is detected when the circle detection unit 73 detects the circle. A voice for this purpose or a voice for notifying whether or not the distance between the centers of the reference mark and the positioning mark is within a predetermined range may be generated.

Further, the display image signal generation unit 72 displays in characters a notification that the positioning of the fork 16 has been completed based on the determination result that the distance between the center of the reference mark and the positioning mark from the determination unit 75 is within a predetermined range. The displayed display image may be generated.

The audio signal generated by the audio signal generation unit 76 is transmitted to the output unit 77. The output unit 77 outputs the display image signal and the audio signal to the monitor 21.

The monitor 21 as a display device includes a display unit 211 and a speaker 212 as an audio output unit. The monitor 21 is provided at a position where the driver sitting in the driver's seat can check it.

The display unit 211 is configured to be capable of displaying an image before superimposition processing captured by the camera 22 and a display image based on the display image signal output from the information processing device 70. The display unit 211 is, for example, a display device using a liquid crystal, an organic EL (Electro-Luminescence), or the like. The display unit 211 may be configured to have an input unit for inputting a user's operation. As the input unit, for example, an operation device such as a touch panel or a keyboard is adopted.

The speaker 212 emits voice based on the voice signal output from the information processing device 70. By providing the speaker 79 in this way, the driver can obtain the support information for positioning the fork 16 and the pallet 30 from the auditory sense in addition to the visual support information.

<Positioning method>
Next, a method of positioning the fork 16 and the insertion port 32 of the pallet 30 at the time of loading by the forklift 10 using the above-mentioned information processing device 70 will be described. FIG. 5 is an example of a display image displayed on the monitor 21 and is a diagram for explaining the positional relationship between the positioning mark and the reference mark at the time of positioning. FIG. 6 is a flow chart showing an example of an information processing method related to image processing of a photographed image using the above-mentioned information processing device 70. FIG. 6 is a flow chart relating to the circle detection unit 73, the position detection unit 74, and the determination unit 76.

Hereinafter, the step of positioning the fork 16 will be described with reference to FIGS. 1, 4, 5, and 6, together with an explanation of an information processing method related to image processing of a display image by the information processing apparatus 70.

As shown in FIG. 1, when the forklift 10 is arranged so as to face the side surface 37 where the insertion port 32 of the pallet 30 is provided, first, the fork 16 and the pallet 30 to be lifted are operated by the driver. The pallet 16 is raised so that its position in the height direction is approximately the same.

Next, as shown in FIG. 4, a part of the luggage 40 including the positioning sticker 50 is photographed by the camera 22. The image acquisition unit 71 of the information processing device 70 acquires the captured image acquired by the camera 22.

The display image signal generation unit 72 generates a display image signal which is a display image in which the captured image acquired by the image acquisition unit 71 and the reference mark 60 are superimposed. The reference mark 60 is prepared in advance corresponding to the positioning sticker 50 to be used, and the positioning sticker 50 is always displayed in the same size on the display image.

The generated display image signal is transmitted from the output unit 77 to the display device 21. A display image based on the transmitted display image signal is displayed on the display unit 211 of the display device 21.

Next, as shown in FIG. 6, the circle detection unit 73 detects the circle in the captured image (S101). As shown in FIG. 4, the detection result is transmitted to the display image signal generation unit 72. When the display image signal generation unit 72 receives the result that the circle is detected by the circle detection unit 73, the outline 50B of the positioning mark 50 in the display image on which the captured image and the reference mark 60 are superimposed is highlighted. Generates the display image signal of the displayed image. In this embodiment, the contour 50B of the positioning mark 50 is colored red and highlighted. The outline of the circular reference mark 60 is colored in green and displayed.

When the circular shape of the positioning mark 50 is detected by the circular detection unit 73, the extending direction of the fork 16 is defined as the XZ plane on the side surface of the pallet 30 on which the insertion port 32 into which the fork 16 is inserted is arranged. Is considered to be in a positional relationship substantially parallel to the Y-axis direction on the pallet 30, and the forklift 10 is considered to be located directly in front of the luggage 40.

The outline 50B of the positioning mark 50 is not highlighted on the display image until the circle detection unit 73 detects the circle of the positioning mark 50. The driver operates and moves the forklift 10 so that the contour 50B of the positioning mark 50 displayed while looking at the display image displayed on the display device 21 is substantially a perfect circle. Then, by confirming on the display unit 211 that the contour 50B of the positioning mark 50 is highlighted, it becomes possible to grasp that the forklift 10 is located directly in front of the luggage 40.

Next, as shown in FIG. 6, the position detection unit 74 uses a display image in which the outline 50B of the positioning mark 50 in the display image on which the captured image and the reference mark 60 are superimposed is highlighted and used as a reference. The positions of the mark 60 and the positioning mark 50 are detected, the coordinates of the center 60A of the reference mark 60 and the center 60B of the positioning mark 50, and the radius values of the reference mark 60 and the positioning mark 50 are detected (S102). As shown in FIG. 4, the detection result is transmitted to the display image signal generation unit 72 and the determination unit 75.

Based on the detection result of the position detection unit 74, the display image signal generation unit 72 highlights the outline 50B of the positioning mark 50 of the display image on which the captured image and the reference mark 60 are superimposed, and further highlights the contour 50B of the reference mark 60. A display image signal is generated, which is a display image in which the line segment 55 connecting 60A and the center 50A of the positioning mark 50 is displayed.

The generated display image signal is transmitted to the display device 21, and the display image is displayed on the display unit 211 based on the transmitted display image signal. FIG. 5A is an example of a display image. As shown in FIG. 5A, the outline 50B of the positioning mark of the display image on which the captured image and the reference mark 60 are superimposed is highlighted on the display unit 211, and the center 60A of the reference mark 60 and the positioning mark are further highlighted. A display image on which the line segment 55 connecting the center 50A of 50 is superimposed is displayed.

While checking the display image displayed on the display unit 211, the driver raises the forklift 10 so that the distance of the line segment 55 is shortened so that the center 50A of the positioning mark and the center 60A of the reference mark substantially coincide with each other. Manipulate. As a result, positioning on the XZ plane is performed.

Next, as shown in FIG. 6, is the distance between the centers of the positioning mark 50A and the center 60A of the reference mark within a predetermined range based on the detection result from the position detection unit 74 by the determination unit 75? Whether or not it is determined (S103). If Yes is determined in S103, the process proceeds to S104. If No is determined in S103, the process returns to S101 and the step is repeated. As shown in FIG. 4, the determination result by the determination unit 75 is transmitted to the display image signal generation unit 72 and the audio signal generation unit 76.

When the determination unit 75 determines that the distance between the centers is within a predetermined range, it is considered that the fork 16 has been positioned on the XZ plane. That is, it is considered that the extending direction of the fork 16 has a positional relationship substantially parallel to the Y-axis direction of the pallet 30, and the tip 16A of the fork 16 faces the insertion port 32.

When the determination unit 75 determines that the distance between the centers is within a predetermined range, the display image signal generation unit 72 generates a display image signal of the display image in which the line segment 55 is highlighted. The generated display image signal is transmitted to the display device 21 and displayed on the display unit 211. The highlighting of the line segment 55 is performed, for example, by displaying the line segment 55 in a color different from that in the case where the distance between the centers is not within a predetermined range. As a result, the driver can intuitively recognize that the fork 16 has been positioned on the XZ plane by confirming the color change of the line segment 55 on the image display.

FIG. 5B is an example of a display image when the distance between the centers is 0 and it is determined that the distance is within a predetermined range. As shown in FIG. 5B, the center 50A of the positioning mark 50 and the center 60A of the reference mark 60 overlap each other. When the centers overlap in this way, the line segment 55 is hidden, and points representing the overlapped centers 50A and 60A are displayed.

Further, when the determination unit 75 determines that the distance between the centers is within a predetermined range, the audio signal generation unit 76 generates an audio signal notifying that the fork 16 has been positioned on the XZ surface. The generated voice signal is transmitted to the display device 21, and the voice is output from the speaker.

When the driver confirms that the fork 16 has been positioned on the XZ surface from the displayed image and sound, the driver operates the forklift 10 so that the positioning mark 50 and the reference mark 60 coincide with each other. That is, the forklift 10 is operated so as to move in parallel with the Y-axis direction of the pallet 30. As a result, the fork 16 is inserted into the pallet 30. The driver can intuitively grasp the insertion amount of the fork 16 by confirming the positional relationship between the positioning mark 50 and the reference mark 60 on the display image.

When the determination unit 75 determines that the value of Rr, which is the difference between the radius R of the reference mark 60 and the radius r of the positioning mark 50, is within a predetermined range, the positioning of the fork 16 in the Y-axis direction is performed. It is considered that there is a sufficient amount of fork 16 plugged in for safe unloading work.

When the determination unit 75 determines that the value of Rr, which is the difference between the radius R and the radius r, is within a predetermined range, the display image signal generation unit 72 positions the fork 16 on the XYZ plane. That is, a display image signal is generated, which is a display image indicating that the positioning of the fork 16 is completed. The generated display image signal is transmitted to the display device 21 and displayed on the display unit 211. The driver can recognize that the positioning of the fork 16 has been completed by checking the positioning end notification displayed on the display unit 211.

Further, when the determination unit 75 determines that the value of Rr, which is the difference between the radius R and the radius r, is within a predetermined range, the audio signal generation unit 76 positions the fork 16 on the XYZ plane. That is, an audio signal is generated to notify that the fork 16 is arranged in the normal position and the positioning of the fork 16 is completed. The generated voice signal is transmitted to the display device 21, and the voice is output from the speaker. The driver can recognize that the positioning of the fork 16 has been completed by the voice from the speaker.

As described above, in the present embodiment, the fork is positioned using the captured image in which the positioning mark is captured and the display image in which the reference mark similar to the positioning mark is superimposed, so that the fork is positioned on the display image. By operating the forklift so that the positioning mark and the reference mark almost match, in addition to positioning with respect to the fork insertion port, positioning in the fork insertion direction is also performed while checking the fork insertion amount on the displayed image. Can be done. As a result, it is possible to prevent the load from collapsing due to insufficient insertion of the fork into the pallet, and safe cargo handling work becomes possible.

In the present embodiment, the circular contour of the positioning mark is detected and the contour is highlighted so that the forklift and the luggage face each other and the forklift is in a suitable position when the fork is inserted. Informed support is provided.

Further, by displaying the line segment connecting the center of the positioning mark and the center of the reference mark, the driver is assisted in positioning the fork and the insertion port on the XZ surface.

Further, after the distance between the center of the positioning mark and the center of the reference mark is within a predetermined range, that is, after the fork and the insertion port are positioned on the XZ surface, the forklift is operated so as to approach the pallet. As a result, the fork is positioned in the Y-axis direction. The amount of fork inserted into the pallet is reflected in the difference in radius between the positioning mark and the reference mark, each of which has a highlighted outline. Since the size of the reference mark displayed on the displayed image does not change, only the size of the contour of the positioning mark changes depending on the distance between the pallet and the camera, the change in the size of the contour of the positioning mark and the positioning mark From the positional relationship with the reference mark, the amount of fork inserted into the pallet can be intuitively grasped. In this way, by superimposing the captured image and the reference mark on the display image, the driver is assisted in positioning the fork in the Y-axis direction.

This embodiment is particularly effective in cargo handling work at high places. Since it is difficult to visually accurately position the fork and the insertion port in the cargo handling work at a high place, positioning using the positioning mark and the reference mark as in the above embodiment is effective.

(Second Embodiment)
In the above-described first embodiment, one positioning mark is provided, but a plurality of positioning marks may be provided. For example, as shown in FIG. 7, four positioning marks 51 may be provided at the four corners of the rectangular surface of the luggage 40 along the side surface of the pallet 30 provided with the insertion port 32.

Also in this case, similarly to the above embodiment, the circular shape of the positioning mark 51 is detected, and the fork is positioned using the center coordinates and radii of the positioning mark and the reference mark, respectively. By providing the plurality of circular positioning marks 51 in this way, the fork positioning accuracy is improved.

(Third Embodiment)
In the above-described embodiment, the display image signal generation unit generates a display image in which the captured image and the reference mark 60 are superimposed, but in addition to the reference mark 60, the reference mark 85 before inserting the fork is restricted. A display image may be generated in which the reference mark 90 for is further superimposed on the captured image. Either the reference mark 85 before inserting the fork or the reference mark 90 for limiting may be provided.

As described in the above embodiment, by using the reference mark 60, it is possible to grasp whether the amount of the fork inserted into the pallet after the fork is inserted from the insertion port is sufficient or insufficient. it can.

The reference mark 85 before inserting the fork has a circular outer shape similar to the reference mark 60, and has a center at the same position as the center of the reference mark 60. The radius of the reference mark 85 before the fork is inserted is smaller than the radius of the reference mark 60, and the reference mark 85 between the forks is located inside the reference mark 60.

The reference mark 85 before inserting the fork is used when positioning the fork on the XZ surface before inserting the fork. When the contour of the positioning mark 50 overlaps with the reference mark 85 before the fork is inserted, it indicates that the tip of the fork 16 is in the state immediately before being inserted into the pallet 30.

With the contour 50B of the positioning mark 50 positioned within the reference mark 85 before inserting the fork, the forklift is set so that the distance between the center 50A of the positioning mark 50 and the center 60A of the reference mark 60 is within a predetermined range. By operating 10, the fork can be positioned on the XZ surface in a state where the tip of the fork 16 is not inserted into the pallet 30. Therefore, it is possible to prevent the tip of the fork 16 from colliding with the pallet 16 or the load 40 and causing the load to collapse when the fork 16 is positioned.

The restriction reference mark 90 has a circular outer shape similar to the reference mark 60, and has a center at the same position as the center of the reference mark 60. The radius of the reference mark 90 for restriction is larger than the radius of the reference mark 60. The restriction reference mark 90 is located so as to surround the reference mark 60.

The reference mark 90 for limiting indicates the limit of the insertion amount of the fork 16 after the fork is inserted. If the contour 50B of the positioning mark 50 becomes larger than the reference mark 90 for restriction, the fork 16 is inserted into the pallet 30 more than necessary, and the pallet 30 and the forklift 10 may collide with each other to damage the pallet 30. Is shown.

In this way, by operating the forklift 10 so that the positioning mark 50 does not become larger than the restriction reference mark 90, it is possible to prevent the pallet 30 and the forklift 10 from colliding with each other and prevent the pallet 30 from being damaged. it can.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications can be made.

For example, in the above embodiment, the information processing device 70 and the camera 22 are separately provided, but these may be provided integrally.

Further, in the above embodiment, the luggage is provided with a positioning sticker, but a positioning sticker may be provided on the side surface where the pallet insertion port is provided. If the pallet is provided with a positioning sticker, the camera is installed so that it is located approximately flush with the pair of forks.

Further, in the above embodiment, although the positioning seals of one type of size have been described as an example, positioning seals of a plurality of types may be used. In this case, reference marks having different sizes to be displayed on the display image are prepared in advance for each positioning sticker having different sizes. For example, different identifiers are assigned to the positioning sticker by printing or the like depending on the difference in size, and the information processing device determines the identifier by pattern recognition or the like from the captured image of the positioning sticker taken, and the reference corresponding to each identifier. The mark may be configured to be displayed on the display image.

Further, in the above embodiment, the forklift 10 is a manned mobile body that requires the driver to board, but may be an unmanned mobile body. In the case of an unmanned moving body, a monitor as a display device is provided in a separate room away from the work place where the forklift travels. The operator can remotely control the forklift while checking the display image on the monitor provided in the separate room.

1 ... Body 10 ... Forklift 15 ... Mast 16 ... Fork 22 ... Camera (imaging unit)
30 ... Palette 40 ... Luggage (carrying object)
50, 51 ... Positioning mark 50A ... Center of positioning mark 55 ... Line segment 60 ... Reference mark 60A ... Center of reference mark 70 ... Information processing device 71 ... Image acquisition unit 72 ... Display image signal generation unit 73 ... Circle detection unit 74 ... Position detection unit 75 ... Judgment unit 85 ... Reference mark before fork insertion 90 ... Reference mark for restriction R ... Radius of reference mark r ... Radius of positioning mark

Claims (8)

An image acquisition unit that acquires a photographed image of a pallet provided with a positioning mark or an object placed on the pallet, which is photographed by an image pickup unit provided so as to be interlocked with the raising and lowering of a fork of a forklift.
Information including a display image signal generation unit that generates a display image signal that is a display image in which the captured image acquired from the image acquisition unit and a reference mark having an outer shape similar to the outer shape of the positioning mark are superimposed. Processing equipment. The information processing device according to claim 1.
A position detection unit that detects the positions of the positioning mark and the reference mark in the display image, and
Based on the position detection result by the position detection unit, the distance between the center of the positioning mark and the center of the reference mark and the distance between the outer shape of the positioning mark and the outer shape of the reference mark are within predetermined ranges. An information processing device further provided with a determination unit for determining the presence or absence. The information processing device according to claim 1 or 2.
An information processing device having a circular outer shape of the positioning mark and the reference mark. The information processing device according to claim 3.
A circular detection unit for detecting the circular shape of the positioning mark is further provided.
When the circular shape of the positioning mark is detected by the circle detection unit, the display image signal generation unit generates a display image signal that becomes a display image in which the outer shape of the positioning mark of the display image is highlighted. Information processing device. The information processing apparatus according to any one of claims 1 to 4.
The display image signal generation unit is an information processing device that generates a display image signal that becomes a display image in which a line segment connecting the center of the positioning mark and the center of the reference mark is further displayed on the display image. The information processing apparatus according to any one of claims 1 to 5.
The display image signal generation unit has an outer shape similar to the captured image, the reference mark, and the outer shape of the reference mark, is located inside the reference mark, and is centered at the same position as the center of the reference mark. An information processing device that generates a display image signal that is a display image in which a reference mark before fork insertion is superimposed. The information processing apparatus according to any one of claims 1 to 6.
The display image signal generation unit has an outer shape similar to the captured image, the reference mark, and the outer shape of the reference mark, is located so as to surround the reference mark, and is located at the same position as the center of the reference mark. An information processing device that generates a display image signal that is a display image in which a reference mark for limiting the distance between the forklift and the pallet, which has a center, is superimposed. A photographed image of a pallet provided with a positioning mark or an object placed on the pallet, taken by a camera provided so as to be interlocked with the raising and lowering of the fork of the forklift, is acquired.
A display image signal is generated, which is a display image in which the captured image and a reference mark having an outer shape similar to the outer shape of the positioning mark are superimposed.
The positions of the positioning mark and the reference mark in the displayed image are detected, and the positions are detected.
Based on the position detection result, it is determined whether or not the distance between the center of the positioning mark and the center of the reference mark is within a predetermined range.
An information processing method for determining whether or not the distance between the outer shape of the positioning mark and the outer shape of the reference mark is within a predetermined range based on the position detection result.

Images