JP7342429B2 - Machine learning data collection method and camera system - Google Patents

Description

The present invention relates to a data collection method for machine learning and a camera system.

A drive recorder mounted on a vehicle is known as a device for recording the operation of a machine (for example, see Patent Document 1).
The drive recorder disclosed in Patent Document 1 captures an image in front of a vehicle and records it on a recording medium. Furthermore, when the drive recorder detects an abnormal event, it extracts a video of a predetermined length of time including the time when the abnormal event was detected from the recorded video.

JP 2018-41123 Publication

A camera system installed in a transport device or a machine tool has the function of recording images before and after a machine signal is input, similar to a drive recorder.
On the other hand, in recent years, artificial intelligence (AI) has been utilized in various fields, and for example, its use in detecting abnormalities and their precursors in transport devices and machine tools is being considered. Specifically, they have created an inference engine that automatically determines anomalies by learning images taken from captured footage at the moment an anomaly occurs.
Of the captured videos, the data used for learning is extracted based on human judgment. Therefore, collecting data for machine learning was time-consuming and inefficient.

An object of the present invention is to efficiently extract machine learning data from videos captured by a camera system.

A plurality of aspects will be described below as means for solving the problem. These aspects can be arbitrarily combined as necessary.

A method for collecting data for machine learning according to an aspect of the present invention is a method for collecting data for machine learning in a camera system equipped with a camera that photographs the operation of a machine. This method comprises the following steps.
◎Photographed video storage step that stores photographed video data of machine operations.
◎Event recognition step that recognizes that a predetermined event has occurred in the machine.
◎First video extraction step of extracting first video data having a length of the first period including the event recognition time from the captured video data.
◎Second video data extraction step of extracting second video data included in the captured video data and having a length of a second period including the event recognition time.
The second period is shorter than the first period.
The second video data is machine learning data for determining the occurrence of a predetermined event.

Note that recognizing that a predetermined event has occurred in the machine means, for example, receiving a signal indicating that a predetermined event has occurred in the machine.
Note that the timing of the step of extracting the first video data and the step of extracting the second video data is not limited. For example, the step of extracting the second video data may be performed before or after the step of collecting the first video data (immediately, after a predetermined period of time, etc.).

Generally, video data for machine learning is a video with a very short period (for example, several frames) necessary for machine learning. The reason why the period is short is that if the data includes unnecessary parts, the accuracy of learning will decrease. On the other hand, even when a person watches such a short video, it is difficult to understand how a predetermined event will occur.
In this method, first video data representing a relatively long period of video is extracted, so the operator can understand how a predetermined event occurs by viewing the video. Further, in this method, second video data representing a relatively short period of video can be used as machine learning data for determining the occurrence of a predetermined event. Here, the extraction of the second moving image data is performed automatically, which is efficient.

In the second video data extraction step, the second video data may be extracted from the first video data.
In this method, since the second video data is extracted from the first video data, there is no need to store the original video data at the stage when the first video data is extracted.
Moreover, it becomes easy to extract a plurality of second moving image data.

In the second video data extraction step, a plurality of second video data may be extracted from the first video data.
With this method, when a plurality of events are recorded in the first moving image data, dedicated second moving image data can be provided to each of the plurality of inference engines. In this case, the plurality of second moving image data may be the same or different.

The signal may include event classification information indicating a classification of the predetermined event.
The method may further include determining a length of the second time period in response to the event classification information.
With this method, it is possible to extract second video data of an appropriate length depending on the type of predetermined event.

The signal may include operation content information indicating the operation content of the machine. For example, in the case of a transport device, the operation contents include transport, unloading, and loading. The action details may include information about the object, such as the type of luggage being carried.
The method may further include a length determining step of determining the length of the second period according to the action content information.
With this method, it is possible to extract second video data of an appropriate length depending on the operation content of the machine.

The signal may include event classification information indicating a classification of the predetermined event.
This method may further include an image area determining step of determining an image area of the second video data from the entire image area of the captured video according to the event classification information.
With this method, it is possible to extract second video data in an appropriate area depending on the type of predetermined event. In other words, only the areas necessary for machine learning can be targeted for the second video data. Therefore, information unnecessary for machine learning is not provided to machine learning.

The signal may include operation content information indicating the operation content of the machine.
This method may further include an image area determining step of determining an image area of the second video data from the entire image area of the captured video according to the action content information.
With this method, it is possible to extract second video data of an appropriate area depending on the operation content of the machine.

The predetermined event may be a machine malfunction. Abnormalities in the machine include abnormalities in the machine itself and abnormalities in the objects of the machine.

A plurality of pieces of second video data may be acquired by repeatedly performing the photographed video data storage step, the event recognition step, the first video data extraction step, and the second video data extraction step.
The method may further include an automatic classification step of automatically classifying the second video data into a plurality of groups.
With this method, the worker does not have to manually classify the second video data, so the workload is reduced.

In the automatic classification step, the second video data may be classified for each type of predetermined event.
With this method, an inference engine can be created for each type of predetermined event.

In the automatic classification step, the second video data may be classified according to the operation content of the machine.
With this method, an inference engine can be created for each type of machine operation.

There may be more than one machine.
A plurality of pieces of second video data may be acquired by performing a photographed video data storage step, an event recognition step, a first video data extraction step, and a second video data extraction step for each machine.
In the automatic classification step, the second video data may be classified for each machine.
This method allows you to create an inference engine for each machine.

A camera system may include multiple cameras.
In the automatic classification step, the second video data may be classified for each camera.
This method allows you to create an inference engine for each camera.

A camera system according to another aspect of the present invention includes a camera that photographs the operation of a machine, a storage unit that stores video data of the photographed operation of the machine, and an event that recognizes that a predetermined event has occurred in the machine. It includes a recognition unit and a controller.
The controller has a first extraction section and a second extraction section.
The first extraction unit extracts first video data indicating a first video having a length of a first period including an event recognition time at which an event is recognized, from the captured video data.
The second extraction unit extracts second video data that is included in the photographed video data and represents a second video having a length of a second period that includes the event recognition time.
The second period is shorter than the first period.
The second video data is machine learning data for determining the occurrence of a predetermined event.
Note that the locations of the controller and the storage section are not particularly limited.

The second extraction unit may extract the second video data from the first video data.
Since this system extracts the first video data representing a relatively long period of video, the operator can understand how a predetermined event occurs by viewing the video. Further, in this system, second video data representing a relatively short period of video can be used as machine learning data for determining the occurrence of a predetermined event. Here, the extraction of the second moving image data is performed automatically, which is efficient.

With the machine learning data collection method and camera system according to the present invention, machine learning data can be efficiently extracted from videos captured by the camera system.

FIG. 1 is a schematic plan view of the automated warehouse of the first embodiment. FIG. 2 is a view taken along arrow II-II in FIG. 1 and is a diagram for explaining a rack and a stacker crane. FIG. 3 is a partially enlarged view of FIG. 2 and is a diagram illustrating a lifting platform and a transfer device. FIG. 3 is a schematic plan view schematically showing the arrangement position and imaging range of a camera. A display screen showing the status of the transfer equipment and shelves. A display screen showing the status of the transfer equipment and shelves. FIG. 3 is a functional block diagram of an image processing controller and a camera controller. A display screen of the first video data with markings. 5 is a flowchart showing the control operation of the image processing controller when an abnormality occurs. 5 is a flowchart showing extraction control operations of the camera controller for extracting first video data and second video data. The schematic diagram which shows the relationship between photographed video data, 1st video data, and 2nd video data. FIG. 7 is a schematic diagram showing the relationship between photographed video data, first video data, and second video data according to the second embodiment. 12 is a flowchart showing the extraction control operation of the first video data and second video data of the camera controller according to the third embodiment. 12 is a flowchart showing the extraction control operation of the first video data and second video data of the camera controller according to the fourth embodiment. The figure which shows the relationship between the extracted 2nd moving image data and 1st moving image data of 4th Embodiment. 12 is a flowchart showing the extraction control operation of the first video data and second video data of the camera controller according to the fifth embodiment. FIG. 7 is a schematic diagram showing the relationship between photographed video data, first video data, and second video data according to the sixth embodiment. FIG. 7 is a functional block diagram of an image processing controller and a camera controller according to a seventh embodiment.

1. First Embodiment (1) Entire automated warehouse An automated warehouse 1 as a first embodiment of the present invention will be described using FIG. 1. FIG. 1 is a schematic plan view of an automated warehouse according to the first embodiment. In this embodiment, the left-right direction in FIG. 1 is the left-right direction (arrow Y) of the automated warehouse 1, and the up-down direction in FIG. 1 is the front-back direction (arrow X) of the automated warehouse 1.
The automated warehouse 1 mainly includes a pair of racks 2 and a stacker crane 3 (an example of a machine).

(2) Rack The pair of racks 2 are arranged to sandwich a stacker crane passage 5 extending in the left-right direction. The rack 2 includes a large number of front columns 7 arranged on the left and right at predetermined intervals, rear columns 9 arranged at a predetermined interval between the front columns 7 behind the front columns 7, and the front columns 7 and rear columns. 9 and a large number of load supporting members 11 provided therein. A shelf 13 is constituted by a pair of left and right cargo support members 11. As is clear from the figure, a load W can be placed on each shelf 13. Note that each load W is placed on a pallet P (see FIGS. 2 and 3) and is moved together with the pallet P. Note that a fork passing gap 15 is formed between the pair of left and right load supporting members 11, which allows vertical movement of a slide fork 29a, which will be described later.

(3) Stacker Crane The stacker crane 3 will be explained using FIGS. 2 and 3. FIG. 2 is a view taken along the line II-II in FIG. 1, and is a diagram for explaining the rack and the stacker crane. FIG. 3 is a partially enlarged view of FIG. 2, and is a diagram illustrating a lifting platform and a transfer device.
A pair of upper and lower running rails 21 are provided along the stacker crane passage 5, and the stacker crane 3 is guided by these running rails 21 so as to be movable left and right. The stacker crane 3 mainly includes a traveling truck 23 having a first mast 22A and a second mast 22B, which are a pair of masts 22, and a lifting platform 27 that is attached to the first mast 22A and the second mast 22B so as to be able to rise and fall freely. , and a transfer device 29 provided on the lifting table 27. The traveling truck 23 has traveling wheels 24 .

The transfer device 29 has a slide fork 29a that is slidably provided in the front-rear direction by a reciprocating mechanism (not shown). The slide fork 29a can transfer the load W to and from the shelves 13 on both sides in the front-rear direction.
Note that the lifting table 27 and the transfer device 29 are fixed to each other, and a part of the lifting table 27 may be considered as a part of the transfer device 29.

The elevating table 27 has elevating guide rollers 28 guided by the first mast 22A and the second mast 22B. A total of four elevating guide rollers 28 are in contact with one mast, one pair each on the upper and lower sides. More specifically, the pair of elevating guide rollers 28 are in contact with the inside portions of the mast 22 on both front and rear sides in the running direction.
In addition, in FIGS. 2 and 3, a pallet P and a load W are placed on the slide forks 29a of the lifting platform 27. In this embodiment, the load W has a structure in which 3x3x3 rectangular parallelepipeds are stacked without gaps. However, the number, shape, and stacking method of individual items of the item W are not limited to the above embodiment.
The lower ends of the first mast 22A and the second mast 22B are connected by a lower frame 25, and the upper ends are connected by an upper frame 26.

The stacker crane 3 includes a control panel 41, a travel motor 59, and a lifting motor 63. The control panel 41 is provided on the opposite side of the first mast 22A to the second mast 22B in the traveling direction. The travel motor 59 is provided on the first mast 22A. The lifting motor 63 is provided on the first mast 22A.
The control panel 41 includes electrical equipment such as an inverter, a converter, and a breaker for the traveling motor 59 and the lifting motor 63. The control panel 41 further includes a control board box (not shown). The control panel 41 has a frame that covers these electrical devices. The control panel 41 is connected to a power source (not shown), a travel motor 59, a lifting motor 63, a slide fork 29a, etc. via a power cable (not shown). The control panel 41 is further connected to the ground control panel, sensors, slide fork 29a, and control power source via a control cable and a communication interface.
The lifting motor 63 can drive the drum 64, as shown in FIG. A wire 40 extends from the drum 64. The wire 40 is passed around a pulley 44 provided on the upper frame 26 and further connected to the lifting platform 27.

(4) Camera As shown in FIGS. 2 to 4, the camera elevator platform 27 includes a first camera 51, a second camera 52, a third camera 53, and a fourth camera 54 (hereinafter referred to collectively as "4 cameras"). Cameras 51 to 54) are installed. FIG. 4 is a schematic plan view schematically showing the arrangement position and imaging range of the camera.
The four cameras 51 to 54 are so-called video cameras, and input captured images (still images and moving images) to a camera controller 90 (described later) as video signals. Note that in the above case, the imaging ranges of the first camera 51 and the second camera 52 are different from the imaging ranges of the third camera 53 and the fourth camera 54. However, the imaging ranges of the four cameras may all be different or may be all the same.

The four cameras 51 to 54 photograph the operation of the stacker crane 3 and create video data. Specifically, the four cameras 51 to 54 take images of the load W placed on the lifting platform 27 and the load W placed on the shelf 13.
The four cameras 51 to 54 are attached to the lifting platform 27 via a frame 45, as shown in FIG. The frame 45 is a pair of members extending in the front-rear direction on both sides of the lifting table 27, that is, the transfer device 29 in the left-right direction. The four cameras 51 to 54 are fixed to both ends of each frame 45 in the front and rear directions. Note that the camera may be attached to a member other than the frame.

The first camera 51 and the second camera 52 are fixed at positions where they can simultaneously image the end of the load W in the front-rear direction and the transfer destination shelf 13 on the end side in the front-rear direction.
The first camera 51 and the second camera 52 are provided on the transfer device 29 above the upper end of the load W placed on the transfer device 29 and outside the load W in the left-right direction. The first camera 51 and the second camera 52 face diagonally downward. In this case, by using the camera to image the load from above, the collapse of the load can be detected regardless of the height of the load W.
As described above, the first camera 51 and the second camera 52 are provided on both sides of the transfer device 29 in the front and rear directions, respectively. Therefore, it is possible to detect an abnormality in the appearance of the load W placed on the transfer device 29 and an incoming item in both transfer directions.

The first camera 51 and the second camera 52 will be explained in detail using FIG. 4.
The first camera 51 is provided on the lower right side of FIG. 4 in a plan view, and is located at an end 31 in the front-rear direction of the load W loaded on the transfer device 29 and at a transfer destination on the side of the end 31 in the front-rear direction. It is provided at a position where the shelves 13 can be imaged at the same time. Note that even if the load W is placed on the transfer device 29, the first camera 51 can detect the first loaded item on the shelf 13 at the transfer destination. This is because, when viewed from the first camera 51, the placement position of the transfer device 29 and the transfer destination shelf 13 are aligned with each other with a shift in the front-rear direction.

The second camera 52 is provided on the upper left side in FIG. It is located at a position where it can be imaged. Note that even if the load W is placed on the transfer device 29, the second camera 52 can detect the first loaded item on the shelf 13 at the transfer destination. This is because, when viewed from the second camera 52, the placement position of the transfer device 29 and the transfer destination shelf 13 are aligned with each other with a shift in the front-rear direction.
Note that the shelf 13 targeted by the first camera 51 and the shelf 13 targeted by the second camera 52 are different shelves.

The third camera 53 and the fourth camera 54 are provided in the transfer device 29 and are located at positions where they can image the left and right ends (that is, the end faces) of the load W placed on the transfer device 29. It will be done.
The third camera 53 and the fourth camera 54 are provided on the transfer device 29 above the upper end of the load W placed on the transfer device 29 and outside the load W in the front-rear direction. The third camera 53 and the fourth camera 54 face diagonally downward.

The third camera 53 and the fourth camera 54 will be explained in detail using FIG. 4.
The third camera 53 is provided on the lower left side of FIG. 4, and is provided at a position where it can image the left-right end portion 33 of the load W loaded on the transfer device 29. Note that the third camera 53 can detect the first-loaded item on the transfer destination shelf 13 if the load W is not placed on the transfer device 29; however, if the load W is placed in the transfer device 29, , it is not possible to detect the first item on the shelf 13 at the transfer destination. This is because, when viewed from the third camera 53, the placement position of the transfer device 29 and the transfer destination shelf 13 overlap.

The fourth camera 54 is provided on the upper right side of FIG. 4, and is provided at a position where it can image the end portion 34 in the left and right direction of the load W loaded on the transfer device 29. Note that if the load W is placed on the transfer device 29, the fourth camera 54 can detect the first-loaded item on the transfer destination shelf 13; however, if the load W is placed in the transfer device 29, , it is not possible to detect the first item on the shelf 13 at the transfer destination. This is because, when viewed from the fourth camera 54, the placement position of the transfer device 29 and the shelf 13 as the transfer destination overlap.
Note that the shelf 13 targeted by the third camera 53 and the shelf 13 targeted by the fourth camera 54 are different shelves. The shelf 13 targeted by the third camera 53 is placed on the upper side of the figure in FIG. 4, and the shelf 13 targeted by the fourth camera 54 is placed on the lower side of the figure in FIG. It is facing the arrow X).

Examples of images from the four cameras 51 to 54 will be explained using FIGS. 5 and 6. 5 and 6 are display screens showing one state of the transfer device and the shelf.
In FIGS. 5 and 6, images captured by four cameras 51 to 54 are displayed on the screen of the display device 89. The upper right part of each figure is the image of the first camera 51, the lower left part is the image of the second camera 52, the lower right part is the image of the third camera 53, and the upper left part is the image of the fourth camera 54. be.
In FIG. 5, no load W is placed on the transfer device 29, and no load W is placed on the shelves 13 on either side in the front-rear direction. Therefore, the load supporting member 11 of the shelf 13 is visible in all images. Note that the part surrounded by broken line A is one shelf 13, and the part surrounded by broken line B is the other shelf.

In FIG. 6, a load W1 is placed on the transfer device 29, and a load W2 and a load W3 are also placed on the shelves 13 on both sides in the front-rear direction (that is, there are first-order items). Therefore, the presence of the load W2 and the state of the end 31 of the load W1 can be confirmed by the image taken by the first camera 51, the presence of the load W3 and the end 32 of the load W1 can be confirmed by the image taken by the second camera 52, and the third The state of the end portion 33 of the load W1 can be confirmed by the image taken by the camera 53, and the condition of the end portion 34 of the load W1 can be confirmed by the image taken by the fourth camera 54.
Furthermore, in FIG. 6, the cargo support member 11 of the shelf 13 is not reflected in the image taken by the first camera 51 and the image taken by the second camera 52. Therefore, the image controller 70 (described later) can grasp that the load W2 and the load W3 are placed on the shelves 13 on both sides in the front-rear direction. However, in this case, the state of the shelf 13 is not reflected in the image taken by the third camera 53 and the image taken by the fourth camera 54.

(5) Control configuration of camera system The control configuration of camera system 100 will be explained using FIG. 7. FIG. 7 is a functional block diagram of the image processing controller and camera controller.
The camera system 100 includes a transport controller 50, an image processing controller 80, and a camera controller 90.

Each controller is a computer having a processor (e.g., CPU), a storage device (e.g., ROM, RAM, HDD, SSD, etc.), and various interfaces (e.g., A/D converter, D/A converter, communication interface, etc.). It is a system. Each controller performs various control operations by executing a program stored in a storage unit (corresponding to part or all of the storage area of the storage device).
Each controller may be composed of a single processor, or may be composed of a plurality of independent processors for each control.
A part or all of the functions of each element of each controller may be realized as a program executable by a computer system that constitutes the control section. In addition, a part of the functions of each element of both controllers may be configured by a custom IC.
Although not shown, each controller is connected to a sensor for detecting the size, shape, and position of an object, a sensor and switch for detecting the status of each device, and an information input device.
Note that each controller is not limited to being mounted on the stacker crane 3, and may be arranged on the ground side in an electrically connected state.

The transport controller 50 has a function of controlling the driving of the traveling wheels 24 of the traveling trolley 23 and is connected to a traveling section 73 including a traveling motor 59. The transport controller 50 has a function of vertically moving the lifting platform 27 along the first mast 22A and the second mast 22B, and is connected to a lifting section 74 including a lifting motor 63. The transport controller 50 has a function of moving the slide fork 29a in the front-back direction, and is connected to a transfer section 75 including a transfer motor (not shown).

The image processing controller 80 collects data for machine learning and determines abnormalities using an inference engine constructed from the data for machine learning.
The image processing controller 80 has a machine learning section 81. The machine learning unit 81 performs machine learning based on second video data (described later), which is data for machine learning, and constructs an inference engine for abnormality determination.
The image processing controller 80 has an abnormality determination section 82. The abnormality determination unit 82 includes one or more abnormality determination inference engines, and performs abnormality determination on first video data (described later).

The image processing controller 80 has an image processing section 83. The image processing unit 83 performs image processing such as marking (described later) on the first moving image data determined to be abnormal. This allows the operator to know the type and cause of the abnormality when referring to the first video data.
The image processing controller 80 has a display control section 84. The display control unit 84 can display, for example, marked first moving image data on a display device 89 (described later).
Markings superimposed on the first video data will be explained using FIG. 8. FIG. 8 is a display screen of the marked first moving image data.
Marking C is displayed superimposed on the first video data. Marking C indicates, for example, a location where the load has been strained. Note that the marking may be characters or other figures.

The image processing controller 80 has a storage section 85. The storage unit 85 stores first video data and second video data as video data of the operation of the machine.
The image processing controller 80 has a communication section 87.

An input device 88 and a display device 89 are connected to the image processing controller 80 .
The input device 88 is a device for an operator to input data and commands to the image processing controller 80. The input device 88 is, for example, a keyboard, a mouse, or a touch panel.
The display device 89 is, for example, a liquid crystal display, an organic EL display, or the like. However, the type, number, and installation position of the displays are not limited to the above embodiments.

The camera controller 90 controls the four cameras 51 to 54 to take images.
Four cameras 51 to 54 are connected to the camera controller 90.

The camera controller 90 has a camera control section 91 that controls imaging by the four cameras 51 to 54.
The camera controller 90 has an image extraction section 92 consisting of image processing solution software for image processing.

The image extraction section 92 includes a first extraction section 97 and a second extraction section 98.
The first extraction unit 97 extracts first video data having a length of a first period including the reception time when the signal was received from the captured video data, and stores it in the storage unit 93.
The second extraction unit 98 extracts second video data having a length of a second period including the reception time from the first video data, and stores it in the storage unit 93. The second video data is machine learning data for determining the occurrence of a predetermined event (for example, a machine abnormality), and the second period is shorter than the first period.

The camera controller 90 has a storage section 93. The storage unit 93 stores image data (photographed video data, first video data, second video data, etc.) captured by the four cameras 51 to 54. Note that the storage unit may be realized by an image server.

The camera controller 90 has a communication section 95.
Camera controller 90 has an input device 96. The input device 96 is a device for an operator to input commands to the camera controller 90. Input device 96 is, for example, a remote control.

(6) Control Operation The control operation of the camera system 100 will be explained.
The control flowchart described below is an example, and each step can be omitted or replaced as necessary. Further, a plurality of steps may be executed simultaneously, or some or all of the steps may be executed overlappingly.
Furthermore, each block in the control flowchart is not limited to a single control operation, and can be replaced with a plurality of control operations expressed by a plurality of blocks.
Note that the operation of each device is the result of instructions from the control unit to each device, and these are expressed by each step of the software application.

(6-1) Control operation of the image processing controller when an abnormality occurs The control operation of the image processing controller 80 when an abnormality occurs will be explained using FIG. FIG. 9 is a flowchart showing the control operation of the image processing controller when an abnormality occurs.
In step S1, it is determined whether or not there is an abnormality in the device based on the photographed image. Specifically, the abnormality determination unit 82 of the image processing controller 80 makes the above determination. It should be noted that the abnormality of the apparatus is, for example, problems such as the load collapsing and the load being caught in FIG. 6.
In step S2, a signal is sent to the camera controller 90. Specifically, the communication unit 87 of the image processing controller 80 transmits a signal to the camera controller 90. The signal is received by the communication unit 95 of the camera controller 90 (corresponding to Yes in step S3 in FIG. 10).

(6-2) Extraction control operation of the first moving image data and second moving image data of the camera controller The extraction control operation of the first moving image data and the second moving image data of the camera controller 90 will be explained using FIG. FIG. 10 is a flowchart showing the extraction control operation of the camera controller for extracting the first video data and the second video data.
In step S3, it waits to receive a signal. Specifically, the communication unit 95 of the camera controller 90 waits for receiving a signal from the image processing controller 80 (corresponding to step S2 in FIG. 9). If a signal is received, the process moves to step S4.

In step S4, first video data is extracted and saved from the captured video data. Specifically, the first extraction section 97 of the image extraction section 92 extracts images before and after the signal input or for a predetermined number of frames from the captured video data, and stores them in the storage section 93 as first video data.
In step S5, the second video data is extracted and saved from the first video data. Specifically, the second extraction unit 98 of the image extraction unit 92 extracts images for a predetermined time or a predetermined number of frames from the first video data as learning data for machine learning, and stores them as second video data in the storage unit. Save to 93.
As described above, in this embodiment, the event recognition time at which it is recognized that a predetermined event has occurred in the machine is the time at which the communication unit 95 of the camera controller 90 receives a signal from the image processing controller 80.
Note that the timing of step S4 and step S5 is not limited. For example, step S5 may be executed immediately after step S4 or after a predetermined period of time has elapsed.

In the camera system 100 described above, the first video data representing a relatively long period of time is extracted, so a person can understand how a predetermined event occurs by viewing the video. Furthermore, in the camera system 100, second video data representing a relatively short period of video can be used as machine learning data for determining the occurrence of a predetermined event. Here, the extraction of the second moving image data is performed automatically, which is efficient.

The relationship between photographed video data, first video data, and second video data will be explained using FIG. 11. FIG. 11 is a schematic diagram showing the relationship between photographed video data, first video data, and second video data.
As is clear from the figure, since the second moving image data is extracted from the first moving image data, there is no need to store the original moving image data at the stage when the first moving image data is extracted. Moreover, it becomes easy to extract a plurality of second moving image data.

2. Second Embodiment In the first embodiment, one second moving image data is extracted for one signal, but a plurality of second moving image data may be extracted.
Such an example will be described as a second embodiment using FIG. 12. FIG. 12 is a schematic diagram showing the relationship between photographed video data, first video data, and second video data according to the second embodiment. Note that since the basic configuration and basic operation are the same as those of the first embodiment, the following description will focus on the differences.

In the step of extracting second video data, a plurality of second video data are extracted from the first video data. Specifically, as shown in FIG. 12, the second extraction unit 98 extracts a plurality of second moving image data from one piece of first moving image data.
In the above case, when a plurality of events are recorded in the first video data, dedicated second video data can be provided to each of the plurality of inference engines. In this case, the plurality of second moving image data may be the same or different. Note that the case of being different means that a part or all of the range of each second moving image data is out of alignment.

3. Third Embodiment Although the length of the second video data was not particularly limited in the first and second embodiments, the length of the second video data may be changed depending on conditions.
Such an example will be described as a third embodiment using FIG. 13. FIG. 13 is a flowchart showing the extraction control operation of the camera controller of the third embodiment for extracting the first video data and the second video data. Note that since the basic configuration and basic operation are the same as those of the first embodiment, the following description will focus on the differences.

Step S6 is inserted between step S3 and step S4 in FIG. 10 of the first embodiment. In step S6, the length of the second video data is determined. Specifically, the second extraction unit 98 determines the length of the second video data based on information included in the signal.
As a result, second video data having a time length (the number of frames before and after the occurrence of an abnormality) that is effective as data for machine learning is obtained. Therefore, unnecessary or inconvenient information for machine learning is not provided to machine learning.

As an example, the information included in the signal is event classification information indicating a classification of a predetermined event. In this case, the time length of the second video data is determined according to the event classification information. As a result, second video data of an appropriate length is extracted. For example, in the case of load shedding, a relatively short time length is selected, and in the case of load lifting, a relatively long time length is selected. This is because in the latter case, observation for a predetermined period of time is required to distinguish between load damage and label peeling. For example, in the case of load shedding, the event continues over a long period of time, but in the case of label peeling, the event only occurs momentarily because the label flaps and sticks out.

As an example, the information included in the signal is operation content information indicating the operation content of the machine. For example, in the case of the stacker crane 3, the operation contents include transporting, unloading, and loading a load. Note that the operation details may include information about the object such as the type of luggage being carried. In this case, the time length of the second video data is determined according to the action content information. As a result, second video data with an appropriate time length is extracted.
In addition, as a modification of this embodiment, the length of the second moving image data may be determined before receiving the signal, and in that case, the length information of the second moving image data is included in the signal.

4. Fourth Embodiment In the first to third embodiments, the area (range) of the second video data was not particularly limited, but the area of the second video data is a partial area that is a part of the entire area of the captured video. may be limited to.
Such an example will be described as a fourth embodiment using FIGS. 14 and 15. FIG. 14 is a flowchart showing the extraction control operation of the first video data and second video data of the camera controller according to the fourth embodiment. FIG. 15 is a diagram showing the relationship between the extracted second video data and first video data in the fourth embodiment. Note that since the basic configuration and basic operation are the same as those of the first embodiment, the following description will focus on the differences.
Step S7 is inserted between step S3 and step S4 in FIG. 10 of the first embodiment. In step S7, an extraction area for the second moving image data is determined. Specifically, the second extraction unit 98 determines the extraction area of the second video data based on the information included in the signal.
In FIG. 15, a plurality of extraction areas D of second moving image data are shown. The extraction area D here is an image area from which it is possible to know whether or not the load W is protruding. The extraction area D is the minimum necessary area, and therefore has an elongated frame shape. However, the shape and number of extraction areas are not particularly limited.
The plurality of extraction areas D may be handled as individual data, or may be handled as part of specific second video data.

In the above case, it is possible to extract second video data in an appropriate area. In other words, the second video data consisting of only the area necessary for machine learning can be targeted for machine learning. Therefore, unnecessary or inconvenient information for machine learning is not provided to machine learning.

As an example, the information included in the signal is event classification information indicating a classification of a predetermined event.
In this case, the area of the second video data is determined according to the event classification information. As a result, an appropriate area can be extracted as the second video data.
As an example, the information included in the signal is operation content information indicating the operation content of the machine. In this case, the area of the second video data is determined according to the action content information. As a result, an appropriate area can be extracted as the second video data.
Note that as a modification of the present embodiment, the area of the second moving image data may be determined before receiving the signal, and in this case, the area information of the second moving image data is included in the signal.

5. Fifth Embodiment In the first to fourth embodiments, the method of providing the second video data to the machine learning unit 81 was not limited, but the method of providing the second video data to the machine learning unit 81 was devised. It's okay.
Such an example will be described as a fifth embodiment using FIG. 16. FIG. 16 is a flowchart showing the extraction control operation of the first moving image data and the second moving image data of the camera controller of the fifth embodiment. Note that since the basic configuration and basic operation are the same as those of the first embodiment, the following description will focus on the differences.

Step S8 is executed after step S5 in FIG. 10 of the first embodiment. In step S8, the second video data is automatically classified into one of a plurality of groups. Specifically, the second extraction unit 98 classifies the second video data into predetermined categories. This creates an inference engine for each classified category.
In this embodiment, the machine learning data can be automatically collected and classified, so there is no need for an operator to manually classify the machine learning data. Therefore, the workload is reduced.

As one classification criterion, the second video data is classified for each predetermined event. In this case, an inference engine can be created for each type of predetermined event.
As one classification criterion, the second video data is classified for each operation content of the machine. In this case, an inference engine can be created for each type of machine operation.

As one classification standard, the second video data is classified for each machine. In this case, an inference engine can be created for each machine.
As one classification criterion, the second video data is classified for each camera. In this case, an inference engine can be created for each camera.

6. Sixth Embodiment In the first to fifth embodiments, the second video data is extracted from the first video data, but it may be extracted from the original captured video data.
Such an example will be described as a sixth embodiment using FIG. 17. FIG. 17 is a schematic diagram showing the relationship between photographed video data, first video data, and second video data according to the sixth embodiment. Note that since the basic configuration and basic operation are the same as those of the first embodiment, the following description will focus on the differences.
In the step of extracting the second moving image data, the second moving image data is extracted from the photographed moving image data. In this case, the order of extraction of the first video data and extraction of the second video data is not limited.

7. Seventh Embodiment In the first to sixth embodiments, the learning section and the abnormality determination section are set as a set as shown in FIG. 7, but the present invention is not limited to such an embodiment, and the learning section and the abnormality determination section are The parts may be provided separately.

Such an example will be described as a seventh embodiment using FIG. 18. FIG. 18 is a functional block diagram of an image processing controller and a camera controller according to the seventh embodiment. Note that descriptions of common points with the first embodiment will be omitted.

The camera system 100A includes a transport controller 50A, an image processing controller 80A, a camera controller 90A, and a machine learning controller 101.
The image processing controller 80A stores the first moving image data and abnormality information received from the camera controller 90A, and outputs them for display.

The image processing controller 80A has an image processing section 83A. The image processing unit 83A performs image processing such as marking on the first moving image data determined to be abnormal. This allows the operator to know the type and cause of the abnormality when referring to the first video data.
The image processing controller 80A has a display control section 84A. The display control unit 84A can display, for example, marked first video data on the display device 89A.

The image processing controller 80A has a storage section 85A. First video data is stored in the storage unit 85A as video data of the operation of the machine.
The image processing controller 80A has a communication section 87A.
An input device 88A and a display device 89A are connected to the image processing controller 80A.

The camera controller 90A controls the four cameras 51 to 54 to take images.
Four cameras 51 to 54 are connected to the camera controller 90A.

The camera controller 90A includes a camera control section 91A that controls imaging by the four cameras 51-54.
The camera controller 90A has an image extraction section 92A made of image processing solution software for image processing.

The image extraction section 92A includes a first extraction section 97A and a second extraction section 98A.
The first extraction unit 97A extracts first video data having a length of a first period including the reception time when the signal was received from the captured video data, and stores it in the storage unit 93A.
The second extraction unit 98A extracts second video data having a length of a second period including the reception time from the first video data, and stores it in the storage unit 93A. The second video data is machine learning data for determining the occurrence of a predetermined event (for example, a machine abnormality), and the second period is shorter than the first period.

The camera controller 90A has a storage section 93A. The storage unit 93 stores image data (photographed video data, first video data, second video data, etc.) captured by the four cameras 51 to 54. Note that the storage unit may be realized by an image server.

The camera controller 90A has a communication section 95A.
The camera controller 90A has an abnormality determination section 82A. The abnormality determination unit 82A includes one or more abnormality determination inference engines, and performs abnormality determination on the first video data. Note that the abnormality determination unit 82A holds a plurality of inference engine data for each abnormality content.

The machine learning controller 101 has a machine learning section 81A. The machine learning unit 81A performs machine learning based on the second video data, which is data for machine learning, and constructs an inference engine for abnormality determination.
The machine learning controller 101 has a storage section 85B.
The machine learning controller 101 has a communication section 87B.

When the abnormality determination unit 82A detects an abnormality in the camera controller 90A, the first extraction unit 97A and the second extraction unit 98A extract the first video data and the second video data, respectively. In this embodiment, the event recognition time at which it is recognized that a predetermined event has occurred in the machine is the time at which the abnormality determination unit 82A detects an abnormality. The first video data is sent to the image processing controller 80A together with the abnormality information. The second video data is transmitted to the machine learning controller 101 together with the abnormality information.
When the camera controller 90A receives new inference engine data from the machine learning controller 101, it updates the inference engine of the abnormality determination unit 82A.

The machine learning controller 101 receives the second video data and abnormality information from the camera controller 90A. The machine learning unit 81A performs reinforcement learning of the inference engine. The machine learning unit 81A holds a learning model for each abnormality content. If new inference engine data is generated as a result of learning by the machine learning unit 81A, the machine learning controller 101 transmits the new inference engine data to the camera controller 90A.

In this embodiment, one machine learning controller 101 is placed, for example, on a warehouse network, and distributes the generated inference engine to a camera unit (a set of camera and camera controller) provided for each of a plurality of stacker cranes. can. In this case, each camera unit performs abnormality determination and collection of learning videos.

8. Common Items of the Embodiments The first to seventh embodiments have the following configurations and functions in common.
The machine learning data collection method of this embodiment is a method of collecting machine learning data in a camera system (for example, camera system 100) that includes a camera that photographs the operation of a machine (for example, stacker crane 3). This method comprises the following steps.
◎Photographed video data storage step that stores photographed video data of machine operations.
◎Event recognition step (for example, Yes in step S3) for recognizing that a predetermined event (for example, an abnormality in the stacker crane 3) has occurred in the machine.
◎A first video extraction step (for example, step S4) of extracting first video data having a length of a first period including the event recognition time at which the event was recognized from the captured video data.
◎A second moving image extraction step (for example, step S5) of extracting second moving image data included in the photographed moving image data and having a length of a second period including the event recognition time.
The second period is shorter than the first period.
The second video data is machine learning data for determining the occurrence of a predetermined event.
Since this method extracts first video data representing a relatively long period of video, a person can understand how a predetermined event occurs by viewing the video. Further, in this method, second video data representing a relatively short period of video can be used as machine learning data for determining the occurrence of a predetermined event. Here, the extraction of the second moving image data is performed automatically, which is efficient.

9. Other Embodiments Although a plurality of embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various changes can be made without departing from the gist of the invention. In particular, the multiple embodiments and modifications described in this specification can be arbitrarily combined as necessary.

In the first embodiment, the machine is a stacker crane, but the type of machine is not limited. For example, the machine may be a shuttle trolley that travels between racks in a warehouse, or a tracked trolley that travels along ground or ceiling rails. Further, the machine may be a machine other than the transport device.

The number of cameras is not particularly limited. The number may be 1 to 3, or 5 or more.
Images from all cameras may not be displayed on the display device at the same time. For example, only images from one camera may be displayed and may be switchable.

The present invention can be widely applied to data collection methods and camera systems for machine learning.

1: Automatic warehouse 2: Rack 3: Stacker crane 5: Stacker crane passage 7: Front support 9: Rear support 11: Load support member 13: Shelf 15: Fork passage gap 21: Traveling rail 22: Mast 22A: First mast 22B: Second mast 23: Traveling cart 24: Traveling wheel 25: Lower frame 26: Upper frame 27: Elevating platform 28: Elevating guide roller 29: Transfer device 29a: Slide fork 31: End portion 32: End portion 33: End part 34 : End part 40 : Wire 41 : Control panel 44 : Pulley 45 : Frame 50 : Transport controller 51 : First camera 52 : Second camera 53 : Third camera 54 : Fourth camera 59 : Travel motor 63 : Lifting Motor 64: Drum 73: Traveling section 74: Lifting section 75: Transfer section 80: Image processing controller 81: Machine learning section 82: Abnormality determination section 83: Image processing section 84: Display control section 85: Storage section 87: Communication section 88: Input device 89: Display device 90: Camera controller 91: Camera control section 92: Image processing section 93: Storage section 95: Communication section 96: Input device 97: First extraction section 98: Second extraction section 100: Camera system

Claims (17)

A method for collecting data for machine learning in a camera system equipped with a camera that photographs the operation of a machine, the method comprising:
a photographed video data storage step of storing photographed video data of the operation of the machine;
an event recognition step of recognizing that a predetermined event has occurred in the machine based on the photographed image and outputting an abnormal signal ;
a first video data extraction step of extracting first video data having a length of a first period including the time of the event recognition from the captured video data;
a second video data extraction step of extracting second video data included in the captured video data and having a length of a second period including the time of the event recognition;
a length determining step of determining the length of the second period according to event classification information indicating a classification of the predetermined event included in the abnormal signal;
Equipped with
The second period is shorter than the first period,
The second moving image data is machine learning data for determining occurrence of the predetermined event. A method for collecting data for machine learning in a camera system equipped with a camera that photographs the operation of a machine, the method comprising:
a photographed video data storage step of storing photographed video data of the operation of the machine;
an event recognition step of recognizing that a predetermined event has occurred in the machine based on the photographed image and outputting an abnormal signal;
a first video data extraction step of extracting first video data having a length of a first period including the time of the event recognition from the captured video data;
a second video data extraction step of extracting second video data included in the captured video data and having a length of a second period including the time of the event recognition;
a length determining step of determining the length of the second period according to operation content information indicating the operation content of the machine included in the abnormal signal;
Equipped with
The second period is shorter than the first period,
The second moving image data is machine learning data for determining occurrence of the predetermined event. A method for collecting data for machine learning in a camera system equipped with a camera that photographs the operation of a machine, the method comprising:
a photographed video data storage step of storing photographed video data of the operation of the machine;
an event recognition step of recognizing that a predetermined event has occurred in the machine based on the photographed image and outputting an abnormal signal;
a first video data extraction step of extracting first video data having a length of a first period including the time of the event recognition from the captured video data;
a second video data extraction step of extracting second video data included in the captured video data and having a length of a second period including the time of the event recognition;
an image area determining step of determining an image area of the second video data from the entire image area of the captured video according to event classification information indicating a classification of the predetermined event included in the abnormal signal;
Equipped with
The second period is shorter than the first period,
The second moving image data is machine learning data for determining occurrence of the predetermined event. A method for collecting data for machine learning in a camera system equipped with a camera that photographs the operation of a machine, the method comprising:
a photographed video data storage step of storing photographed video data of the operation of the machine;
an event recognition step of recognizing that a predetermined event has occurred in the machine based on the photographed image and outputting an abnormal signal;
a first video data extraction step of extracting first video data having a length of a first period including the time of the event recognition from the captured video data;
a second video data extraction step of extracting second video data included in the captured video data and having a length of a second period including the time of the event recognition;
an image area determining step of determining an image area of the second video data from the entire image area of the captured video according to operation content information indicating the operation content of the machine included in the abnormal signal;
Equipped with
The second period is shorter than the first period,
The second moving image data is machine learning data for determining occurrence of the predetermined event. 5. The machine learning data collection method according to claim 1 , wherein in the second video data extraction step, the second video data is extracted from the first video data. 6. The machine learning data collection method according to claim 5 , wherein in the second video data extraction step, a plurality of the second video data are extracted from the first video data. The data collection method for machine learning according to any one of claims 1 to 6 , wherein the predetermined event is a machine abnormality. A plurality of second video data is obtained by repeatedly performing the photographed video data storage step, the event recognition step, the first video data extraction step, and the second video data extraction step,
The data collection method for machine learning according to any one of claims 1 to 7 , further comprising an automatic classification step of automatically classifying the second video data into a plurality of groups. 9. The machine learning data collection method according to claim 8 , wherein in the automatic classification step, the second video data is classified for each type of the predetermined event. 9. The machine learning data collection method according to claim 8 , wherein in the automatic classification step, the second video data is classified for each operation content of the machine. There are multiple machines;
By executing the photographed video data storage step, the event recognition step, the first video data extraction step, and the second video data extraction step for each machine, a plurality of the second video data are obtained,
9. The machine learning data collection method according to claim 8 , wherein in the automatic classification step, the second video data is classified for each machine. The camera system includes a plurality of the cameras,
9. The machine learning data collection method according to claim 8 , wherein in the automatic classification step, the second video data is classified for each camera. A camera that records the operation of the machine,
a storage unit that stores photographed video data of the operation of the machine;
an event recognition unit that recognizes that a predetermined event has occurred in the machine based on the captured image and outputs an abnormal signal;
comprising a controller;
The controller includes:
a first extraction unit that extracts first video data indicating a first video having a length of a first period including an event recognition time at which the event was recognized from the captured video data;
a second extraction unit that extracts second video data included in the photographed video data, the second video data representing a second video having a second period including the event recognition time; death,
The second period is shorter than the first period,
The second video data is machine learning data for determining the occurrence of the predetermined event,
The abnormal signal includes event classification information indicating a classification of the predetermined event,
the second extraction unit determines the length of the second period according to the event classification information;
camera system. A camera that records the operation of the machine,
a storage unit that stores photographed video data of the operation of the machine;
an event recognition unit that recognizes that a predetermined event has occurred in the machine based on the captured image and outputs an abnormal signal;
comprising a controller;
The controller includes:
a first extraction unit that extracts first video data indicating a first video having a length of a first period including an event recognition time at which the event was recognized from the captured video data;
a second extraction unit that extracts second video data included in the photographed video data, the second video data representing a second video having a second period including the event recognition time; death,
The second period is shorter than the first period,
The second video data is machine learning data for determining the occurrence of the predetermined event,
The abnormal signal includes operation content information indicating the operation content of the machine,
the second extraction unit determines the length of the second period according to the operation content information;
camera system. A camera that records the operation of the machine,
a storage unit that stores photographed video data of the operation of the machine;
an event recognition unit that recognizes that a predetermined event has occurred in the machine based on the captured image and outputs an abnormal signal;
comprising a controller;
The controller includes:
a first extraction unit that extracts first video data indicating a first video having a length of a first period including an event recognition time at which the event was recognized from the captured video data;
a second extraction unit that extracts second video data included in the photographed video data, the second video data representing a second video having a second period including the event recognition time; death,
The second period is shorter than the first period,
The second video data is machine learning data for determining the occurrence of the predetermined event,
The abnormal signal includes event classification information indicating a classification of the predetermined event,
The second extraction unit determines an image area of the second video data from the entire image area of the captured video according to the event classification information.
camera system. A camera that records the operation of the machine,
a storage unit that stores photographed video data of the operation of the machine;
an event recognition unit that recognizes that a predetermined event has occurred in the machine based on the captured image and outputs an abnormal signal;
comprising a controller;
The controller includes:
a first extraction unit that extracts first video data indicating a first video having a length of a first period including an event recognition time at which the event was recognized from the captured video data;
a second extraction unit that extracts second video data included in the photographed video data, the second video data representing a second video having a second period including the event recognition time; death,
The second period is shorter than the first period,
The second video data is machine learning data for determining the occurrence of the predetermined event,
The abnormal signal includes operation content information indicating the operation content of the machine,
The second extraction unit determines an image area of the second video data from the entire image area of the captured video according to the action content information.
camera system. 17. The camera system according to claim 13, wherein the second extraction unit extracts the second video data from the first video data.

Images