JP2022187508A - Information processing system, program, and information processing method - Google Patents

Abstract

To provide an information processing system, a program, and an information processing method which allow for reduction in load on a server and delay time due to reduction in data volume by inputting only data extracted by automatic recognition in an image collection device to the server.SOLUTION: In the information processing system in which a plurality of imaging devices, a plurality of image collection devices, a server, and a carrier transporting device are connected via a network, a controller 43 of the server which implements determination of a transport sequence, method, and so on of a thing carried on a target object comprises: an acceptance unit 431 which accepts image data including "the target object received in field angles", sent from the image collection devices; an identification unit 432 which identifies the thing carried on the target object on the basis of information associated with the target object and information detectable from the image data; a specification unit 433 which determines a transport sequence and method for the carried thing; and a transmission unit 434 which notifies the carrier transporting device of the determined information.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing system, a program, and an information processing method.

When performing line work in a factory or loading and unloading cargo from a truck bed, it is common to confirm the target object by image recognition using an observation camera installed near the work arm. . This is because if there is an error in parts or the like in the line work, the quality of the product cannot be ensured, and if there is an error in carrying out or carrying in the cargo, it will be necessary to make major revisions to the subsequent schedule.

Patent Document 1 describes an image recognition authentication system that transmits image data obtained by photographing with a camera to a server, creates a data reduction image, and performs recognition and authentication processing of a subject based on the reduced image. It is

JP 2009-218695 A

However, if the server itself receives image data as disclosed in Patent Document 1, the cost and load of the server increase. Furthermore, since the data reduction image is compressed after the best shot is determined, the image used for image authentication in the moving image required in the line work area at the factory may be compressed, which may cause problems with authentication accuracy. can also be

In view of the above circumstances, the present invention provides an information processing system, a program, and an information processing system capable of reducing the load on the server and the delay time by reducing the amount of data by inputting only the data extracted by automatic recognition in the image collecting device into the server. An object is to provide an information processing method.

According to one aspect of the present invention, an information processing system, a program, and an information processing method are provided. This information processing system, program, and information processing method are configured to execute the following steps. In the extracting step, an image acquisition device recognizes a target object included in image data received by the image acquisition device, and extracts only image data in which the target object is included in the angle of view. The receiving step receives the extracted image data. In the identifying step, an object carried by the target object is identified based on information linked to the target object and information read from the extracted image data. The identifying step identifies the order and method of transportation of the objects carried by the target object. In the notification step, the identified information is notified to the carried object transporting device.

According to this, by inputting only the data extracted by the automatic recognition in the image collection device to the server, the load on the server and the delay time can be reduced by reducing the amount of data.

1 is an overall view of an information processing system according to an embodiment; FIG. 3 is a block diagram showing the hardware configuration of the image acquisition device 3; FIG. 4 is a block diagram showing the hardware configuration of the server 4; FIG. 3 is a block diagram showing functions realized by the image acquisition device 3 (control unit 33); FIG. 4 is a block diagram showing functions realized by the server 4 (control unit 43); FIG. It is a flowchart figure of this embodiment.

Embodiments of the present invention will be described below with reference to the drawings. Various features shown in the embodiments shown below can be combined with each other.

1. Hardware Configuration Section 1 describes the hardware configuration of this embodiment.

1.1 Information processing system 1
FIG. 1 is an overall view of an information processing system according to this embodiment. The information processing system 1 includes image capturing devices 2 (eg, image capturing devices 2-1, 2-2, . . . , 2-n) and image collecting devices 3 (eg, image capturing devices 3-1, 3- 2, . . . , 3-n), a server 4, and a carrying device 5, which are connected through a network. These components are further described. Here, the system exemplified by the information processing system 1 consists of one or more devices or components.

1.2 Imaging device 2
A plurality of imaging devices 2 are installed so as not to generate a blind spot in a predetermined space in order to facilitate identification of an object carried by the target object. In addition, it is desirable that the imaging device 2 be installed above or outside the lane or lane so as not to interfere with the movement of the object transporting device 5 . The imaging device 2 is, for example, a fixed-point observation camera. If you want to image a wide range and maintain a high image resolution, a 4K (number of pixels: 3840 × 2160) camera is preferable in order to reduce the amount of data transmitted to the image acquisition device 3. , 30 FPS (frames per second) or less, more preferably 5 FPS or less, specifically, for example, 5, 10, 15, 20, 25, 30 FPS, between any two of the numerical values exemplified here may be within the range of By lowering the frame rate, it is possible to suppress the amount of data transmitted to the image acquisition device 3, which will be described later, and thus the amount of data transmitted to the server 4 as well.

The imaging device 2 is connected via a network to a communication unit 31 of the image acquisition device 3, which will be described later, and is configured to be capable of transferring only image data in which the target object is included in the angle of view to the image acquisition device 3. .

Further, as the imaging device 2, a device capable of measuring not only visible light but also a band such as an ultraviolet region or an infrared region that cannot be perceived by humans may be adopted. By adopting such an imaging device 2, the information processing system 1 according to this embodiment can be implemented even in a dark field.

1.3 Image acquisition device 3
The image acquisition device 3 is provided corresponding to each image capturing device 2 and has the same number as the number associated with the image capturing device 2 . In addition, when one of the imaging devices 2 captures the target object in the angle of view, the image collecting device 3 is arranged so that the other imaging devices 2 can be linked in order to improve the ability to identify the object carried by the target object. may be provided corresponding to a plurality of imaging devices 2 . The number of image pickup devices 2 connected to one of the image acquisition devices 3 is not particularly specified as long as the amount of data transmitted from the image pickup devices 2 can be accommodated. The image collection device 3 is a terminal device typified by an edge device.

FIG. 2(a) is a block diagram showing the hardware configuration of the image acquisition device 3. As shown in FIG. The image acquisition device 3 has a communication unit 31 , a storage unit 32 , and a control unit 33 , and these components are electrically connected via a communication bus 30 inside the image acquisition device 3 . Each component will be further described.

The communication unit 31 preferably uses wired communication means such as USB, IEEE1394, Thunderbolt, wired LAN network communication, etc., but wireless LAN network communication, mobile communication such as LTE/3G, Bluetooth (registered trademark) communication, etc., can be used as necessary. may be included. That is, it is more preferable to implement as a set of these communication means. The image acquisition device 3 receives image data from the imaging device 2 and transmits image data in which the target object is included in the angle of view via the communication unit 31 . Details will be described later.

The storage unit 32 temporarily stores the image data received from the imaging device 2 described above. This may be, for example, a storage device such as a solid state drive (SSD) for storing various programs related to the image acquisition device 3 executed by the control unit 33, or a temporary storage device related to program calculation. It can be implemented as a memory such as a Random Access Memory (RAM) that stores information (arguments, arrays, etc.) required for the . A combination of these may also be used. In particular, "extract only the image data of the target object" described in step S103 of FIG. 4 is performed, and stored until "delete from memory etc." is performed in step S104 of FIG.

The control unit 33 processes and controls overall operations related to the image acquisition device 3 . The control unit 33 is, for example, a central processing unit (CPU) (not shown). The control unit 33 recognizes the target object by reading out a predetermined program stored in the storage unit 32, and realizes extraction of only image data in which the target object is accommodated in the angle of view. That is, information processing by software stored in the storage unit 32 can be specifically realized by the control unit 33 which is an example of hardware, and can be executed as a functional unit included in the control unit 33 . These are further detailed in the next section. Note that the control unit 33 is not limited to a single unit, and may be implemented so as to have a plurality of control units 33 for each function. A combination thereof may also be used.

1.4 Server 4
FIG. 2B is a block diagram showing the hardware configuration of the server 4. As shown in FIG. Like the image acquisition device 3 described above, the server 4 has a communication unit 41 , a storage unit 42 , and a control unit 43 . It is connected. Therefore, only parts different from those described above will be described for each component.

The server 4 receives image data in which the target object is included in the angle of view from the image acquisition device 3 via the communication unit 41 . It may be connected to the image pick-up device 2 in order to determine which part of all the image data transmitted from the image pick-up device 2 to the image acquisition device 3 corresponds to the image data. Details will be described later.

The storage unit 42 stores, as analysis target data, image data in which the target object is contained in the angle of view emitted by the image acquisition device 3 described above. In particular, the data to be analyzed is stored until "save data to be analyzed as teacher data" described in step S108 of FIG. become.

The control unit 43 reads out a predetermined program stored in the storage unit 42 to realize the "determination of transportation order, method, etc." of the objects carried by the target object described in step S106 of FIG.

1.5 Carrier 5
The carried object transporting device 5 receives information about the “transportation order/method” issued from the server 4, and executes the “transportation order/method”. In addition, when the object transporting device 5 is operated, a loudspeaker attached to the object transporting device 5 may be used to call attention by voice in consideration of safety and the like.

2. Functional Configuration This section describes the functional configuration of this embodiment. As described above, information processing by software stored in the storage units 32 and 42 is specifically realized by the control units 33 and 43, which are examples of hardware, so that the functional units included in the control units 33 and 43 can be run as

FIG. 3A is a block diagram showing functions realized by the image acquisition device 3 (control unit 33). Specifically, the image acquisition device 3 (control unit 33 ) includes a reception unit 331 , a recognition unit 332 , an extraction unit 333 and a transmission unit 334 .

The reception unit 331 is configured to receive various types of information. For example, the reception unit 331 receives all image data emitted from the imaging device 2 .

The recognition unit 332 is configured to automatically recognize whether or not the target object is included in the angle of view in all the image data transmitted from the imaging device 2 . For example, the recognition unit 332 calls a predetermined program stored in the storage unit 32, extracts (edge extracts) feature amounts from all image data transmitted from the imaging device 2, performs analysis, and automatically recognizes the target object. recognize.
It should be noted that the automatic recognition is more preferably machine learning based on training data that is stored in the server 4 as needed and sent to the image collection device 3, and the feature quantity extraction algorithm may be included as necessary. The machine learning algorithm is not particularly limited, and k-nearest neighbor method, logistic regression, support vector machine, neural network, topic model, Gaussian mixture model, etc. may be employed as appropriate.

The extraction unit 333 is configured to specify an image in which the target object is contained in the angle of view by the automatic recognition, and extract only the image. At this time, images that have not been extracted are automatically deleted from the memory or the like provided in the image acquisition device 3 . Further, when one of the image acquisition devices 3 is connected to a plurality of image capturing devices 2, the extraction of the images is performed in conjunction with each other.

The transmission unit 334 is configured to transmit the extracted image data to the server 4 . Note that when one of the image acquisition devices 3 is connected to a plurality of image pickup devices 2, the transmission unit 334 determines whether or not some images of the image pickup devices 2 are missing at a specific time. It is configured to transmit to the server 4 if there is no problem after confirming interlocking.

FIG. 3B is a block diagram showing functions realized by the server 4 (control unit 43). Specifically, the server 4 (control unit 43 ) includes a reception unit 431 , an analysis unit 432 , an extraction unit 433 and a transmission unit 434 .

The reception unit 431 is configured to receive various types of information. For example, the reception unit 431 receives image data “a target object is contained in the angle of view” emitted from the image acquisition device 3 .

The identification unit 432 identifies an object carried by the target object based on the information linked to the target object and the information read from the image data "the target object is contained in the angle of view". configured to For example, the information that can be read from the image data is the manufacturing number or model number printed on the back of the object carried by the target object, and the character number written on the container.

After identification by the identification unit 432, the identification unit 433 reads out a predetermined program in the storage unit 42, analyzes the "transportation order and method" of the object carried by the target object according to the user's purpose, configured to identify
In order to be able to flexibly cope with the ever-changing site, the notification of the analyzed and specified information to the transporting device 5 is set by the user after a predetermined time has elapsed after the end of the analysis. may

3. Information Processing Method In this section, the information processing method of the information processing system 1 described above will be described. This information processing method includes the following steps. In the extracting step, an image acquisition device recognizes a target object included in image data received by the image acquisition device, and extracts only image data in which the target object is included in the angle of view. The receiving step receives the extracted image data. In the identification step, an object carried by the target object is identified based on information linked to the target object and information read from the extracted image data. The identification step identifies the order and method of transportation of the objects carried by the target object.
In the notification step, the identified information is notified to the carried object transporting device.

FIG. 4 is a flow chart showing the flow of information processing executed by the information processing system 1. As shown in FIG. Hereinafter, description will be made along each flow of this flowchart.

When an operator, who is a user, tries to operate the object transporting device 5, the object transporting device 5 is started. do. The image acquisition device 3 is configured to receive all image data from the image capture device 2 (step S101).
The image acquisition device 3 automatically recognizes whether or not the image data received from the image pickup device 2 includes the image of the target object (step S102). As described above, the automatic recognition includes an edge extraction method for extracting feature amounts, machine learning using supervised data, and the like. Furthermore, what combined these may be used.

When the automatic recognition is performed, the image acquisition device 3 extracts only image data in which the target object is included in the angle of view (step S103), and transmits it to the server 4. FIG. On the other hand, for image data in which the target object is not accommodated in the angle of view, a storage device such as a solid state drive (SSD) or a random access memory (RAM) provided in the image acquisition device 3 is stored. is erased from the memory (step S104). The data deletion in step S104 is performed as soon as possible after step S103 because all the data that has leaked to step S103 is deleted and the image acquisition device 3 accepts all the image data from the image capturing device 2. preferable.

The image data in which the target object is included in the angle of view, which has been transmitted to the server 4, is taken in as an analysis target (step S105). After identifying the objects carried by the target object based on the information linked to the target object and the information that can be read from the extracted image data, analyze the order and method of transporting the objects. and identified (step S106). The analysis and identification are executed by reading out the predetermined program stored in the storage unit 42 of the server 4 as described above, and needless to say, various parameters are adjusted according to the purpose of the user.

After that, the specified information is transmitted to the carrying device 5 as a carrying device operation command (step S107). The carrying device 5 receives the operation command, and performs carry-out and carry-in. Then the series of flows ends. If the automatic image recognition in step S102 is machine learning using supervised data, the data to be analyzed is stored in the server 4 as supervised data after step S107 (step S108). The teaching data in step S102 may be set to be updated when the system is next started (step S109).

4. Others Regarding the information processing system 1 according to the present embodiment, the following aspects may be adopted.
In the information processing system 1, recognition of the target object in the image acquisition device 3 is based on learned data in which a plurality of types of target objects are learned in advance as teacher data.
The use of machine learning is more preferable because the image collection device 3 can realize highly accurate automatic recognition.

In the information processing system 1 using machine learning described above, the image acquisition device 3 further includes a learning step, in which image data in which the target object is accommodated in the angle of view is used as teacher data for the learning. create or update existing data.
As described in step S109 in FIG. 4, since the real data that has been successfully automatically recognized is updated as the teacher data, the precision of the automatic recognition of the image collecting device 3 is improved, which is more preferable.

In the information processing system 1, the information linked to the target object is a two-dimensional barcode. For example, the two-dimensional barcode is a QR code (registered trademark), which is quick to create and read, and not only is it less likely to cause a delay in issuing an operation command (step S107 in FIG. 4) to the object transporting device 5, This is preferable because it is also possible to improve the identification ability of the identification unit 432 of No. 4.

In the information processing system 1, the image data received by the image collecting device is image data captured from a fixed point so that a plurality of lanes fit within the angle of view, and the target object is the driver's seat of a land transportation machine. This is the upper part of the ceiling.
The heads of cars and trucks are square, which facilitates the edge extraction described above. Furthermore, the use of machine learning is preferable because more accurate automatic recognition can be achieved.

The image data described above is image data captured from a fixed point so that multiple lanes fit within the angle of view, and the target object is the ceiling above the driver's seat of a land transport machine. The carrier is a container, and the carrier is a container crane.
This is more preferable because the order in which the cargo is carried out by the container crane is required to be efficient, and the reduction in the amount of data in particular has the advantage of reducing the load on the server and reducing the delay time.

The information processing system 1 includes a camera and a server, the camera is configured to generate image data in which the target object is accommodated in an angle of view, and the server is configured to collect the image and the camera. It is connected to a device via a network and configured to receive only image data in which a target object is contained in an angle of view from the image acquisition device.
It is also possible to add the functions of the image capturing device 3 to the image capturing device 2, and even if there is no space for installing the image capturing device 3, the information processing system 1 is the image capturing device 2. A system in which the camera and the server 4 are integrated is preferable because the object of the present invention can be achieved.

Although the configuration of the information processing system 1 has been described in the above embodiment, a program that causes a computer to execute each step in the information processing system 1 may be provided.
This is more preferable because the object of the present invention can be achieved even if there is no space for the installation of the image acquisition device 3 described above.

Finally, while various embodiments of the invention have been described, these have been presented by way of example and are not intended to limit the scope of the invention. The novel embodiment can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. The embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

1: Image processing system 1
2: image capturing device 2
3: Image acquisition device 3
30: communication bus 30
31: communication unit 31
32: storage unit 32
33: control unit 33
331: reception unit 331
332: recognition unit 332
333: extraction unit 333
334: Transmitter 334
4: Server 4
40: communication bus 40
41: Communication unit 41
42: storage unit 42
43: control unit 43
431: reception unit 431
432: identification unit 432
433: Identification unit 433
434: Transmitter 434
5: Carrier 5

Claims (15)

An information processing system,
configured to perform each of the following steps,
In the extracting step, an image acquisition device recognizes a target object included in image data received by the image acquisition device, and extracts only image data in which the target object is contained in an angle of view;
The receiving step receives the extracted image data,
In the identification step, based on information linked to the target object and information read from the extracted image data, an object carried by the target object is identified;
the identifying step identifies the order and method of transportation of the objects carried by the target object;
In the notification step, the identified information is notified to the carrier transporting device. In the information processing system according to claim 1,
The recognition of the target object in the image acquisition device is based on learned data obtained by previously learning a plurality of types of the target object as teacher data. In the information processing system according to claim 2,
The image acquisition device further comprises a learning step,
In the learning step, the learned data is generated or updated using image data in which the target object is accommodated in an angle of view as teacher data. In the information processing system according to any one of claims 1 to 3,
The information linked to the target object is a two-dimensional barcode. In the information processing system according to any one of claims 1 to 4,
The image data received by the image collecting device is image data captured from a fixed point so that multiple lanes fit within the angle of view,
The target object is a ceiling part above the driver's seat of a land transport machine In the information processing system according to claim 5,
the carrier is a container;
The object carrier is a container crane. In the information processing system according to any one of claims 1 to 6,
comprising a camera and a server,
the camera is configured to generate image data in which the target object is contained in an angle of view;
The server is
connected to the camera and the image acquisition device via a network;
configured to accept from said image acquisition device only image data in which a target object is contained in an angle of view. An information processing method,
comprising the following steps,
In the extracting step, an image acquisition device recognizes a target object included in image data received by the image acquisition device, and extracts only image data in which the target object is contained in an angle of view;
The receiving step receives the extracted image data,
In the identification step, based on information linked to the target object and information read from the extracted image data, an object carried by the target object is identified;
the identifying step identifies the order and method of transportation of the objects carried by the target object;
The method, wherein the notifying step notifies the carrier carrier of the identified information. In the information processing method according to claim 8,
The method, wherein the recognition of the target object in the image acquisition device is based on learned data in which a plurality of types of the target object are pre-learned as teacher data. In the information processing method according to claim 9,
The image acquisition device further comprises a learning step,
In the learning step, the learned data is generated or updated using image data in which the target object is accommodated in an angle of view as teacher data. In the information processing method according to any one of claims 8 to 10,
A method, wherein the information associated with the target object is a two-dimensional barcode. In the information processing method according to any one of claims 8 to 11,
The image data received by the image collecting device is image data captured from a fixed point so that multiple lanes fit within the angle of view,
A method according to claim 1, wherein the target object is a ceiling portion above a driver's seat of a land transport vehicle. In the information processing method according to claim 12,
the carrier is a container;
The method, wherein the carrier carrier is a container crane. In the information processing method according to any one of claims 8 to 13,
comprising a camera and a server,
the camera is configured to generate image data in which the target object is contained in an angle of view;
The server is
connected to the camera and the image acquisition device via a network;
A method configured to accept only image data in which a target object is contained in an angle of view from the image acquisition device. a program,
A computer that executes each step in the information processing system according to any one of claims 1 to 7.

Images