WO2022190470A1

WO2022190470A1 - Image processing device, method therefor, and image processing program

Info

Publication number: WO2022190470A1
Application number: PCT/JP2021/043070
Authority: WO
Inventors: 拓実會下; 卓馬寺田; 光太星野; 朋晟平岡
Original assignee: 株式会社日立製作所
Priority date: 2021-03-08
Filing date: 2021-11-24
Publication date: 2022-09-15
Also published as: JP2022136898A

Abstract

The present invention efficiently determines a monitoring region based on the movement line region of a moving body.　This image processing device has: a movement region detection unit 112 that detects a plurality of movement regions from moving image data; a movement region storage unit 141 that stores the plurality of movement regions; a basis generation unit 121 that generates a basis vector based on the plurality of movement regions; and a region extraction unit 122 that extracts a candidate region from the basis vector generated by the basis generation unit.

Description

IMAGE PROCESSING DEVICE AND METHOD, AND IMAGE PROCESSING PROGRAM

The present invention relates to an image processing apparatus and method, and an image processing program, and more particularly to a monitoring technique using moving images.

Surveillance technology using moving images such as video is used in various fields. For example, in manufacturing sites, for the purpose of quality control, ensuring safety of workers, improving productivity, etc., the introduction of a mechanism for grasping the work situation using images of the work environment is being promoted. In a system that grasps the work situation using images, determining the movement area of the target object as the monitoring area prevents erroneous recognition of objects other than the target object, reduces the amount of data processing, and furthermore, allows multiple target objects to be monitored. It is essential to grasp the work status of each individual.

Since the monitoring area differs depending on the target object in the work environment and the installation status of the imaging device, it is necessary to set the monitoring region of the target object for the image of each imaging device. Therefore, when installing a plurality of imaging devices, it is necessary for an operator to manually determine a monitoring region for all target objects in the images of all the imaging devices.

Regarding determination of a monitoring area using a moving image, for example, in Patent Document 1, a moving vector of a monitored object extracted from a moving image and boundary line information detected from a background image are used to determine a boundary line. A monitoring device is disclosed that automatically determines a monitoring area by estimating a boundary line of a moving area of a monitoring object moving in a lane having a .

JP 2020-49911 A

The technique described in Patent Literature 1 is based on the premise that a monitoring area for an object such as a vehicle moving in a lane with a boundary line is determined. do. However, when the monitoring area is determined from the area where the person moves, the boundary line does not necessarily exist in the area where the person moves. For example, in the case of a worker handling products flowing on a conveyor belt in a factory, the belt conveyor may be a boundary line, but the worker does not always work around the belt conveyor, and while moving to another place Work may proceed. In that case, the technique of Patent Document 1 cannot cope with determination of the monitoring area.

One possible solution is to cover all areas where people may move (flow line areas) and extract and determine monitoring areas. However, if a monitoring area is extracted from a wide flow line area, the number of candidate monitoring areas may increase. As a result, in order to extract and analyze many monitored areas, the processing time and storage capacity of the computer increase, and the operating cost of the system increases. Therefore, it is desired to determine a useful monitoring area from the flow line area of moving bodies such as people, that is, to efficiently determine the monitoring area, while keeping costs as low as possible.

Therefore, an object of the present invention is to provide an image processing technique that can efficiently determine a monitoring area based on a moving object's flow line area.

A preferable example of the image processing device according to the present invention is
a moving area detection unit that detects a plurality of moving areas from moving image data;
a movement area storage unit that stores the plurality of movement areas;
a basis generator that generates basis vectors based on the plurality of moving regions;
a region extraction unit that extracts a candidate region from the base vectors generated by the base generation unit;
It is an image processing apparatus having
The present invention can also be grasped as an image processing method and an image processing program executed by the image processing apparatus.

According to the present invention, it is possible to efficiently determine the monitoring area based on the moving body flow line area.

1 is a diagram showing a schematic configuration example of an image processing system according to an embodiment; FIG. It is a figure which shows the hardware structural example of an image processing apparatus. 2 is a diagram illustrating an example of functional configuration of an image processing apparatus; FIG. 3 is a block diagram showing a processing example of an image processing device; FIG. 7 is a flow chart showing an example of monitoring area determination processing by an image processing apparatus; FIG. 4 is a diagram showing an example of a moving area; FIG. 4 is a diagram showing an example of basis vectors and candidate regions; 4 is a diagram showing an example of an area correspondence table 145; FIG. FIG. 14 is a diagram showing an example of an area selection table 146; FIG. FIG. 10 is a diagram showing an example of a display screen displaying selection results of candidate regions;

A preferred embodiment will be described below with reference to the drawings. In the following description, the same reference numerals may be given to the same or similar configurations, and redundant description may be omitted. Further, in the following description, examples, annotations, or identifiers (numbers, alphabets, etc.) may be indicated in parentheses when distinguishing between examples, annotations, or similar configurations.

<Configuration example of image processing system 1>
FIG. 1 shows a schematic configuration example of an image processing system 1 . The image processing system 1 is configured by connecting an imaging device 3 and an image processing device 100 as an information processing device so as to be able to communicate with each other via a wired or wireless communication means 2 . Although one imaging device 3 is illustrated in FIG. 1, it is preferable that a plurality of imaging devices 3 be installed. A plurality of imaging devices 3 are installed so as to cover a flow line area in which moving objects such as people and machines may move, and output moving image data obtained by photographing working environments of the moving objects. As the plurality of imaging devices 3, for example, a camera (digital camera (RGB camera), infrared camera, thermography camera, time of flight (TOF) camera, stereo cameras, etc.).

The communication means 2 is, for example, a communication means conforming to various communication standards such as USB (Universal Serial Bus), RS-232C, LAN (Local Area Network), WAN (Wide Area Network), the Internet, a dedicated line, and the like. Other devices such as a mobile terminal may be connected to the communication means 2 . The image processing device 100 performs processing for determining a monitoring area based on moving image data acquired by the imaging device 3 .

<Hardware Configuration Example of Image Processing Apparatus 100>
FIG. 2 shows the hardware configuration of the image processing apparatus 100. As shown in FIG.
The image processing apparatus 100 is an information processing apparatus (computer) and includes a processor 11 , a main memory device 12 , an auxiliary memory device 13 , an input device 14 , an output device 15 and a communication device 16 .

The processor 11 is, for example, a device that performs arithmetic processing, such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), AI (Artificial Intelligence) chip, and the like. The main storage device 12 is a device that stores programs and data, and includes, for example, ROM (Read Only Memory) (SRAM (Static Random Access Memory), NVRAM (Non Volatile RAM), mask ROM (Mask Read Only Memory), PROM (Programmable ROM), etc.), RAM (Random Access Memory) (DRAM (Dynamic Random Access Memory), etc.). The auxiliary storage device 13 is a hard disk drive, flash memory, SSD (Solid State Drive), optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), etc. . Programs and data stored in the auxiliary storage device 13 are read into the main storage device 12 as needed.

The input device 14 is a user interface that receives information from the user, such as a keyboard, mouse, card reader, touch panel, and the like. The output device 15 is a user interface that outputs various information (display output, audio output, print output, etc.). These include an output device (speaker), a printing device, and the like.

The communication device 16 is a communication interface that communicates with other devices via the communication means 2, such as a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, and the like. The communication device 16 can also function as a device that receives information from other devices such as mobile terminals that are communicably connected. The communication device 16 can also function as a device that transmits information to other devices that are communicatively connected. The image processing device 100 communicates with the imaging device 3 via the communication means 2 by the communication device 16 .

The above functions of the image processing apparatus 100 are implemented by the processor 11 reading out and executing programs stored in the main storage device 12 . Note that it may be realized by hardware (FPGA, ASIC, AI chip, etc.) that constitutes the image processing apparatus 100 .

<Functional Configuration Example of Image Processing Apparatus 100>
FIG. 3 shows an example of the functional configuration of the image processing apparatus 100. As shown in FIG.
The image processing apparatus 100 includes an acquisition unit 110 , an extraction unit 120 , a selection unit 130 and a storage unit 140 . In addition to the functions described above, the image processing apparatus 100 may further include functions such as an operating system, a device driver, a file system, and a DBMS (DataBase Management System).

Here, the storage unit 140 stores a movement area storage unit 141, moving image data 142, base vectors 143, candidate areas 144, area correspondence table 145, and area selection table 146. The storage unit 140 stores such information as, for example, a database table provided by a DBMS or a file provided by a file system.

The acquisition unit 110 receives and acquires data transmitted from other devices. Acquisition unit 110 includes data acquisition unit 111 and moving region detection unit 112 . The data acquisition unit 111 acquires moving image data 142 transmitted from the imaging device 3 . The movement area detection unit 112 detects a movement area from the moving image data 142 and stores the obtained movement area in the movement area storage unit 141 .

The extraction unit 120 extracts candidate areas based on a plurality of moving areas stored in the moving area storage unit 141 . The extractor includes a base generator 121 and a region extractor 122 . The base generation unit 121 generates a plurality of base vectors 143 based on the plurality of motion regions stored in the motion region storage unit 141 . A region extraction unit 122 extracts a candidate region 144 from the basis vectors 143 .

The selection unit 130 selects the candidate area 144 and outputs it as a monitoring area. The selection unit includes an area integration unit 131 and an area selection unit 132 . The area integration unit 131 integrates the overlapping candidate areas 144 . The region selection unit 132 selects candidate regions 144 based on the region selection table 146 .

<Processing of image processing device>
FIG. 4 is a block diagram showing an example of processing performed by the image processing apparatus 100 when determining a monitoring area of a moving image of a working environment.

(1) First, the moving image data 142' of the working environment captured by each of the plurality of imaging devices 3 is acquired. For example, the moving image data 142' is a moving image of a plurality of imaging devices installed in the work environment capturing the working of target objects to be monitored, such as workers, machine tools, and other manufacturing equipment. .

(2) The data acquisition unit 111 of the acquisition unit 110 acquires the moving image data 142 ′ transmitted from the imaging device 3 and stores it in the moving image data 142 of the storage unit 140 . Further, the movement area detection unit 112 detects a movement area from the moving image data 142 ′ and stores the detected movement area in the movement area storage unit 141 .
For example, the movement area detection unit 112 detects an area in which movement occurs between a plurality of frames as a movement area by optical flow or background subtraction.

(3) The base generation unit 121 of the extraction unit 120 generates one or more base vectors 143 based on the plurality of moving regions stored in the moving region storage unit 141 . Then, the area extracting unit 122 extracts the candidate area 144 from the base vector 143 .
For example, the base generation unit 121 generates one or more eigenvectors obtained by principal component analysis of a plurality of movement regions as the base vectors 143, and the region extraction unit 122 extracts predetermined A set of elements exceeding the value is extracted as a candidate region 144 .

(4) The area integration unit 131 of the selection unit 130 integrates the overlapping candidate areas 144 . Then, the area selection unit 132 selects candidate areas 144 based on the area selection table 146 .

(5) Thereby, the image processing apparatus 100 determines the statistically extracted area as the monitoring area based on one or more basis vectors generated from the plurality of moving areas included in the moving image data 142. be able to. Therefore, even for an unknown target object, it becomes possible to determine the monitoring region from the moving region of the target object, and efficiency in determining the monitoring region can be achieved.

<Determination of monitoring area>
FIG. 5 is a flowchart showing monitoring area determination processing by the image processing apparatus 100 .
In the image processing apparatus 100, first, the data acquisition unit 111 acquires a plurality of moving image data 142' transmitted from the plurality of imaging devices 3 and stores them in the moving image data 142 (S510). Then, the movement area detection unit 112 detects a movement area from the moving image data 142 (S520), and stores the detected movement area in the movement area storage unit 141 (S530).

Specifically, for example, the movement area detection unit 112 detects an area that has moved between a plurality of frames as a movement area by optical flow, background subtraction, or the like.

FIG. 6 shows an example of an image included in the moving image data 142 and an example of a moving area detected from the moving image including the image. FIG. 6(1) shows a worker G10-1 moving rightward on the image G1 and a robot arm G10-2 moving leftward on the image G1. Further, FIG. 6(2) shows the movement area G2 detected from the moving image including the image G1. 2 is detected. Note that the number of moving regions G20 to be detected is not limited, and a plurality of moving regions G20 do not need to be distinguished because they overlap. In the moving region G2, the moving region detection unit 112 sets “1” to each pixel of the moving regions G20-1 and G20-2, and sets “0” to each pixel of the other regions. Note that the pixel value set for each pixel in the moving region G20-1 and the moving region G20-2 may be other than "1". For example, a value indicating the amount of movement of each pixel calculated by optical flow from one or more pixels corresponding to each pixel may be set as the pixel value.

Next, the base generation unit 121 generates a plurality of basis vectors 143 (M1, M2, . . . , Mk (k is a predetermined integer of 1 or more )) is generated (S540). Specifically, for example, the basis generation unit 121 generates one or more eigenvectors obtained by principal component analysis of a plurality of moving regions as basis vectors (M1, M2, . . . , Mk).

FIG. 7 shows an eigenvector (base vector M1) of the first principal component obtained by principal component analysis of a plurality of moving regions, as an example of the base vector. The value of k is generated by principal component analysis in the order of the eigenvector of the first principal component (base vector M1) and the eigenvector of the second principal component (base vector M2). may be up to the eigenvector of the k-th principal component (basis vector Mk). Further, the basis generation unit 121 may use a plurality of basis obtained by independent component analysis of a plurality of moving regions as a basis vector. Alternatively, weights of an autoencoder that match the input movement area and the output movement area may be used as basis vectors.

Next, the region extracting unit 122 extracts candidate regions 144 (R1, R2, …, Rn (n is an integer equal to or greater than 1)) from the basis vectors 143 (M1, M2, …, Mk) (S550). . Specifically, for example, the region extracting unit 122 extracts a set of elements having a positive value exceeding a predetermined value and a set of elements having a negative value exceeding a predetermined value among the elements of the base vector M1. are extracted as a candidate region R1 for the base vector M1 and a candidate region R2 for the base vector M1, respectively. Similar operations are performed on basis vectors M1, M2, …, Mk to obtain candidate regions R1, R2, …, Rn.

FIG. 7 shows a candidate region R1 and a candidate region R2 extracted from the base vector M1 as an example of candidate regions. In FIG. 7, in the candidate region R1 of the base vector M1, a region O1 of a set of elements having positive values exceeding a predetermined value among the elements of the base vector M1 is extracted. Also, in the candidate region R2 of the base vector M1, a region O2 of a set of elements having a negative value exceeding a predetermined value among the elements of the base vector M1 is extracted. In the candidate region R1 and the candidate region R2, it is assumed that each pixel of the region O1 and the region O2 is set to "1" and each pixel of the other region is set to "0". A pixel value other than "1" may be set to the pixel. Alternatively, the region extracting unit 122 may divide the region O into a plurality of regions using a region division method for the candidate region R of the base vector M, and use each of the divided regions as the candidate region of the base vector M. . For example, the candidate region R1 of the base vector M1 is divided into multiple regions (O11, O12, …, O1p) using a region division method such as the Watershed method, and the divided regions (O11, O12 , …, O1p) may be used as candidate regions for basis vectors M1.

FIG. 8 shows an example of the area correspondence table 145. FIG. The region correspondence table 145 manages candidate region names and corresponding base vectors. In the area correspondence table 145, the candidate area with "1" set in the column of the basis vector Mi indicates that the candidate area is extracted from the basis vector Mi. For example, candidate regions R1 and R2 have "1" set in the column of base vector M1, so candidate regions R1 and R2 are candidate regions extracted from base vector M1. The region correspondence table 145 is updated each time the region extraction unit 122 extracts a candidate region or each time the region integration unit 131 integrates the candidate regions. After the above process, the area integrating section 131 integrates the overlapping candidate areas 144 based on the area correspondence table 145 (S560).

Here, the process of integrating the overlapping candidate areas 144 by the area integration unit 131 will be described. The region integration unit 131 first refers to the region correspondence table 145 and selects one candidate region Ri from the candidate regions (R1, R2, . . . , Rn). Next, referring to the region correspondence table 145, a candidate region Rj that overlaps with the candidate region Ri is selected from candidate regions extracted from base vectors different from the candidate region Ri. Specifically, for example, using an index representing the similarity of sets such as the Jaccard coefficient or the Simpson coefficient, the pixel set included in the region Oi of the candidate region Ri and the pixel set included in the region Oj of the candidate region Rj are calculated. A degree of similarity is calculated, and if the calculated degree of similarity exceeds a predetermined value, it is determined that the candidate regions Ri and Rj overlap.

Next, when it is determined that the candidate region Ri and the candidate region Rj overlap, the region integration unit 131 compares the areas of the candidate region Ri and the candidate region Rj, and selects a candidate region with a small area or a candidate region with a large area. The candidate regions Ri and Rj are integrated by deleting them from the correspondence table 145 . Note that the integration method is not limited to the method of deleting candidate regions with small or large areas. The vectors may be compared, and candidate regions extracted from eigenvectors with low contribution rates may be deleted from the region correspondence table 145 .

As described above, when the basis vectors are principal component analysis, it is possible to narrow down to a small number of candidate regions by selecting the eigenvectors (basis vectors) of the upper principal components. Furthermore, by executing the area integration process, it is possible to narrow down to a smaller number of candidate areas. As a result, it is possible to reduce the amount of calculation and the processing time required for analyzing the image data.

Next, the area selection unit 132 selects the candidate area 144 based on the area selection table 146 (S570), outputs the selected candidate area 144 as the monitoring area (S580), and performs the process of determining the monitoring area. finish.

FIG. 9 shows an example of the area selection table 146. FIG. The area selection table 146 manages candidate area names and corresponding selection results. A candidate area for which "1" is set in the selection result column of the area selection table 146 indicates that it is a selected candidate area. For example, since "1" is set in the selection result column for candidate regions R1 and R2, candidate regions R1 and R2 are selected candidate regions.

FIG. 10 is a diagram showing an example of a display screen displaying selection results of candidate regions.
The candidate area selected by the area selection unit 132 is displayed on the output device 15 of the image processing apparatus 100 as a monitoring area. The selection result may be displayed on a display of a terminal (not shown) connected via the communication device 16. FIG.

A display screen 1000 includes a monitoring area result table 1001 and a monitoring area display 1002 . The monitoring area result table 1001 is data displaying the contents of the area correspondence table 145 and the area selection table 146 . In the monitoring area display 1002, the candidate areas for which "1" is set in the selection result column of the area selection table 146 are displayed together with the rectangle surrounding the candidate area and the name of the candidate area superimposed. For example, since "1" is set in the selection result column of the region selection table 146 for the candidate region R1 and the candidate region R2, the candidate region R1 and the candidate region R2 are the rectangles surrounding each candidate region and each candidate region. displayed with the first name. The user can recognize which candidate area has been selected and how close or overlapping the selected areas R1 and R2 are by looking at the

displays

1001 and 1002 on the display screen.

As described above, according to the present embodiment, statistically extracted regions are determined as monitoring regions based on one or more base vectors generated from a plurality of moving regions in the moving image data 142. can be done. Therefore, even for an unknown target object, it becomes possible to determine the monitoring region from the moving region of the target object, and efficiency in determining the monitoring region can be achieved.

It goes without saying that the present invention is not limited to the above-described embodiments, and can be modified in various ways without departing from the gist of the present invention. For example, the above embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Moreover, it is possible to add, delete, or replace a part of the configuration of the above embodiment with another configuration.

For example, in the above embodiment, moving image data acquired by the imaging device 3 is processed. The moving image data may be processed in real time, or the moving image data acquired and stored in the past may be processed afterward. Furthermore, moving image data (virtual moving image data) generated for simulation by the processor 11 or another processing device may be processed as well as the moving image data acquired from the imaging device 3 .

In addition, each of the above configurations, functional units, processing units, processing means, etc. may be implemented in hardware, for example, by designing a part or all of them using an integrated circuit. Further, each configuration, function, etc. described above may be implemented by a program that causes the processor to implement each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (Solid State Drives), and recording media such as IC cards, SD cards, and DVDs.

In addition, the arrangement form of various functional units, various processing units, and various databases of each information processing apparatus described above is merely an example. The arrangement form of various functional units, various processing units, and various databases can be changed to an optimum arrangement form from the viewpoint of the performance, processing efficiency, communication efficiency, etc. of hardware and software provided in these devices.

In addition, the configuration of the database that stores the various data described above (schema, etc.) can be changed as appropriate from the viewpoint of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency, etc.

1: image processing system 2: communication means 3: imaging device 11: processor 12: main storage device 13: auxiliary storage device 14: input device 15: output device 16: communication device 100: image processing device 110: acquisition unit 111: data Acquisition unit 112: moving region detection unit 120: extraction unit 121: base generation unit 122: region extraction unit 130: selection unit 131: region integration unit 132: region selection unit 140: storage unit 141: movement region storage unit 142: moving image Data 143: Base vector 144: Candidate area 145: Area correspondence table 146: Area selection table

Claims

a moving area detection unit that detects a plurality of moving areas from moving image data;
a movement area storage unit that stores the plurality of movement areas;
a basis generator that generates basis vectors based on the plurality of moving regions;
a region extraction unit that extracts a candidate region from the base vectors generated by the base generation unit;
An image processing device having
The basis generation unit generates eigenvectors obtained by principal component analysis of the plurality of movement regions as basis vectors.
The image processing apparatus according to claim 1.
a region integration unit that integrates the overlapping candidate regions extracted by the region extraction unit;
an area selection unit that selects the candidate area from the result of integration by the area integration unit;
The image processing apparatus according to claim 1.
a data acquisition unit that acquires a plurality of moving image data acquired by a plurality of imaging devices installed in an area where a moving object may move;
a moving image data storage unit that stores the plurality of moving image data acquired by the data acquisition unit;
2. The image processing apparatus according to claim 1, wherein said movement area detection section detects said plurality of movement areas based on said plurality of moving image data acquired by said data acquisition section.
The basis generator sequentially generates eigenvectors of the first principal component to eigenvectors of the k-th principal component whose contribution rate or cumulative contribution rate exceeds a predetermined value (k is a plurality of integers) by the principal component analysis. ),
3. The image processing apparatus according to claim 2.
The region extraction unit extracts a region O1 of a set of elements having a positive value exceeding a predetermined value among the elements of the base vector M1 in the candidate region R1 of the base vector M1, and extracts a region O1 of the base vector M1 candidate. extracting a region O2 of a set of elements having a negative value exceeding a predetermined value among the elements of the base vector M1 in the region R2;
create a region correspondence table that manages the candidate region names of the extracted candidate regions and their corresponding basis vectors;
The image processing apparatus according to claim 1.
When determining that a plurality of candidate regions overlap, the region integration unit integrates the candidate regions by comparing the areas of the plurality of candidate regions and deleting a candidate region with a small area or a candidate region with a large area. ,
4. The image processing apparatus according to claim 3.
Names of candidate areas selected by the area selection unit (candidate area names), selection results, a monitoring area result table showing base vectors corresponding to the candidate area names, and display of the selected candidate areas. Having an output device for displaying a display screen,
The image processing apparatus according to claim 1.
a moving region detection step of detecting a plurality of moving regions from moving image data;
a movement area storage step of storing the plurality of movement areas;
a basis generating step of generating basis vectors based on the plurality of moving regions;
a region extraction step of extracting a candidate region from the base vectors generated by the base generation step;
An image processing method comprising:
In the base generation step, eigenvectors obtained by principal component analysis of the plurality of moving regions are generated as basis vectors.
10. The image processing method according to claim 9.
a region integration step of integrating the overlapping candidate regions extracted by the region extraction step;
an area selection step of selecting the candidate area from the result of integration by the area integration step;
10. The image processing method according to claim 9.
An image processing program executed by a computer is
a moving area detection unit that detects a plurality of moving areas from moving image data;
a movement area storage unit that stores the plurality of movement areas;
a basis generator that generates basis vectors based on the plurality of moving regions;
a region extraction unit that extracts a candidate region from the base vectors generated by the base generation unit;
A program for image processing.