WO2018070032A1

WO2018070032A1 - Image processing apparatus

Info

Publication number: WO2018070032A1
Application number: PCT/JP2016/080489
Authority: WO
Inventors: 信夫大石
Original assignee: 富士機械製造株式会社
Priority date: 2016-10-14
Filing date: 2016-10-14
Publication date: 2018-04-19
Also published as: JPWO2018070032A1

Abstract

A video camera 11 acquires a signal of a part of a captured image, calculates a feature amount including the gradient direction and the gradient intensity of luminance of a local region (cell), and determines whether a prescribed object is detected in the captured image by matching the calculated cell feature amount with a standard feature amount including the gradient direction and the gradient intensity of luminance obtained by capturing the prescribed object. The video camera 11 starts recording the captured image when it is determined that the prescribed object has entered the captured image.

Description

Image processing device

The present invention relates to an image processing apparatus.

Conventionally, as an image processing apparatus, for example, high-speed writing of an imaging result to a memory is terminated based on a timing at which an image of a specific pattern is detected in a specific order based on an image recognition result, and the memory is stored in this memory. There has been proposed a technique for recording a written imaging result on a recording medium (see, for example, Patent Document 1). In this apparatus, it is assumed that a desired scene can be surely taken in slow motion by applying it to a video camera in which recording means is integrated.

JP 2008-141282 A

However, in the apparatus described in Patent Document 1, for example, pattern matching, a learning function (Ada-Boost), an image difference table, and the like are used to target a specific pattern in a detection area that is one area included in an image. It has been desired to detect an object at a higher speed.

The present invention has been made in view of such a problem, and a main object of the present invention is to provide an image processing apparatus that can capture an object moving at high speed more reliably.

The present invention adopts the following means in order to achieve the main object described above.

The image processing apparatus disclosed in this specification is:
A calculation unit that acquires a signal of a part of the captured image captured by the imaging unit and calculates a feature amount including a gradient direction and gradient strength of the luminance of the local region;
The predetermined feature including the luminance gradient direction and gradient intensity obtained by imaging the predetermined object is matched with the calculated feature of the local region, and the predetermined object is detected in the captured image. A determination unit for determining whether or not
A control unit that starts recording the captured image when the determination unit determines that the predetermined object has entered the captured image;
It is equipped with.

In this apparatus, a partial signal of a captured image is acquired, a feature amount including a luminance gradient direction and gradient intensity of a local region is calculated, and a luminance gradient direction and gradient intensity obtained by imaging a predetermined target object It is determined whether or not a predetermined object is detected in the captured image by matching the fixed feature quantity including the feature quantity of the local region obtained by the calculation. The apparatus starts recording the captured image when it is determined that the predetermined object has entered the captured image. In this device, feature quantities are calculated in rough units of luminance gradient direction and gradient intensity, and a predetermined target can be detected at higher speed by matching feature quantities in local regions. An object can be imaged more reliably. For example, an object that moves at high speed can be reliably recognized within one frame, and the start and stop of imaging and recording can be controlled.

1 is a schematic explanatory diagram illustrating an example of a camera system 10. FIG. FIG. 2 is a block diagram of an imaging unit 12, an image processing unit 20, and a recording device 41. Explanatory drawing showing an example of an image matching process. Explanatory drawing showing an example of a template. The flowchart which shows an example of a moving image automatic imaging | photography process routine. The flowchart which shows an example of an image matching process routine. The timing chart of an image input, a matching process, and a recording control signal. 6 is a timing chart of conventional image input, matching processing, and recording control signals.

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic explanatory diagram illustrating an example of a camera system 10 that is an example of the present invention. The camera system 10 includes a video camera 11, a computer (PC) 40, and a recording device 41. The video camera 11 is a device that captures an image such as a moving image or a still image. The computer 40 captures image data captured by the video camera 11 and displays it on a display or the like. The recording device 41 records image data captured by the video camera 11.

The video camera 11 includes an imaging unit 12, a control unit 14, a timing control unit 16, and an image processing unit 20. For example, the video camera 11 is configured as a high frame rate camera capable of capturing an image at a maximum of 1000 fps (the cycle of one frame is 1 ms). The imaging unit 12 includes an imaging element 13. The imaging element 13 is an element that generates charges by receiving light and outputs the generated charges. The image sensor 13 may be a CMOS image sensor, for example. The control unit 14 controls the entire apparatus and includes a CPU 15. The video camera 11 starts moving image shooting when an object registered in advance enters the imaging range, and ends moving image shooting when the object is out of the imaging range or the maximum recording time elapses. Execute shooting mode. As will be described in detail later, the control unit 14 performs a process of starting to continue storing the captured image (starting recording) when the image processing unit 20 determines that the predetermined object has entered the captured image. Further, the control unit 14 performs a process of stopping the continuous storage of the captured image (stops recording) when a predetermined target is removed from the captured image by the image processing unit 20. The timing control unit 16 generates various clocks used for the operation of the image processing unit 20.

FIG. 2 is a block diagram of the imaging unit 12, the image processing unit 20, and the recording device 41. In this video camera 11, the image processing unit 20 performs an object matching process on the image signal input from the imaging unit 12. 3A and 3B are explanatory diagrams illustrating an example of image matching processing of the captured image 50. FIG. 3A is an explanatory diagram of feature amounts of the cells 51 and the pixels 52, and FIG. 3B is an explanatory diagram of binary coding. FIG. 3C is an explanatory diagram of the matching process. This matching process will be described in detail later. The image processing unit 20 outputs a recording control signal (Rec Trigger = ON or OFF) to the recording device 41 according to the presence / absence of an object. When the recording control signal is input, the recording device 41 starts and stops recording of the image signal output from the imaging unit 12.

The image processing unit 20 includes a feature amount calculation unit 21 and an image determination unit 30. The feature amount calculation unit 21 acquires a partial signal of the captured image captured by the imaging unit 12 and calculates a feature amount including the gradient direction and gradient strength of the luminance of the local region (cell). As shown in FIG. 3A, the cell 51 is an area in which an image is divided by a predetermined number of pixels (for example, 7 × 7, 3 × 3, etc.). The feature quantity computing unit 21 inputs the signal from the imaging unit 12 and simultaneously executes the process described below. The feature amount calculation unit 21 includes a function calculation unit 22, a gradient strength calculation unit 23, a gradient direction calculation unit 24, a histogram generation unit 25, a histogram RAM 26, a sort unit 27, and a binary code conversion unit 28. The function calculation unit 22 is a circuit that calculates the function Gx and Gy of the feature quantity including the gradient direction and the gradient strength. These functions Gx and Gy are obtained by using, for example, the luminance values according to equations (1) and (2). The gradient strength calculation unit 23 is a circuit that calculates the gradient strength of luminance using the function Gx and Gy of the feature amount. This gradient strength is obtained by equation (3). The gradient direction calculation unit 24 is a circuit that calculates the gradient direction of the luminance using the function Gx, Gy of the feature amount. This gradient strength is obtained by equation (4). The gradient direction calculation unit 24 outputs the gradient direction divided for each angle. The gradient direction calculation unit 24 outputs a gradient direction obtained by dividing 180 ° into eight BINs. The function calculator 22, the gradient strength calculator 23, and the gradient direction calculator 24 each have a plurality of circuits so that a plurality of pixels can be processed simultaneously. For example, the function calculation unit 22 has eight calculation circuits and can process eight pixels simultaneously. The gradient strength calculation unit 23 and the gradient direction calculation unit 24 are the same as the function calculation unit 22.

The histogram generation unit 25 is a circuit that generates a histogram of each cell representing the relationship between the gradient direction and the gradient intensity. The histogram generator 25 has, for example, eight circuits and can process eight cells in parallel. The histogram RAM 26 is a circuit that stores the generated histogram, and may be, for example, a dual port RAM (DPRAM) having both a read data output terminal and a write data output terminal. The sorting unit 27 is a circuit that sorts cell histograms in descending order of gradient strength. The binary code conversion unit 28 is an arithmetic circuit that generates a binary code using an angle having the highest gradient strength in a cell histogram as a HOG (Histograms of Oriented Gradients) feature amount of the cell. The binary code of the feature amount is defined by an 8-digit code corresponding to 0 to 7 by dividing 180 ° into 8 parts (see FIG. 3B). The feature amount calculation unit 21 includes a plurality of circuits, and can calculate a feature amount in parallel with respect to a plurality of pixels and cells.

The image determination unit 30 performs matching between the fixed feature amount including the gradient direction and the gradient strength of the luminance obtained by imaging the predetermined object and the calculated feature amount of the local region (cell), and performs the predetermined object. Is detected in the captured image. The image determination unit 30 is a determination circuit that performs a matching process with a feature amount of an input image using a binary-coded fixed feature amount (template). The image determination unit 30 includes a binary code register 31, a binary code RAM 32, a register 33, a binary code register 34, a binary code RAM 35, a matching determination unit 36, a matching rate RAM 37, and a sorting unit 38. The binary code register 31, the binary code RAM 32, and the register 33 are circuits that store the feature amount (binary code) of the input image. Each of the binary code register 31 and the binary code RAM 32 has eight storage circuits. These storage circuits are collectively read from the binary code RAM 32 for one cell line, stored in the register 33, and operate as a shift register to form a sliding window.

The binary code register 34 and the binary code RAM 35 are storage circuits for inputting and storing the feature amount of the template output from the control unit 14. The matching determination unit 36 is a circuit that performs matching between the feature amount of the template obtained in advance from the image of the object and the feature amount of the input image, and counts the matching rate (see FIG. 3C). The matching determination unit 36 includes a matching check unit 36a and a matching rate counting unit 36b. The matching check unit 36a performs matching between the feature amount of the template and the feature amount of the input image. The matching rate counting unit 36b counts the matching rate between the feature amount of the template and the feature amount of the input image. The matching rate RAM 37 is a storage circuit that stores the matching rate determined by the matching determination unit 36. The sorting unit 38 is a circuit that sorts the relevance ratios and outputs a higher-order determination result to the control unit 14. The matching between the feature quantity of the template and the feature quantity of the input image is performed at each position while moving the image one cell at a time. The image determination unit 30 includes a plurality of registers (binary code register 31, binary code RAM 32, and A plurality of cell feature values are stored in each of the registers, and matching is performed by operating the stored feature values for each cell as a shift register. For this reason, the image determination unit 30 can realize a higher-speed matching process than that for reading out one cell at a time.

Here, the template is explained. FIG. 4 is an explanatory diagram illustrating an example of a template. 3 and 4 and the like will be described using an automobile for easy understanding, but the moving object is not limited to the automobile. Similar to the input image described above, this template is a reference cell feature amount obtained based on an image obtained by capturing an object. This template may be generated using, for example, a binary code including at least a first feature amount of a cell and a second feature amount at a position shifted by a predetermined amount with respect to the cell. This template includes a first template obtained from a cell divided at the first position. The template also includes a second template obtained from a cell divided at a second position moved by a predetermined number of pixels (for example, (−3, −3)) from the first position to the negative side. Furthermore, this template also includes a third template obtained from a cell divided at a third position moved by a predetermined number of pixels (for example, (3, 3)) from the first position to the positive side. The template of FIG. 4 is obtained by synthesizing three binary codes generated from three types of histograms by binary-coding the top three BINs that exceed a predetermined threshold in 8 BINs in the first to third templates. Is obtained as one binary code. Since the image processing unit 20 matches the feature amount of the input image using this template, matching the three positions at a time can improve robustness and reduce the load on the circuit. Further, the purpose of realizing the matching with the three templates in which the cell cut points are arbitrarily changed is to improve the detection rate. The captured image is distorted and appears to change shape due to the parallax due to the change in the relative position of the camera to the object, so the robustness against the parallax is improved by matching simultaneously with the template of three viewpoints .

In this video camera 11, the imaging unit 12 may be capable of high-speed imaging at a rate exceeding 240 fps, for example, a maximum of 1000 fps. At this time, the image determination unit 30 of the image processing unit 20 may detect the predetermined object within one frame after the object enters the captured image, for example, within 1 μsec. The image processing unit 20 performs a high-speed matching process by calculating a binary-coded HOG feature quantity by a parallelized circuit and constructing a sliding window using a shift register using the feature quantity. be able to. The video camera 11 can be used, for example, when recording an object moving at high speed. It is preferable that the object does not change its direction (direction) or size (distance) during movement. The video camera 11 can be used, for example, in a parts camera that takes a picture while moving a collected component in a mounting apparatus that mounts the component on a substrate. The user can store and save the captured image in the recording device 41 or perform slow motion playback on the computer 40.

Next, the operation of the camera system 10 of the present embodiment configured as described above, first, processing for automatically capturing a moving image of a predetermined moving object will be described. FIG. 5 is a flowchart illustrating an example of a moving image automatic shooting processing routine executed by the video camera 11. FIG. 6 is a flowchart illustrating an example of an image matching processing routine executed by the image processing unit 20. The moving image automatic shooting processing routine is executed by the CPU 15 of the control unit 14 after the user inputs execution of the moving image automatic shooting processing. It is assumed that the user registers an image of an object before executing the automatic photographing process, and the control unit 14 creates a template for the object.

When this routine is started, the CPU 15 inputs an image signal from the imaging unit 12 (step S100). Next, the CPU 15 inputs an image matching result from the image processing unit 20 (step S110), and determines whether or not an object is detected in the image (step S120). This determination is performed based on, for example, whether or not the relevance ratio as the image matching result is equal to or higher than a predetermined threshold (for example, 75% or higher). When the object is not detected in the image, the CPU 15 executes the processing from step S100 onward. When the object is detected in the image, the CPU 15 starts recording (step S130). Next, as in steps S100 to S120, the CPU 15 inputs an image signal from the imaging unit 12 (step S140), inputs an image matching result from the image processing unit 20 (step S150), and images the object. It is determined whether or not it has been detected (step S160). When the object is detected in the image, that is, when the state in which the object is present in the image continues, the CPU 15 determines whether or not the recordable time of the recording device 41 has been exceeded (step S170). When the possible time has not been exceeded, the processing after step S140 is continuously executed.

On the other hand, when the recordable time is exceeded in step S170, or when the object is not detected in the image in step S160, that is, when the object moves out of the image, the CPU 15 sets that time as the recording end time. (Step S180). Subsequently, the CPU 15 ends the recording and executes the processes after step S100. In addition to the recording device 41, the output destination of the image may be any one or more of, for example, a storage medium (DVD, HD, etc.) provided in the video camera 11 or the computer 40. The captured moving image is reproduced by the computer 40, for example. For example, since the imaging unit 12 captures images with a high image quality such as 1000 fps, the image capture unit 12 can perform slow reproduction or the like with high-quality images.

Next, image matching processing (FIGS. 3 and 6) executed by the image processing unit 20 will be described. This routine is executed by the image processing unit 20 every time an image signal is input from the imaging unit 12. This routine is executed by each circuit included in the image processing unit 20. When this routine is started, the feature amount calculation unit 21 of the image processing unit 20 inputs an image signal (step S200), calculates functions Gx and Gy related to the gradient direction and gradient intensity of the luminance (step S210), and the calculation result Is used to calculate the gradient intensity and gradient direction of the brightness of each cell (step S220, FIG. 3A). Next, the feature amount calculation unit 21 divides 180 ° into eight BINs (step S230), generates a histogram for each cell, and stores the histogram in the histogram RAM 26 (step S240). Next, the feature amount calculation unit 21 sorts the histogram values (step S250), generates a binary code in which one direction is assigned to 1 bit with a value corresponding to the representative value of the cell (FIG. 3B), and The data is stored in the registers (binary code register 31, binary code RAM 32, and register 33) of the determination unit 30 (step S260). These processes are executed in parallel for a plurality of pixels and cells using a plurality of circuits.

Subsequently, the image determination unit 30 inputs the template image from the control unit 14 and stores it in the binary code register 34 and the binary code RAM 35, and performs a matching process between the template and the feature amount of the input image (step S270). At this time, the image determination unit 30 performs matching processing while shifting the feature amount of the input image of the cell for one line. In addition, since the image determination unit 30 performs matching processing using binary-coded feature quantities, simple operations such as AND and OR are possible, and determination can be performed at high speed (see FIG. 3C). Then, the image determination unit 30 stores the matching rates in the matching rate RAM 37 (step S280), sorts the matching rates, and outputs the top three matching rates to the control unit 14 as matching results (step S290). Exit. Such processing is repeated every time an image signal is input.

FIG. 7 is a timing chart of image input, matching processing, and recording control signals. FIG. 8 is a timing chart of conventional image input, matching processing, and recording control signals. In the conventional image processing unit, as shown in FIG. 8, the matching process takes several frames or the like, and recording cannot be started at an appropriate timing. In such a video camera, since the start of recording is delayed, a large-capacity frame memory for interpolating it is necessary. On the other hand, the image processing unit 20 can perform feature amount calculation and matching processing simultaneously with image signal input. As shown in FIG. 7, the matching result immediately after one image transmission by the imaging unit 12. Therefore, it is possible to start recording an image from the next frame in which the object is detected. In addition, the image processing unit 20 can expand the search target range in the image to the entire image. In addition to an object that moves at a constant speed and a constant direction, an object that randomly enters the imaging range, an object that moves randomly, and the like Can also be detected in real time.

Here, the correspondence between the components of the present embodiment and the components of the present invention will be clarified. The imaging unit 12 of the present embodiment corresponds to an imaging unit, the feature amount calculation unit 21 corresponds to a calculation unit, the image determination unit 30 corresponds to a determination unit, and the control unit 14 corresponds to a control unit.

The video camera 11 of the camera system 10 of the present embodiment described above acquires a partial signal of the captured image, calculates a feature amount including the gradient direction and gradient intensity of the luminance of the local region (cell), and obtains a predetermined target. A predetermined target object is detected in the captured image by matching the fixed feature (template) feature value including the gradient direction and gradient intensity of the brightness obtained by imaging the object with the feature value of the cell obtained by the calculation. It is determined whether or not. The video camera 11 starts recording the captured image when it is determined that the predetermined object has entered the captured image. The video camera 11 calculates feature amounts in rough units of luminance gradient direction and gradient intensity, and can detect a predetermined target at a higher speed by matching the feature amounts in the local region. The object can be imaged more reliably. In general, in an apparatus for shooting a moving image, a user inputs start and end of shooting, but there is a delay from the start operation to actual recording, and a decisive moment may be missed. . The video camera 11 can quickly detect an object and start capturing a moving image.

Further, since the control unit 14 stops the recording of the captured image when the predetermined object is removed from the captured image by the image determination unit 30, the image capturing can be started and stopped at a more appropriate timing. Furthermore, the feature amount calculation unit 21 is a calculation circuit that creates a histogram for each gradient direction divided into a plurality of directions, and generates a binary code as a feature amount of a cell whose representative value is a direction having a strong gradient strength. The image determination unit 30 is a determination circuit that performs matching using a binary-coded standard feature value. In this video camera 11, since each process is performed by hardware called a circuit, an object can be detected at a higher speed. Furthermore, the image determination unit 30 performs matching using a binary code including at least a first feature amount of a cell and a second feature amount at a position shifted by a predetermined amount with respect to the cell as a standard feature amount of the cell. Since the image determination unit 30 detects an object by matching with a fixed feature amount including a plurality of feature amounts, for example, matching can be performed at once including a case where a local region is shifted. It is possible to further reduce the rate reduction and further reduce the increase in the number of matching times to shorten the processing time. In addition, since the image determination unit 30 performs matching processing using binary code, simple operations such as AND and OR are possible, and higher-speed operation processing can be realized.

Further, the imaging unit 12 can perform high-speed imaging at a rate exceeding 240 fps, and the image determination unit 30 detects a predetermined target within one frame after the target enters the captured image. Since the image processing unit 20 can detect an object at a higher speed, the object can be detected in real time without delay. Further, the image determination unit 30 has a plurality of registers, stores the calculated feature values of the cells in each of the registers, and performs matching by operating the stored feature values for each cell as a shift register. Since the image determination unit 30 is configured to move (shift) the data in the circuit, it is not necessary to read out the feature amount each time, and matching processing can be performed at a higher speed. Furthermore, since the feature amount calculation unit 21 includes a plurality of circuits and calculates the feature amount in parallel by parallelizing a plurality of pixels, it is possible to realize faster calculation of the feature amount.

It should be noted that the present invention is not limited to the above-described embodiment, and it goes without saying that the present invention can be implemented in various modes as long as it belongs to the technical scope of the present invention.

For example, in the above-described embodiment, feature values that are binary-coded are used. However, the present invention is not limited to this, and feature values other than binary codes may be used.

In the above-described embodiment, it has been described that one template is created from three templates. However, the present invention is not particularly limited, and one template may be created from two templates, and one template may be created from four or more templates. A template may be created. The template may be appropriately created in consideration of the high speed with which the object is detected.

In the above-described embodiment, the image determination unit 30 includes a plurality of registers and performs a matching process by operating a shift register. However, this may be omitted. In order to speed up the matching process, it is desirable that the image determination unit 30 operates as a shift register.

In the above-described embodiment, the feature amount calculation unit 21 performs parallel processing using a plurality of circuits, but this may be omitted. In order to speed up the matching process, it is desirable that the feature amount calculation unit 21 performs parallel processing using a plurality of circuits.

The present invention can be used in the technical field of an apparatus that captures and plays back moving images.

10 camera system, 11 video camera, 12 image pickup unit, 13 image pickup device, 14 control unit, 15 CPU, 16 timing control unit, 20 image processing unit, 21 feature amount calculation unit, 22 function calculation unit, 23 gradient intensity calculation unit, 24 gradient direction calculation unit, 25 histogram generation unit, 26 histogram RAM, 27 sort unit, 28 binary code conversion unit, 30 image determination unit, 31 binary code register, 32 binary code RAM, 33 register, 34 binary code register, 35 binary Code RAM, 36 Matching determination unit, 36a Matching check unit, 36b Matching rate counting unit, 37 Matching rate RAM, 38 Sorting unit, 40 Computer, 41 Recording device, 50 Captured image , 51 cell, 52 pixels.

Claims

A calculation unit that acquires a signal of a part of the captured image captured by the imaging unit and calculates a feature amount including a gradient direction and gradient strength of the luminance of the local region;
The predetermined feature including the luminance gradient direction and gradient intensity obtained by imaging the predetermined object is matched with the calculated feature of the local region, and the predetermined object is detected in the captured image. A determination unit for determining whether or not
A control unit that starts recording the captured image when the determination unit determines that the predetermined object has entered the captured image;
An image processing apparatus.
The image processing apparatus according to claim 1, wherein the control unit stops recording of the captured image when the predetermined object is removed from the captured image by the determination unit.
The computing unit is a computing circuit that creates a histogram for each gradient direction divided into a plurality of directions and generates a binary code as the feature amount of the local region with a direction having a strong gradient strength as a representative value,
The image processing apparatus according to claim 1, wherein the determination unit is a determination circuit that performs the matching using a binary-coded standard feature value.
The determination unit performs the matching using a binary code including at least a first feature amount of the local region and a second feature amount at a position shifted by a predetermined amount with respect to the local region as the standard feature amount of the local region. The image processing apparatus according to claim 3.
5. The image processing device according to claim 1, wherein the determination unit detects the predetermined object within one frame after the object enters the captured image.
The determination unit includes a plurality of registers, stores the calculated feature values of the local region in each of the registers, and performs the matching by operating the stored feature values for each local region as a shift register. 6. The image processing apparatus according to any one of 1 to 5.
The image processing apparatus according to any one of claims 1 to 6, wherein there are a plurality of the calculation units, and the feature amounts are calculated in parallel with respect to a plurality of pixels.