JP7191706B2 - Information processing device, program, marker for image recognition - Google Patents

Description

The present invention relates to an information processing apparatus and the like.

For example, based on image information acquired by an imaging device installed in a factory or warehouse, etc., while recognizing objects inside the factory or warehouse (for example, workers, work equipment, products in the warehouse, etc.), A system for monitoring the inside of factories, warehouses, etc. is known (see, for example, Patent Document 1).

JP 2012-174223 A

By the way, in the case of a factory or a warehouse, depending on the installation environment, for example, there is a possibility that the orientation of the imaging device may change due to the influence of vibration or the like. For example, it is desirable to perform parameter corrections related to object recognition. Therefore, for example, by installing a predetermined marker in advance on a fixed part such as a wall or a pillar inside a factory or a warehouse, and monitoring changes in the position, shape, etc. of the marker in the image information of the imaging device, posture changes may be monitored.

However, depending on the environment of a factory or warehouse, for example, the illuminance environment may change significantly due to changes in natural light entering through windows depending on the time of day or changes in natural light entering through doors due to opening and closing of doors. There may be cases where proper recognition is not possible. In other words, a large change in the illuminance environment may reduce the accuracy of monitoring posture changes of the imaging device. Therefore, it is expected to improve the robustness of the monitoring function for changes in the posture of the imaging device against changes in the illuminance environment.

In view of the above problem, it is an object of the present invention to provide an information processing apparatus and the like capable of improving the robustness of a monitoring function with respect to changes in illuminance with respect to changes in posture of an imaging device.

To achieve the above object, in one embodiment of the present invention,
Based on a composite image generated from a plurality of captured images with mutually different exposure times of an imaging device that captures an imaging range in which a plurality of markers are set, the time of the positions of the plurality of markers in the captured image of the imaging device. a posture change measuring unit that measures a change in the posture of the imaging device by measuring the change;
An information processing device is provided.

In another embodiment of the invention,
information processing equipment,
Based on a composite image generated from a plurality of captured images with mutually different exposure times of an imaging device that captures an imaging range in which a plurality of markers are set, the time of the positions of the plurality of markers in the captured image of the imaging device. causing an attitude change measurement step of measuring a change in attitude of the imaging device to be performed by measuring the change;
A program is provided.

According to the above-described embodiments, it is possible to provide an information processing apparatus and the like capable of improving the robustness of the monitoring function with respect to changes in illuminance with respect to changes in posture of the imaging device.

It is a figure which shows the outline|summary of an example of the monitoring system which concerns on one Embodiment. It is a figure showing an example of hardware constitutions of a monitoring system concerning one embodiment. It is a functional block diagram showing an example of functional composition of a surveillance system concerning one embodiment. FIG. 4 is a diagram showing a first example of markers (marker set); FIG. 10 is a diagram showing a second example of markers; 4 is a flowchart schematically showing an example of posture change monitoring processing by a monitoring device; FIG. 3 is a diagram showing an example of a captured image group (a plurality of captured images) with mutually different exposure times acquired by a camera; 4 is a flowchart schematically showing an example of marker position measurement processing by a monitoring device; FIG. 4 is a diagram showing a process of extracting a marker area image from a captured image; FIG. 4 is a diagram for explaining a first example of a method of acquiring an image for marker position measurement; FIG. 10 is a diagram for explaining a second example of a method of acquiring an image for marker position measurement; It is a figure which shows an example of the change of the marker position on a captured image.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings.

[Summary of monitoring system]
First, an overview of a monitoring system 1 according to the present embodiment will be described with reference to FIG.

FIG. 1 is a diagram showing an overview of an example of a monitoring system (monitoring system 1) according to this embodiment.

As shown in FIG. 1 , a monitoring system 1 according to this embodiment includes a camera 10 and a monitoring device 20 .

A camera 10 (an example of an imaging device) is attached to a relatively high position such as a ceiling or a wall of a predetermined monitoring space 100 such as a factory or warehouse, for example. The camera 10 captures an image for monitoring a predetermined monitored object within the monitored space 100, and has a function of monitoring the monitored object within the monitored space 100 based on the image captured by the camera 10 (hereinafter referred to as a “space monitoring function”). ) to a predetermined external device. Objects to be monitored may include, for example, workers, vehicles, work machines (eg, cranes, forklifts), products awaiting shipment, and the like.

Also, the camera 10 is communicably connected to the monitoring device 20 through, for example, a communication cable, captures an image of the monitored space 100 under the control of the monitoring device 20 , and outputs the captured image to the monitoring device 20 . In addition, the camera 10 is connected to the monitoring device 20 by predetermined wireless communication (for example, mobile communication based on a mobile communication network with a base station as an end, or short-range communication such as WiFi or Bluetooth (both registered trademarks)). may be communicatively connected to the

A plurality of markers 30 are fixedly installed in the imaging range of the camera 10 in the monitored space 100 . For example, the plurality of markers 30 are installed on non-moving objects (fixed parts) such as walls and pillars in the monitored space 100 . That is, the captured image of the camera 10 includes (that is, appears in) a plurality of markers 30 .

A monitoring device 20 (an example of an information processing device) monitors posture changes of the camera 10 by monitoring temporal changes of the plurality of markers 30 included in the captured image of the camera 10 . Hereinafter, the function of monitoring the attitude change of the camera 10 by the monitoring device 20 may be referred to as "camera attitude change monitoring function". The monitoring device 20 may be installed in the building in which the monitored space 100 exists or in a building in the vicinity of the building, or may be installed in a remote location relatively far from the building in which the monitored space 100 exists. The monitoring device 20 is, for example, a desktop or laptop computer terminal. Also, the monitoring device 20 may be a server device arranged remotely from a facility such as a factory or a warehouse corresponding to the monitored space 100, for example.

Note that the monitoring device 20 may have a space monitoring function. That is, the space monitoring function and the camera attitude change monitoring function may be realized by different devices or may be realized by a common device.

[Configuration of monitoring system (monitoring device)]
Next, the configuration of the monitoring system 1 (monitoring device) according to the present embodiment will be described with reference to FIGS. 2 and 3 in addition to FIG.

FIG. 2 is a diagram showing an example of the hardware configuration of the monitoring device 20. As shown in FIG. FIG. 3 is a functional block diagram showing an example of the functional configuration of the monitoring system 1. As shown in FIG.

The functions of the monitoring device 20 may be realized by arbitrary hardware or a combination of hardware and software. As shown in FIG. 2B, for example, the monitoring device 20 includes a drive device 21, an auxiliary storage device 22, a memory device 23, a CPU 24, an interface device 25, a display device 26, and an input device 27. are connected by a bus B2.

A program that realizes various functions of the monitoring device 20 is stored in a portable memory such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. provided by the recording medium 21A. When the recording medium 21A recording the program is set in the drive device 21, the program is installed in the auxiliary storage device 22 from the recording medium 21A via the drive device 21. FIG. Alternatively, the program may be downloaded from another computer via a communication network and installed in the auxiliary storage device 22 .

The auxiliary storage device 22 stores various installed programs, as well as necessary files and data.

The memory device 23 reads the program from the auxiliary storage device 22 and stores it when a program activation instruction is received.

The CPU 24 executes various programs stored in the memory device 23 and implements various functions related to the monitoring device 20 according to the programs.

The interface device 25 is used as an interface for connecting to communication networks (for example, communication networks NW1 and NW2).

The display device 26 displays a GUI (Graphical User Interface) according to a program executed by the CPU 24, for example.

The input device 27 is used to allow a worker, administrator, or the like of the monitoring device 20 to input various operational instructions regarding the monitoring device 20 .

The monitoring device 20 includes, for example, an imaging control unit 201, a marker position measurement unit 202, and a camera attitude change function unit realized by executing one or more programs installed in the auxiliary storage device 22 on the CPU 24. It includes a measurement section 203 and an error detection section 204 .

The imaging control unit 201 outputs a control command to the camera 10 and causes the camera 10 to acquire a captured image.

The marker position measuring unit 202 measures the positions of the multiple markers 30 on the captured image of the camera 10 . Specifically, the marker position measurement unit 202 measures a predetermined representative position (for example, the center position of the shape portion) of the marker 30 on the captured image of the camera 10 .

A camera posture change measuring unit 203 (an example of a posture change measuring unit) measures the posture of the camera 10 based on the temporal changes in the positions of the markers 30 on the captured image of the camera 10, which are measured by the marker position measuring unit 202. Measure change.

When the space monitoring function is realized by the monitoring device 20, the camera posture change measuring unit 203 transmits the measurement result to the components (functional units) related to the space monitoring function. As a result, the component related to the space monitoring function determines whether it is necessary to correct the parameters for monitoring the monitoring target based on the image captured by the camera 10, based on the measurement results, and whether correction is necessary. In some cases, corrections can be made.

Also, when the space monitoring function is realized by an external device other than the monitoring device 20, the camera posture change measuring unit 203 transmits the measurement result to the external device. As a result, the external device determines whether or not it is necessary to correct the parameters for monitoring the monitoring target based on the image captured by the camera 10 based on the measurement result, and if correction is necessary, corrects the parameters. You can do

The error detection unit 204 detects a predetermined error related to the camera posture change measurement function (for example, an error that cannot measure the posture change of the camera 10 or an error that the measurement accuracy of the posture change of the camera 10 is relatively low). (measurement accuracy drop error, etc.) is detected.

[Specific examples of markers]
Next, a specific example of the marker 30 will be described with reference to FIGS. 4 (FIGS. 4A and 4B).

<First example of marker>
First, a first example of the marker 30 will be described with reference to FIG. 4A.

FIG. 4A is a diagram showing a first example of the marker 30. FIG.

As shown in FIG. 4A, marker 30 includes two markers 30A and 30B. That is, the marker 30 is composed of a pair of markers 30A and 30B. Hereinafter, the marker 30 of this example will be referred to as a "marker set 30" for convenience.

The marker 30A includes a shape portion 30Aa and a background portion 30Bb.

The shape portion 30Aa is an object of image recognition in the captured image captured by the camera 10 with the marker set 30 (marker 30A) as the subject. The shape portion 30Aa is configured by, for example, a geometric shape (a combination of four squares in this example). The shape portion 30Aa is configured to produce a relatively large luminance gradient (luminance difference) with the background portion 30Ab on the captured image captured by the camera 10 as the subject. In this example, it is composed of a relatively dark color, that is, a color with low lightness (black in this example). As a result, the shape portion 30Aa can be recognized as an object on the captured image captured by the camera 10 by contrast with the background portion 30Ab having a relatively light color, that is, a color with high brightness (white in this example). easier to be Also, the shape portion 30Aa is configured to have a relatively low degree of reflection. For example, the shape portion 30Aa may be made of a light-absorbing material, specifically, a light-absorbing paint or the like. As a result, it is possible to prevent the edge of the shape portion 30Aa from being blurred in the captured image of the camera 10 even in an environment with relatively high illuminance. In other words, the marker 30A is configured to be easily recognized as an image in an environment with relatively high illuminance.

Note that the number of geometric shapes such as squares combined as the shape portion 30Aa is
The number may be three or less, or may be five or more, as long as image recognition on the captured image is possible. The same applies to the second example of the marker 30B and the marker 30 below.

The background portion 30Ab configures the background of the shape portion 30Aa that is the object of image recognition. The background portion 30Ab is configured such that a relatively large luminance gradient (luminance difference) is generated between the background portion 30Ab and the shape portion 30Aa on the captured image captured by the camera 10 as a subject. In this example, the background portion 30Ab is composed of a relatively bright color (white in this example). In addition, the background portion 30Ab is configured to have a relatively high degree of reflection. For example, the shape portion 30Aa may be formed of a diffusely reflective material, specifically, a paint or the like having diffusely reflective properties. As a result, the boundary between the shape portion 30Aa and the background portion 30Ab, that is, the edge of the shape portion 30Aa can be clearly seen on the captured image of the camera 10, and the shape portion 30Aa can be more easily image-recognized.

Like the marker 30A, the marker 30B includes a shape portion 30Ba and a background portion 30Bb.

The shape portion 30Ba is an object of image recognition in the captured image captured by the camera 10 with the marker set 30 (marker 30B) as the subject. The shape part 30Ba is configured by, for example, a geometric shape (combination of four squares in this example), as in the case of the marker 30A. The shape portion 30Ba is configured to produce a relatively large luminance gradient (luminance difference) with the background portion 30Bb on the image captured by the camera 10 as the subject. In this example, the shape portion 30Ba is composed of a color with relatively high brightness (white in this example). As a result, the shape portion 30Ba is easily recognized as a subject in the captured image captured by the camera 10 due to the contrast with the background portion 30Bb having a relatively low brightness (black in this example). Further, the shape portion 30Ba is configured to have a relatively high degree of reflection. For example, the shape portion 30Ba may be formed of a diffusely reflective material, specifically, a paint or the like having diffusely reflective properties. This makes it possible to make the edge of the shape portion 30Aa stand out in the captured image of the camera 10 even in an environment with relatively low illuminance. In other words, the marker 30A is configured to be easily recognized as an image in an environment with relatively low illuminance.

The background portion 30Bb constitutes the background of the shape portion 30Ba that is the object of image recognition. The background portion 30Bb is configured such that a relatively large luminance gradient (luminance difference) is generated between the background portion 30Bb and the shape portion 30Ba on the captured image captured by the camera 10 as a subject. In this example, the background portion 30Bb is composed of a color with relatively low brightness (black in this example). Also, the background portion 30Bb is configured to have a relatively low degree of reflection. For example, the shape portion 30Ba may be formed of a light absorbing material, specifically, a light absorbing paint or the like. As a result, the boundary between the shape portion 30Ba and the background portion 30Bb, that is, the edge of the shape portion 30Ba can be clearly seen on the captured image of the camera 10, and the shape portion 30Ba can be more easily recognized as an image.

Thus, in this example, the marker set 30 is composed of two markers 30A and 30B with different degrees of reflection. That is, the marker set 30 consists of a marker 30A, which is easily recognized in the captured image captured by the camera 10 in an environment with relatively high illuminance, and a captured image captured by the camera 10 in an environment with relatively low illuminance. The marker 30B is composed of a marker 30B whose image can be easily recognized by As a result, the markers 30A and 30B are installed on fixed portions such as walls and pillars of the monitored space 100 in a predetermined relative positional relationship, so that even in a situation where the illuminance environment of the monitored space 100 changes greatly, The possibility that one of the markers 30A and 30B will be image-recognized increases. That is, the markers 30A and 30B can improve the robustness of the image recognition processing of the markers 30A and 30B by the monitoring device 20 with respect to changes in the illuminance environment.

Note that the marker set 30 may include three or more markers with different degrees of reflection.

<Second example of marker>
Next, a second example of the marker 30 will be described with reference to FIG. 4B.

FIG. 4B is a diagram showing a second example of the marker 30. As shown in FIG.

As shown in FIG. 4B, the marker 30 includes a shape portion 30a, a background portion 30b, and a change portion 30c.

The shape portion 30a is an object of image recognition in the captured image captured by the camera 10 with the marker 30 as the subject. The shape part 30a is configured by, for example, a geometric shape (a combination of four squares in this example). The shape portion 30a is configured to produce a relatively large luminance gradient (luminance difference) with the background portion 30b on the captured image captured by the camera 10 as the subject. In this example, it is composed of a color with relatively low lightness (black in this example). As a result, the shape portion 30a is easily recognized as a subject in the captured image captured by the camera 10 due to the contrast with the background portion 30b having a relatively bright color (white in this example).

The background portion 30b constitutes the background of the shape portion 30a, which is the object of image recognition. The background portion 30b is configured such that a relatively large luminance gradient (luminance difference) is generated between the background portion 30b and the shape portion 30a on the captured image captured by the camera 10 as a subject. In this example, the background portion 30b is composed of a relatively bright color (white in this example).

The changing part 30c is a component that changes the appearance of the captured image captured by the camera 10 with the marker 30 as the subject depending on the illuminance environment of the imaging range (monitoring space 100), the exposure time of the camera 10, and the like. In this example, the changing section 30c includes changing sections 30c1 and 30c2.

The changing portion 30c1 is provided at the upper end portion of the marker 30 (above the shaped portion 30a). The changing portion 30c1 changes from left to right on a background of the same color (that is, black) as the shape portion 30a from the color of the background portion 30b (that is, white) to the color of the shape portion 30a (that is, black). is a gradation bar representing a linear gradient that

The changing portion 30c2 is provided at the lower end portion of the marker 30 (below the shape portion 30a). Similar to the changing portion 30c1, the changing portion 30c2 is formed on a background of the same color (that is, white) as that of the background portion 30b, from left to right, from the color of the background portion 30b (that is, white) to the color of the shape portion 30a (that is, white). That is, a gradation bar that represents a linear gradient that changes to black.

Either one of the changing portions 30c1 and 30c2 may be omitted.

The appearance of the gradation bar representing the linear gradation between the shape portion 30a and the background portion 30b on the captured image of the camera 10 changes depending on the illuminance environment of the imaging range and the exposure time of the camera 10. FIG. Therefore, as will be described later, the monitoring device 20 (marker position measuring unit 202) determines how the changing portion 30c (gradation bar) looks for each of a plurality of captured images with mutually different exposure times. It is possible to judge whether or not the appearance is suitable for

As described above, in this example, the marker 30 is provided with the changing portion 30c whose appearance on the captured image of the camera 10 changes according to the change in the exposure time of the camera 10 . As a result, the marker 30 can select a captured image suitable for image recognition of the marker 30 from a plurality of captured images with mutually different exposure times. Therefore, the marker 30 can improve the robustness of the image recognition processing of the markers 30A and 30B by the monitoring device 20 with respect to changes in the illuminance environment.

[Overview of measurement method for changes in camera posture]
Next, with reference to FIGS. 5 and 6, the outline of the method for measuring the attitude change of the camera 10 will be described.

FIG. 5 is a flowchart schematically showing an example of processing for monitoring posture change of the camera 10 by the monitoring device 20 (hereinafter referred to as “posture change monitoring processing”). FIG. 6 is a diagram showing an example of a group of captured images (a plurality of captured images) with mutually different exposure times acquired by the camera 10. As shown in FIG.

As shown in FIG. 5, in step S102, the imaging control unit 201 outputs a control command to the camera 10 to generate a plurality of captured images (for example, 3 to 5 captured images) with mutually different exposure times. The image is captured by the camera 10, and the process proceeds to step S104.

For example, as shown in FIG. 6, the camera 10 acquires three captured images 610-630 with mutually different exposure times. Thereby, the camera 10 can acquire the captured images 610 to 630 in which the marker 30 looks different.

Returning to FIG. 5, in step S104, the marker position measurement unit 202 determines the positions of the markers 30 (that is, initial position) is measured (marker position measurement process), and the process proceeds to step S106. Details of the marker position measurement process will be described later.

In step S106, the marker position measurement unit 202 saves (stores) the measured positions of the plurality of markers 30 as initial positions in the auxiliary storage device 22 or the like, and proceeds to step S108.

In step S<b>108 , the imaging control unit 201 determines whether or not the timing for measuring the attitude change of the camera 10 (hereinafter, simply “measurement timing”) has arrived. The measurement timing may be, for example, a predetermined time such as every day or every predetermined day. If the measurement timing has not arrived, the imaging control unit 201 waits until the measurement timing arrives, and if the measurement timing has arrived, the process proceeds to step S110.

In step S110, the imaging control unit 201 outputs a control command to the camera 10 to cause the camera 10 to capture a plurality of captured images with mutually different exposure times (see FIG. 6), as in step S102. move on.

In step S112, the marker position measurement unit 202 measures the positions (that is, the current positions) of the multiple markers 30 in the captured images based on the multiple captured images captured by the camera 10 with different exposure times. Processing marker position measurement processing is performed, and the process proceeds to step S114.

In step S114, the camera posture change measuring unit 203 measures the current positions of the plurality of markers 30 in the captured image measured in step S112 and Based on the initial positions of the plurality of markers 30, the change in posture of the camera 10 is measured, and the process proceeds to step S116. Further, if the process of this step has been performed in the past, the camera posture change measuring unit 203 determines the current positions of the plurality of markers 30 in the captured image measured in step S112 this time and the current positions measured in step S112 last time. The posture change of the camera 10 may be measured based on the past positions of the plurality of markers 30 in the captured image.

In step S116, the error detection unit 204 determines whether or not various errors have been detected. The error detector 116 proceeds to step S118 if an error is detected, and returns to step S108 if no error is detected.

In step S118, the error detection unit 204 outputs an alert through the display device 26 or the like, and ends the current process.

As described above, the monitoring device 20 measures changes in the posture of the camera 10 at predetermined measurement timings, and implements a camera posture change monitoring function, unless an error is detected.

[Details of marker position measurement processing]
Next, details of the marker position measurement process (steps S104 and S112 in FIG. 5) will be described with reference to FIGS. 7 to 9 (FIGS. 9A and 9B).

FIG. 7 is a flowchart schematically showing an example of marker position measurement processing by the monitoring device 20. As shown in FIG. FIG. 8 is a diagram showing a process of extracting a local image (hereinafter referred to as "marker area image") of the area of the marker set 30 (markers 30A and 30B) from the captured image of the camera 10. As shown in FIG. 9A and 9B are diagrams for explaining a first example and a second example, respectively, of a method of acquiring an image for measuring the position of the marker set 30 (hereinafter referred to as "marker position measurement image").

<First example of marker position measurement processing>
First, a first example of marker position measurement processing will be described with reference to FIGS. 7, 8, and 9A. Specifically, marker position measurement processing when the marker set 30 of FIG. 4A described above is employed will be described.

As shown in FIG. 7, in step S202, the marker position measuring unit 202 obtains an image of an area assumed to include each of the plurality of marker sets 30 (hereinafter referred to as " Marker Global Image”) is extracted. For example, the marker position measurement unit 202 extracts (that is, cuts out) the marker global image along the coordinate range of the captured image defined in advance for each of the multiple marker sets 30 . This is because, since the installation locations of the plurality of marker sets 30 are known, it is possible to roughly define the positions where the plurality of marker sets 30 are included on the captured image. As a result, a plurality of global marker images with mutually different exposure times are acquired for each of the plurality of marker sets 30 .

In step S204, the marker position measuring unit 202 extracts an image of an area substantially occupied by the marker set 30 from the marker global image (hereinafter referred to as "marker area image ”). At this time, the marker position measurement unit 202 does not need to perform a method such as shape measurement for each of the plurality of images with mutually different exposure times. Specifically, the marker position measurement unit 202 extracts a marker area image by shape measurement or the like from one captured image suitable for extracting a marker area image, and for other captured images, the same coordinate range is used as the marker area image. should be extracted as As a result, a plurality of marker area images with mutually different exposure times are acquired for each of the plurality of marker sets.

For example, as shown in FIG. 8, the marker position measurement unit 202 extracts a marker global image 820 by cutting out a predefined coordinate range 811 from the captured image 810 (step S202). Then, the marker position measurement unit 202 applies a method such as shape measurement, cuts out the region 821 substantially occupied by the marker 30 (marker set 30), and extracts the marker region image 830 (step S204).

Returning to FIG. 7, in step S206, the marker position measurement unit 202 measures the positions of the markers 30 out of the plurality of marker area images with mutually different exposure times for each of the plurality of markers 30A and the plurality of markers 30B. A marker area image (image for marker position measurement) most suitable for is acquired. Specifically, the marker position measurement unit 202 calculates the positions of the markers 30A and 30B on the captured images of the markers 30A and 30B based on the statistical information and the shape information regarding the brightness of the markers 30A and 30B for each of a plurality of marker area images having mutually different exposure times. A feature amount for determining ease of recognition is acquired, and an image for marker position measurement is acquired based on the acquired feature amount.

For example, as shown in FIG. 9A, when marker region images 910, 920, and 930 of marker 30A with mutually different exposure times are acquired, marker position measurement unit 202 sets image histograms (brightness histograms) corresponding to each of them. Calculate 915,925,935. Then, the marker position measurement unit 202 acquires images for marker position measurement based on the image histograms 915 , 925 and 935 . For example, the marker position measurement unit 202 acquires, as the marker position measurement image, the marker region image corresponding to the histogram that minimizes the sum of the products of the x-coordinate corresponding to the brightness of the histogram and the y-coordinate corresponding to the number of pixels. You can

Returning to FIG. 7, in step S208, the marker position measurement unit 202 measures the positions of the markers 30A and 30B for each of the multiple markers 30A and multiple markers 30B based on the acquired marker position measurement image. For example, the marker position measurement unit 202 detects a pattern of a marker position measurement image and digital data (for example, CAD (Computer Aided Design) data) representing the shapes of the markers 30A and 30B prepared in advance in the auxiliary storage device 22 or the like. By matching, the center positions of the shape portions 30Aa and 30Ba of the markers 30A and 30B in the image captured by the camera 10 (image for marker position measurement) may be measured. Thereby, the marker position measuring section 202 can measure the positions of the markers 30A and 30B (center positions of the shape sections 30Aa and 30Ba) with higher accuracy.

Note that when the marker set 30 of FIG. 4A described above is used, a marker global image and a marker region image (marker position measurement image) are acquired based on one captured image acquired with a standard exposure time. good too. Since the marker set 30 includes a plurality of markers with different illuminance environments that are easily image-recognizable, even in a situation where the illuminance environment can change significantly depending on the timing at which the captured image is acquired, marker position measurement can be performed. This is because there is a high possibility that the unit 202 can appropriately recognize any one of the markers included in the marker set 30 , and as a result, the position of the marker set 30 can be recognized.

<Second example of marker position measurement processing>
Next, a second example of marker position measurement processing will be described with reference to FIGS. 7, 8, and 9B.

The processing in steps S202 and S204 in FIG. 7 is the same as in the first example described above, so description thereof will be omitted.

In step S<b>206 , the marker position measurement unit 202 acquires (generates) a composite image for marker position measurement from a plurality of marker area images with mutually different exposure times for each of the plurality of markers 30 .

For example, as shown in FIG. 9B, when marker region images 940, 950, and 960 with mutually different exposure times are acquired, the marker position measurement unit 202 performs HDR (High Dynamic Range) analysis from the marker region images 940, 950, and 960. Range) image 970 is acquired. As a result, the marker position measuring unit 202 can make the features of the marker 30 stand out. This makes it easier to recognize the position of the marker 30, and the accuracy of position measurement of the marker 30 can be improved.

The marker position measurement unit 202 may generate a composite image from all marker area images with mutually different exposure times, or may select a plurality of marker area images with mutually different exposure times for image recognition of the marker 30 . A composite image may be generated by extracting two or more marker area images that are suitable, that is, in which the markers 30 are included in a manner that is relatively easy to image recognize. In the latter case, the marker position measurement unit 202 extracts a marker area image suitable for image recognition of the marker 30 based on how the changing portion 30c of the marker 30 appears on the marker area image. Specifically, the marker position measurement unit 202 performs image recognition of the marker 30 by comparing the changed portion 30c on the marker area image with reference image data corresponding to the appearance of the changed portion 30c suitable for image recognition. A suitable marker region image may be extracted.

Returning to FIG. 7, the processing of step S208 is the same as that of the first example described above, so the description thereof is omitted.

Note that when the marker 30 of FIG. 4B described above is used, the marker position measurement unit 202 may generate an HDR image. In other words, the marker position measurement unit 202 directly uses the captured image most suitable for image recognition (that is, position measurement) of the marker 30, which is selected from a plurality of marker area images with mutually different exposure times. The position of marker 30 may be measured.

[Details of Camera Posture Change Measurement Processing]
Next, with reference to FIG. 10, details of the camera attitude change measurement process (step S114 in FIG. 5) will be described. Hereinafter, in this description, the marker set 30 in FIG. 4A and the marker in FIG. 4B will be collectively referred to as the marker 30, and the description will proceed.

FIG. 10 is a diagram showing an example of changes in the position of the marker 30 on the captured image.

The camera posture change measurement unit 203 measures the current position and the past position (for example, the initial position acquired in step S104 of FIG. 5 or the previous step Based on the position of the marker 30 acquired in S112, the change in posture of the camera 10 is measured. Specifically, the camera posture change measuring unit 203 measures the vertical displacement (hereinafter referred to as “x displacement”) dx of the camera 10, the horizontal displacement (hereinafter referred to as “y displacement”) dy, and the roll direction rotational displacement ( Hereafter, "roll displacement") r, rotational displacement in the yaw direction (hereinafter, "yaw displacement") yaw, and rotational displacement in the pitching direction (hereinafter, "pitch displacement") pitch are measured.

For example, as shown in FIG. 10, in the captured image 1000 of the camera 10, the upper left marker 30 moves from the pixel coordinates (x ₀ , y ₀ ) to the pixel coordinates (X ₀ , Y ₀ ) (in the lower left direction). ) and the lower right marker 30 is displaced from the pixel coordinates (x ₁ , y ₁ ) to the pixel coordinates (X ₁ , Y ₁ ) (in the lower right direction), the roll displacement r of the camera 10, The x-displacement dx, y-displacement dy, yaw displacement yaw, and pitch displacement pitch are expressed by the following equations (1) to (5).

Note that _{px in equation (4) and p y in} equation (5) represent pixel sizes in the vertical direction (longitudinal direction) and horizontal direction (horizontal direction) of the image captured by the camera 10, respectively. Also, f in equations (4) and (5) represents the lens focal length of the camera 10 . Also, m in formulas (1) to (3) is represented by the following formula (6).

Further, when three or more markers 30 are installed, the camera posture change measurement unit 203 may set a plurality of combinations of two target markers and measure the change in posture of the camera 10 for each combination. In this case, the camera posture change measuring unit 203 performs statistical processing (for example, clustering processing, average value calculation processing, median value calculation processing, etc.) on the measurement results for each combination to obtain the posture of the camera 10. can be calculated.

[Action of this embodiment]
Next, the operation of the monitoring system 1 (monitoring device 20) and the marker (marker set) 30 according to this embodiment will be described.

In this embodiment, the camera posture change measuring unit 203 has an imaging range in which a plurality of fixed marker sets 30 including a plurality of markers (markers 30A and 30B) that are relatively easy to image-recognize under mutually different illuminance environments are installed. The change in the posture of the camera 10 may be measured by measuring the temporal change in the positions of the plurality of marker sets 30 in the captured image of the camera 10 based on the captured image of the camera 10 that captures the .

As a result, the monitoring device 20 selects, for each of the plurality of marker sets 30, a marker suitable for image recognition in the illuminance environment of the imaging range when the captured image was acquired from among the plurality of markers included in the marker set 30. can do. Therefore, the monitoring device 20 recognizes any one of the plurality of markers and detects the presence of the plurality of markers even in a situation where the illuminance environment of the imaging range changes relatively greatly at each acquisition timing of the captured image. The position change of the camera 10 can be measured by the change in the position of the set 30 over time. Therefore, the monitoring device 20 can improve the robustness of the camera attitude change monitoring function with respect to changes in illuminance.

In addition, in this embodiment, the camera posture change measuring unit 203 measures temporal changes in the positions of the plurality of marker sets 30 based on an image captured by the camera 10 that has an exposure time different from that of the image captured by the camera 10 for the space monitoring function. A change in pose of the camera 10 may be measured by measuring .

As a result, the monitoring device 20 can measure changes in the posture of the camera 10 using captured images with exposure times suitable for image recognition of the plurality of marker sets 30 in the camera posture change monitoring function. Therefore, the monitoring device 20 can further improve the robustness of the camera attitude change monitoring function against changes in illuminance.

In addition, in this embodiment, the camera posture change measurement unit 203 calculates a plurality of marker sets in the captured image of the camera 10 based on a plurality of captured images with mutually different exposure times, which are acquired by the camera 10 at predetermined timings. Thirty positions may be measured over time.

As a result, the monitoring device 20 uses captured images suitable for image recognition of the marker set 30 in the illuminance environment of the imaging range at the timing when the captured images are acquired, which are selected from captured images with mutually different exposure times. can be used to measure changes in the attitude of the camera 10 . Therefore, the monitoring device 20 can further improve the robustness of the camera attitude change monitoring function against changes in illuminance.

In addition, in the present embodiment, the marker position measuring unit 202 selects one or more of the plurality of markers (markers 30A and 30B) included in the marker set 30 that are relatively easy to image-recognize in the image captured by the camera 10. The positions of the marker sets may be determined based on the positions of the markers in the .

As a result, the monitoring device 20 specifically measures the change in the posture of the camera 10 using the captured image suitable for image recognition of the marker set 30 in the illumination environment of the imaging range at the timing when the captured image was acquired. be able to.

In addition, in the present embodiment, the marker position measuring unit 202 uses the histogram of the image portion corresponding to each of the plurality of markers (markers 30A and 30B) included in the marker set 30 in the image captured by the camera 10 to determine the relative position. One or two or more markers that are easily image-recognizable are extracted. Then, the marker position measurement unit 202 measures the position of the marker set based on the positions of one or more markers extracted from the plurality of markers included in the marker set 30 in the captured image of the camera 10 .

As a result, the monitoring device 20 selects an image suitable for image recognition of the marker set 30 in the illuminance environment of the imaging range at the timing at which the captured image was acquired, from among the plurality of captured images having mutually different exposure times. Images can be extracted.

In addition, in the present embodiment, the marker set 30 includes a plurality of markers (markers 30A and 30B) with different degrees of reflection.

As a result, the marker set 30 can change the easily recognizable marker in the image captured by the camera 10 as the subject in accordance with the change in the illuminance environment at the installation location. Therefore, the marker set 30 can be easily recognized as a subject in the captured image captured by the camera 10 even in a situation where the illuminance environment at the installation location changes relatively greatly. Therefore, the marker set 30 can further improve the robustness of the camera posture change monitoring function against illumination changes.

Further, in the present embodiment, two markers (markers 30A and 30B) among the plurality of markers included in the marker set 30 are the shape portion to be image-recognized and the background portion forming the background of the shape portion. Colors are inverted.

As a result, the marker set 30 can specifically differentiate the degree of reflection between the two markers.

Further, in the present embodiment, two markers (markers 30A and 30B) out of the plurality of markers included in the marker set 30 are mutually associated with the shape portion to be image-recognized and the background portion forming the background of the shape portion. The relative reflection between and is inverted.

As a result, the marker set 30 can specifically differentiate the degree of reflection between the two markers.

In addition, in the present embodiment, the camera posture change measuring unit 203 measures a plurality of different exposure times obtained at predetermined timings of the camera 10 that captures an imaging range in which the plurality of markers 30 are fixedly installed. A change in the attitude of the camera 10 may be measured by measuring changes over time in the positions of the plurality of markers 30 in the image captured by the camera 10 based on the image captured by the camera 10 .

As a result, the monitoring device 20 selects, for example, a captured image in which the features of a plurality of markers can be easily recognized from a plurality of captured images having different exposure times, or generates a composite image in which the features of a plurality of markers can be easily recognized. can be Therefore, the monitoring device 20 can appropriately recognize the plurality of markers even in a situation where the illuminance environment changes greatly at each captured image acquisition timing, and can A change in the pose of the camera 10 can be measured. Therefore, the monitoring device 20 can improve the robustness of the camera attitude change measurement function with respect to changes in illuminance.

Further, in the present embodiment, the camera posture change measuring unit 203 calculates the positions of the plurality of markers 30 in the captured image of the camera 10 based on a composite image (HDR image) generated from a plurality of captured images with mutually different exposure times. The change in pose of the camera 10 may be measured by measuring the time change in .

As a result, the monitoring device 20 can detect a plurality of markers generated from a plurality of captured images with different exposure times even in a situation where the illuminance environment changes significantly at each captured image acquisition timing. A plurality of markers can be appropriately recognized using a synthesized image whose features are easily recognized.

In addition, in this embodiment, the camera posture change measuring unit 203 selects from two or more captured images in which the plurality of markers 30 are included in a manner in which the plurality of markers 30 are relatively easily recognized among the plurality of captured images with mutually different exposure times. Based on the generated composite image (HDR image), temporal changes in the positions of the multiple markers 30 in the image captured by the camera 10 may be measured.

As a result, the monitoring device 20 can use a composite image (HDR image) that makes it easier to recognize the features of the markers. Therefore, the monitoring device 20 can further improve the robustness of the camera attitude change measurement function with respect to changes in illuminance.

Further, in the present embodiment, each of the plurality of markers 30 includes a changing portion whose appearance in the captured image of the camera 10 changes according to changes in the exposure time, in addition to the shape portion targeted for image recognition. Then, the marker position measurement unit 202 determines the relative positions of the markers 30 from among the plurality of captured images having different exposure times based on how the changing portions of the plurality of markers 30 appear in the plurality of captured images. Captured images that are included in an easily recognizable manner may be extracted.

As a result, the monitoring device 20 specifically determines whether or not the plurality of markers 30 are likely to be image-recognized based on the appearance of the changed portion in the captured image of the camera 10, and determines whether the captured image is suitable for generating a composite image. can be extracted.

In addition, in this embodiment, the camera posture change measuring unit 203 measures temporal changes in the positions of the markers 30 based on an image captured by the camera 10 that has an exposure time different from that of the image captured by the camera 10 for the space monitoring function. By measuring, a change in pose of the camera 10 may be measured.

As a result, the monitoring device 20 can measure changes in the attitude of the camera 10 using captured images with exposure times suitable for image recognition of the plurality of markers 30 in the camera attitude change monitoring function. Therefore, the monitoring device 20 can further improve the robustness of the camera attitude change monitoring function against changes in illuminance.

In addition, in the present embodiment, the marker position measurement unit 202 generates digital data (CAD data) representing the shapes of the markers 30 based on a composite image (HDR image) generated from a plurality of captured images with mutually different exposure times. ) may measure the positions of the plurality of markers 30 in the image captured by the camera 10 .

As a result, the monitoring device 20 can more accurately measure the positions of the markers 30 on the image captured by the camera 10 . Therefore, the monitoring device 20 can improve the measurement accuracy of the change in posture of the camera 10 .

Further, in the present embodiment, the marker 30 includes a shape portion 30a to be image-recognized, and a changing portion 30c whose appearance on the captured image changes according to changes in the exposure time of the camera 10. FIG.

As a result, the marker 30 can be picked up from a plurality of images captured by the camera 10 with mutually different exposure times based on how the changing portion in the image captured by the camera 10 appears. The image can be recognized by the monitoring device 20 . Therefore, the marker 30 can improve the robustness of the camera posture change monitoring function of the monitoring device 20 against changes in illuminance.

Also, in the present embodiment, the changing portion 30c is a gradation bar that represents a linear gradation between the color of the shape portion 30a and the color of the background portion 30b.

Thus, the marker 30 can specifically change the appearance of the changing portion 30c in the captured image of the camera 10 in accordance with the change in the exposure time of the camera 10 .

Although the embodiments for carrying out the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various can be transformed or changed.

1 Surveillance System 10 Camera 20 Surveillance Device (Information Processing Device)
30 marker set (marker set for target image recognition)
30A, 30B marker 201 imaging control unit 202 marker position measurement unit 203 camera posture change measurement unit (posture change measurement unit)
204 error detector

Claims (7)

Positions of the plurality of markers in the captured image of the imaging device based on a plurality of captured images with mutually different exposure times acquired at predetermined timings by an imaging device that captures an imaging range in which a plurality of markers are installed. a posture change measuring unit that measures a change in the posture of the imaging device by measuring a time change of
Information processing equipment. The posture change measuring unit measures changes in the positions of the plurality of markers in the captured image of the imaging device over time, based on a composite image generated from the plurality of captured images, thereby measuring changes in the posture of the imaging device. to measure the
The information processing device according to claim 1 . The posture change measuring unit measures the position of the imaging device based on the composite image generated from two or more captured images in which the plurality of markers are included in the plurality of captured images in such a manner that the images are relatively easily recognized. measuring changes over time in the positions of the plurality of markers in the captured image;
The information processing apparatus according to claim 2. further comprising a captured image extracting unit that extracts, from among the plurality of captured images, a captured image containing the plurality of markers in such a manner that the images are relatively easily recognized;
each of the plurality of markers includes, in addition to the shape portion to be image-recognized, a changing portion whose appearance in the captured image of the imaging device changes according to a change in exposure time;
The captured image extraction unit extracts two or more captured images from the plurality of captured images based on how the changed portions of the plurality of markers appear in the plurality of captured images.
The information processing apparatus according to claim 3. The imaging device is provided for monitoring a predetermined monitoring target within the imaging range,
The posture change measuring unit measures a change in the posture of the imaging device based on an image captured by the imaging device that has an exposure time different from that of the image captured by the imaging device for monitoring the monitoring target.
The information processing apparatus according to any one of claims 1 to 4. further comprising a marker position measurement unit that measures the positions of the plurality of markers in the captured image of the imaging device based on the synthesized image;
The marker position measurement unit measures the positions of the plurality of markers in the captured image of the imaging device by pattern matching for digital data representing the shapes of the plurality of markers.
The information processing apparatus according to any one of claims 2 to 4 . information processing equipment,
Positions of the plurality of markers in the captured image of the imaging device, based on a plurality of captured images with different exposure times acquired at predetermined timings by an imaging device that captures an imaging range in which the plurality of markers are installed. causing an attitude change measurement step of measuring a change in the attitude of the imaging device by measuring a time change of
program.

Images