JP7219915B2 - Tracking lighting device and lighting tracking system - Google Patents

Description

The present invention relates to a tracking lighting device and a lighting tracking system, and more particularly, to a device for lighting while tracking a specific person in an indoor reception hall, etc., and a device for tracking lighting for a specific person in an indoor venue. It is about the system.

Technology for tracking a tracked object such as a moving person is one of the important issues in computer vision, such as surveillance systems and automobile safety technology. 1), a method of tracking with a particle filter using the likelihood of a human head model (see Non-Patent Documents 2 and 3), or a method of tracking with a particle filter using an appearance model by hue or HoG (Non-Patent Reference 4).

Visvanathan Ramesh: “Real-Time Tracking of Non-Rigid Objects using Mean Shift”, Proceedings IEEE Conference on Computer Vision and Pattern Recognition 2000, (2000) Tatsuya Suzuki: “Stabilization of Human Tracking by Introduction of Environment Model”, Transactions of the Institute of Electronics, Information and Communication Engineers, Vol.J88-D-II, No.8, pp.1592-1600, (2005) Takanori Kobayashi: "Human 3D Tracking by Integration of Particle Filter and Cascade Classifier", Transactions of the Institute of Electronics, Information and Communication Engineers, Vol.J90-D-IIm, No.8, pp.1-10, (2007) Sangeun Lee: “Human Tracking using Particle Filter with Reliable Appearance Model”, SICE Annual Conference 2013, pp.1418-1424,(2013)

However, since the technique disclosed in Non-Patent Document 1 uses pixel values to calculate the distance to the template, it is vulnerable to sudden illumination changes and occlusion. In addition, although the techniques disclosed in Non-Patent Documents 2 and 3 can reduce the influence of occlusion, they cannot track a person who has characteristics as a whole, and furthermore, pixel values are used to evaluate hypotheses. Therefore, there was a possibility that the image would be lost in an environment where the lighting changes rapidly. Furthermore, the technology disclosed in Non-Patent Document 4 is similar to Non-Patent Documents 2 and 3 in that the influence of occlusion can be reduced. It has to be rebuilt each time, and depending on the feature amount, there is a possibility that it will be lost in an environment where the lighting changes abruptly.

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and an object of the present invention is to provide a tracking lighting device and lighting tracking system capable of coping with changes in lighting while reducing the influence of occlusion. .

Accordingly, the present invention relating to a tracking illumination device comprises: image acquisition means; processing means for determining an object to be tracked from the image acquired by the image acquisition means; and illumination means for illuminating while changing the irradiation direction, the processing means includes a target rectangle detection section for detecting a target rectangle for a specific frame input from the image acquisition means. , a likelihood calculation unit that calculates a rectangle likelihood for each target rectangle based on the output value of the convolutional neural network structure for this target rectangle, and a likelihood map is created from a set of the target rectangle position and the rectangle likelihood. a likelihood map creation unit; and a determination unit that determines a target rectangle having the highest likelihood in the likelihood map as a target rectangle of the tracking target, wherein the illumination means determines the tracking target determined by the determination unit. The illumination direction is appropriately changed so that the target rectangle of the object is positioned at the center of the illumination area.

According to the above configuration, the target rectangle is detected by the target rectangle detection unit, the likelihood calculation unit calculates the likelihood for the target rectangle, and the likelihood map creation unit creates the likelihood map. It becomes possible to determine the target rectangle with the highest likelihood in the map as the tracking target.

The tracking illumination device of the present invention having the configuration described above further includes a tracking target movement estimating unit, and the movement estimating unit uses a particle filter to determine the target rectangle of the tracking target determined by the determination unit. A tracking estimate may be made.

In the case of the above configuration, it is possible to estimate the target rectangle of the tracked object according to the likelihood of particles, thereby reducing the influence of occlusion.

Further, in the invention relating to the tracking type illumination device having each of the configurations described above, the image acquisition means includes a drive unit for changing the image acquisition direction so that the target rectangle of the tracking target is positioned at the center of the acquired image. , the determination unit determines that the ratio of the sum of likelihoods of the target rectangle of the tracking target to the sum of likelihoods in all regions in the likelihood map created by the likelihood map creation unit or the latest likelihood map is Lost determination is performed on the basis of being equal to or less than a predetermined value, and the movement estimating unit regards the target rectangle located in the center of the image as the target rectangle of the tracking object when the determining unit determines that the movement is lost. motion estimation.

In the case of the above configuration, since the determination unit performs the lost determination, when the lost determination is made, the movement estimation can be continued with the target rectangle provisionally positioned in the center of the image as the tracking target. Here, the ratio of the sum of the likelihoods of the target rectangle of the tracking target to the sum of the likelihoods of the entire area in the likelihood map means that the likelihood corresponding to approximately 10% of the sum of the inductions of the entire area is tracked. It is preferable that it is the sum of the target rectangles of the object, and 1/10 of the lower limit may be set as the lower limit, and if the sum of likelihoods is less than that, it may be determined as lost. In addition, since the image acquisition means is provided with a driving unit, the target rectangle of the tracking target is arranged in the center of the acquired image. Therefore, when the lost determination is made, the target rectangle in the center of the acquired image is estimated as the tracking target. It is possible to continue tracking. As a result, it is possible to reduce the effects of ambient lighting changes and occlusion.

Further, in the invention relating to the tracking lighting device having each of the configurations described above, the likelihood map creation unit generates the latest likelihood updated by the average value of the likelihood maps in the past four frames and the likelihood map in the specific frame. It can be created as a map.

According to the configuration as described above, the likelihood map is updated by the average value referring to the past four frames, so the target rectangle with the highest likelihood in the likelihood map is continuous from the past frame. , can improve the accuracy of the likelihood represented in the latest likelihood map.

On the other hand, the present invention relating to a lighting tracking system is a lighting tracking system for an indoor venue using the tracking type lighting device according to any one of the configurations described above, wherein the tracking target is a person having a characteristic appearance. wherein the processing means comprises a storage unit for storing teacher data relating to outward features shared by the person and movement trajectories of the person; and the determination unit is created by the likelihood map creation unit. When the sum of the likelihoods of the target rectangles of the tracking target in the likelihood map or the latest likelihood map is equal to or less than a predetermined value, the target rectangle of the tracking target is specified by comparison with teacher data relating to the person. It is characterized by

According to the above configuration, for example, when the lighting is tracked in a wedding reception hall, the object to be illuminated (for example, the bride and groom) has an external feature that can be clearly distinguished from others. , it is possible to store a plurality of tracking targets (for example, a bride and groom) as teacher data and compare them with the teacher data to increase the likelihood of the target rectangle of the tracking target. Furthermore, even if the sum of the likelihoods of the target rectangles in the likelihood map is low, it is possible to determine the tracking target by comparing it with teacher data relating to its appearance features. This makes it possible to reduce the effects of ambient lighting changes and occlusion.

Further, the present invention related to a lighting tracking system is a lighting tracking system in an indoor venue using the tracking type lighting device according to any one of the above configurations, wherein the image acquiring means is installed integrally with the lighting means. and the center of the obtained image is aligned with the center of the irradiation range irradiated by the lighting means, the lighting means is capable of switching between an automatic tracking state and a manual tracking state, The movement estimating unit is characterized in that, of the input image input immediately after switching from the manual tracking state to the automatic tracking state, the target rectangle located in the center is regarded as the target rectangle of the tracking target and determined. It is something to do.

According to the above configuration, the orientation of the image acquisition means is changed in accordance with the change in the direction of irradiation by the illumination means, and the center of the acquired image coincides with the center of the irradiation range. In this state, if the tracking target is positioned at the center of the range illuminated by the illumination means, the tracking target will inevitably be placed at the center of the obtained image. When switching to the automatic tracking state in this state, it is possible to determine the target rectangle of the tracking target based on the target rectangle located in the center without distinguishing the target rectangle of the tracking target from among multiple target rectangles. becomes. This allows early recovery at the start of an auto-track state or from a lost state.

In the above-configured invention, the lighting means may include a predetermined driving section, and may stop driving the driving section when the determination section determines that the tracking target is in a lost state. .

In the case of the above configuration, when the tracking target object is lost, it is possible to discriminate the target rectangle of the tracking target from a plurality of target rectangles, so that the irradiation direction of the illumination means can be prevented from being recklessly changed. In the case of lost state, by temporarily switching to the manual tracking state, recovery from the lost state is facilitated, and since the irradiation direction does not fluctuate irregularly, it is possible to prevent disturbing the atmosphere in the venue. can.

Further, in the above configuration, the image acquiring means changes the direction integrally with the lighting means, and the target rectangle of the tracking target determined by the determining section is the center of the image acquired by the image acquiring means. The driving unit may be controlled so as to be in a state of being positioned at

In the case of the above configuration, the control of the direction of the lighting means is based on the image obtained by the image obtaining means, instead of detecting the irradiation state of the lighting. With such control, the target rectangle of the tracking target is always placed in the center of the image acquired by the image acquisition means, and as a result, the tracking target can be placed in the center of the irradiation range of the illumination means.

In the invention relating to the illumination tracking system having each of the configurations described above, the image acquisition means and the illumination means may be installed at any two locations among the ceiling and wall surfaces of the indoor venue.

According to the above configuration, since it is possible to irradiate the object while tracking it from two different points, the image processing of one of the two image acquisition means and the illumination means becomes lost and the illumination cannot be tracked. Even in such a state, if the other image processing is normally performed, at least the other illumination based on the other image processing can continue to track the tracking object.

ADVANTAGE OF THE INVENTION According to this invention which concerns on a tracking-type illuminating device, the tracking target object can be tracked, reducing the influence of an occlusion, and even when illumination changes, tracking can be carried out. Furthermore, in the case of a configuration in which the orientation of the image acquisition means is adjusted so that the target rectangle of the tracking target is positioned at the center of the acquired image, even if a lost state occurs, the acquired image (input image ) can be returned to the tracking state as the tracking target.

According to the lighting tracking system of the present invention, the above-described tracking lighting device can cope with lighting changes while reducing the influence of occlusion. In a banquet or the like, a person or the like to be illuminated can be tracked while being compared with teacher data, thereby improving robustness. In a configuration capable of switching between the manual tracking state and the automatic tracking state, lighting tracking can be executed in the automatic tracking state after preparation in the manual tracking state.

It is an explanatory view showing an embodiment of the present invention concerning an illumination tracking system. It is an explanatory view showing an outline of a tracking type illuminating device. FIG. 4 is an explanatory diagram showing the concept of a target rectangle; 1 is a schematic diagram of a convolutional neural network structure; FIG. It is an explanatory view showing the concept of a likelihood map.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows an outline of an illumination tracking system. As shown in FIG. 1(a), the embodiment of the lighting tracking system has tracking lighting devices 1a and 1b installed at one or two locations (two locations in the figure) in a room of a specific venue A. It irradiates specific objects (for example, people) H1 and H2 with illumination (spotlight) in the room.

Each of the lighting devices 1a and 1b uses a tracking type lighting device, which will be described later. For example, in the case of a wedding reception hall, since the groom H1 and the bride H2 move together in the room, the illumination direction of the illumination (spotlight) is changed so as to track the two men H1 and H2. .

Here, as an embodiment of the tracking type lighting device 1, as shown in FIG. is installed in In addition, the tracking illumination device 1 is provided with a processing device 4 as a processing means built in or externally provided, and can receive input of image data captured by the camera 3 and perform image processing.

The tracking lighting device 1 includes a pivot 21 that can pivot in the horizontal direction and a rotary shaft 22 that allows angle adjustment in the vertical direction. It is installed on the surface via a driving part such as a motor. The angle adjustment may be driven by providing the rotating shaft 22 with a driving portion.

By the way, the controller 5 outputs a control signal (applies a control voltage) to the drive section (motor), and drives the drive section (motor) according to the output value. This controller 5 is for performing necessary driving (changing the direction of the lighting device 1) according to the result of image processing by the processing device 4 described above.

By connecting a monitor (display device) 6 to the processing device 4, images captured by the camera 3 and input to the processing device 4 and the state processed by the processing device 4 can be displayed. By visually confirming the monitor 6, the operator can judge whether the tracking state is normal or not.

As shown in FIG. 2 , the processing device 4 includes a target rectangle detection unit 41 , a likelihood calculation unit 42 , a likelihood map generation unit 43 , and a determination unit 44 . A target rectangle is detected for an input image (specific frame). A target rectangle is a rectangular area that can surround a tracking target (for example, a person) with a predetermined width and a predetermined length in the entire image (rectangle). , and is specified in order to determine whether or not it is a tracking target. The processing device 4 is provided with a storage unit 40, which can temporarily store various calculation results and output data in addition to the input data.

The likelihood calculation unit 42 calculates the likelihood from the results (output values of the convolutional neural network structure) identified by the convolutional neural network for each target rectangle. The degree is the likelihood of the target rectangle (rectangular likelihood). The likelihood map creating section 43 creates a likelihood map from a set of the positions of the target rectangles and the likelihoods of the target rectangles. The likelihood map is a map showing the level of likelihood in each target rectangle (displaying the level of likelihood in the entire image). By creating this likelihood map, it becomes possible to specify a target rectangle to be tracked from among a plurality of target rectangles. Therefore, the determination unit 44 determines the target rectangle with the highest likelihood in the likelihood map as the target rectangle of the tracking target.

Specifically, as shown in FIG. 3, when the tracking target is a person, a rectangle (target rectangle) of width w×height h is placed in a region that can be determined to be a person in a specific frame (FIG. 3(a)). Set (Fig. 3(b)). This target rectangle is identified by a convolutional neural network as to whether it can be tracked (FIG. 3(c)). The structure of the convolutional neural network at this time is as shown in FIG.

More specifically, for human rectangle detection, an image In of the n-th frame is input, and a set of human rectangles (width w, height h, center coordinates (x, y)) surrounding the person is output. F(I) is used to obtain a set Rn of human rectangles in the n-th frame. This set R _n can be expressed as a function F as shown below. Yolo (Joseph Redmon: “YOLO9000: Better, Faster, Stronger,” arXiv:1612.08242, (2016)) trained by ImageNet (http://www.image-net.org) is used as the function F. can do.

In identifying the tracking target, an image I _i ' is cut out from In by each element ri of R _n , and the classifier G identifies I _i '. As for G, the convolutional neural network described above can be used.

As a result of this identification, if I _i ' is the tracking target, the rectangle ri is added to Rn' and the likelihood li is added to Ln, as in the following equation.

The likelihood map is an h×w matrix of the same size as the input image In, and the (i, j) component indicates how many tracking targets exist at the (i, j) coordinates in the input image In. shall be The likelihood map created here may be used as the final likelihood map, but the likelihood map created in the n-th frame is used as a temporary map and averaged with the likelihood maps in the past 4 frames. may be used as the final likelihood map. If the state is illustrated, it can be expressed as shown in FIG.

Here, the illumination (spotlight) 2 is directed so as to track the tracking object (person) as described above, and is driven by a signal from the controller. At this time, since the spotlight 2 and the camera 3 are installed in a single housing, if the image acquired by the camera 3 is used as a reference and driven so as to obtain a desired image, the spotlight will inevitably The irradiation direction of the light 2 is changed. That is, by aligning the center of the range (usually circular) illuminated by the spotlight 2 with the center of the image acquired by the camera 3, the target rectangle of the tracking target is always positioned at the center of the acquired image. If the orientation of the camera 3 is corrected in this way, the tracking target will be present in the center of the range of light emitted by the spotlight 2 .

Although the embodiments of the present invention are as described above, each of the above embodiments is an illustration of the present invention, and the present invention is not limited to these embodiments. Therefore, elements of the above embodiments can be changed or other elements can be added.

For example, in the lighting tracking system, it is possible to switch between manual tracking mode and automatic tracking mode when operating the lighting device. Manual tracking mode is an operation mode in which the operator manually irradiates a spotlight in a predetermined direction (manual tracking state). This is an operation mode in which the target rectangle of the object is determined and the spotlight is tracked (automatic tracking state). By enabling such mode switching, if the tracking target is lost in the automatic tracking state, the operator can continue tracking in the manual tracking state.

In this mode switching, in the manual tracking state, the operator places the tracking target in the center of the spotlight, and at this time, the tracking target is simultaneously photographed in the center of the image acquired by the camera. Then, it is possible to use the manual tracking state to start the automatic tracking process after specifying the tracked object in advance.

In addition, the processing device is provided with a storage unit in advance, and teacher data is stored in the storage unit, and tracking is performed by comparison with the teacher data at the time of the final determination by the determination means, or in the lost state or near-lost state. You may discriminate|determine a target object.

At that time, in the determination unit 44, the likelihood map generated by the likelihood map generation unit 43 (the latest likelihood map obtained by averaging the past four frames) is calculated by calculating It is possible to set the lost determination based on whether the ratio of the sum of the likelihoods of the target rectangle is equal to or less than a predetermined value. In this case, the ratio of the sum of the likelihoods of the target rectangle of the tracking target to the sum of the likelihoods of the entire area in the likelihood map means that the likelihood corresponding to approximately 10% of the sum of the inductions of the entire area is tracked. It is preferable to be the sum of the target rectangles of the object, and 1/10 is set as the lower limit, and if the likelihood sum is less than that, it can be determined as lost.

Furthermore, as estimation of the target rectangle of the tracking target, a likelihood distribution may be estimated using particles indicating the coordinate position of the target rectangle. Assuming that the part with the highest density of the coordinate positions indicated by the particles estimates the part with the highest likelihood, the center of gravity of the particles can be set as the tracking target.

<Experimental example>
In image processing, a processing operation experiment was conducted to determine whether or not a tracked object can be identified from an acquired image. For the experiment, three videos were used: a video with one person (A), a video with two people passing each other (B), and a video with two out of three people intertwined and abrupt lighting changes (C). ). Further, the processing method is to set the target rectangle as described above, identify it by the convolutional neural network, output the likelihood, create a likelihood map, and then perform the determination.

The results of the experiments are shown in the table below, including a comparison with Mean Shift.

From the above results, it can be seen that the tracking accuracy is improved compared to Mean Shift (Table 2). On the other hand, the processing speed was about 20 times slower than Mean Shift (Table 3). However, if high-speed processing and accurate tracking are possible, the spotlight will move when the tracked object moves slightly, as in the case of use in an indoor venue, for example. Since this causes the emitted light to flicker, it is possible to obtain the effect that natural tracking can be achieved by moving the spotlight a little slower (while lagging behind the tracked object).

1, 1a, 1b tracking type illumination device 2 spotlight (illumination means)
3 camera (image acquisition means)
4 processing equipment (processing means)
5 controller 6 monitor 40 storage unit 41 object rectangle detection unit 42 likelihood calculation unit 43 likelihood map creation unit 44 determination unit

Claims (9)

An image acquiring means, a processing means for discriminating an object to be tracked from the image acquired by the image acquiring means, and a means for illuminating the tracking target discriminated by the processing means while changing the irradiation direction. In a tracking lighting device comprising lighting means,
The processing means includes a target rectangle detection unit for detecting a target rectangle for a specific frame input from the image acquisition means, and a rectangle likelihood for each target rectangle based on an output value of a convolutional neural network structure for the target rectangle. a likelihood calculating unit for calculating a likelihood map, a likelihood map creating unit for creating a likelihood map from a set of the positions of the target rectangles and the rectangular likelihoods, and a target rectangle having the highest likelihood among the likelihood maps as a tracking target. A determination unit for determining a target rectangle of an object,
The tracking lighting device, wherein the lighting means appropriately changes the irradiation direction so that the target rectangle of the tracking target determined by the determination unit is positioned at the center of the illumination area. 2. The tracking according to claim 1, further comprising a tracking object movement estimating unit, wherein the movement estimating unit performs tracking estimation with a particle filter for the target rectangle of the tracking object determined by the determining unit. type lighting device. The image acquisition means includes a driving unit that changes an image acquisition direction so that the target rectangle of the tracking target is positioned at the center of the acquired image,
The determination unit determines a predetermined ratio of the sum of likelihoods of the target rectangle of the tracking target to the sum of likelihoods in all regions in the likelihood map created by the likelihood map creation unit or the latest likelihood map. Lost judgment is performed based on being below the value,
3. The tracking illumination system according to claim 2, wherein the movement estimation unit performs movement estimation by regarding a target rectangle located in the center of the image as a target rectangle of the tracking object when the determination unit makes a lost determination. Device. 4. The likelihood map generation unit according to any one of claims 1 to 3, wherein the likelihood map generation unit generates a latest likelihood map updated by an average value of the likelihood maps in the past four frames and the likelihood map in the specific frame. The tracking lighting device according to . A lighting tracking system in an indoor venue using the tracking lighting device according to any one of claims 1 to 4,
the tracked object is a person having a characteristic in appearance,
The processing means comprises a storage unit that stores teacher data relating to external features that the person has in common and movement trajectories of the person,
The determination unit determines that, when the sum of the likelihoods of the target rectangles of the tracking target in the likelihood map created by the likelihood map creation unit or the latest likelihood map is equal to or less than a predetermined value, A lighting tracking system in an indoor venue, characterized in that it identifies a target rectangle of a tracking target by comparing the . A lighting tracking system in an indoor venue using the tracking lighting device according to claim 2 or 3 ,
The image acquisition means is installed integrally with the illumination means, and the center of the acquired image is aligned with the center of the irradiation range illuminated by the illumination means,
The lighting means enables switching between an automatic tracking state and a manual tracking state,
The movement estimating unit determines a target rectangle located in the center of the input image input immediately after switching from the manual tracking state to the automatic tracking state as the target rectangle of the tracking target. A lighting tracking system for indoor venues. 7. The indoor venue according to claim 6, wherein the lighting means includes a predetermined driving section, and stops driving the driving section when the determination section determines that the tracking target is in a lost state. lighting tracking system. The image acquisition means changes the direction integrally with the illumination means, and the target rectangle of the tracking object determined by the determination unit is positioned at the center of the image obtained by the image acquisition means. 8. The lighting tracking system in an indoor venue according to claim 7, wherein the driving unit is controlled so as to. 9. The lighting tracking system for an indoor venue according to any one of claims 5 to 8, wherein said image acquiring means and said lighting means are installed at any two locations among a ceiling and a wall surface of said indoor venue.

Images