JP2022054247A - Information processing device and information processing method - Google Patents

Abstract

To provide a technique for enabling a user to select the desired subject among subjects in a captured image.SOLUTION: An information processing device acquires an image captured by an imaging unit. The information processing device selects one or more subjects as a selected subject on the basis of the movement of the subject in the acquired captured image and information pertaining to the field of view of the imaging unit or information pertaining to a viewing operation on the captured image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for selecting a subject in an image.

In the technique of extracting a specific subject image from the images supplied in time series and tracking the subject, the human face region and the human body region in the moving image are specified. Subject tracking techniques can be used in many areas, such as communication conferencing, man-machine interfaces, security, monitor systems for tracking arbitrary subjects, and image compression.

For digital still cameras and digital video cameras, a technique has been proposed in which a subject image in an image specified by an operation using a touch panel or the like is extracted and tracked to optimize the focus state and exposure state for the subject.

Patent Document 1 discloses an image pickup device that calculates a motion vector of a subject in an image and selects a subject having the highest contrast among subjects having a movement amount of a predetermined value or more as a tracking target.

Patent No. 5589527

Dalal and Triggs. Histograms of Oriented Gradients for Human Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2005 B. K. P. Horn and B. G. Schunk, "Determining Optical Flow, Artificial Intelligence", vol. 17, pp. 185-203, 1981. Philippp Fisher, et al. FlowNet: Learning Optical Flow with Convolutional Network, 2015. Joseph Redmon, et al. You Only Look Access: Unified, Real-Time Object Detection, 2015

In the subject selection method disclosed in Patent Document 1, the subject is selected based on the movement amount and contrast of the subject, but when there are a plurality of subjects and the movement amount of the subject desired to be selected by the photographer is small or the contrast is high. If it is not the maximum, the selection accuracy of the subject is reduced. The present invention provides a technique for enabling a user to select a desired subject among subjects in a captured image.

The uniform state of the present invention relates to the acquisition means for acquiring the captured image captured by the imaging unit, the movement of the subject in the captured image acquired by the acquisition unit, and the visual field of the imaging unit or the captured image. It is characterized by comprising information related to a browsing operation and selection means for selecting one or more subjects as a selection subject based on the information.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a technique for enabling a user to select a desired subject among subjects in a captured image.

(A) is a block diagram showing a hardware configuration example of a computer device, and (b) is a block diagram showing a system configuration example. (A) is a flowchart showing the operation of the information processing apparatus 100, (b) is a flowchart of processing for updating (learning) the operation parameters of the attention calculation unit 125, and (c) is an example of a sequence of captured images. figure. (A) is a diagram showing a specific example of the NN model, and (b) is a diagram showing a configuration in which the functions of the information acquisition unit 122, the detection unit 123, the motion calculation unit 124, and the attention level calculation unit 125 are performed using CNN. (C) is a diagram showing a display example of a captured image. (A) is a diagram showing an example of the appearance of an image pickup apparatus, and (b) is a diagram showing an example of display on the display unit 451. (A) is a block diagram showing an example of a functional configuration of a system, (b) is a flowchart showing the operation of the information processing apparatus 500, and (c) is a diagram showing an example of a captured image. (A) is a block diagram showing a functional configuration example of the system, (b) is a diagram showing a display example on the display screen of the operation unit 601 of the partial image 650, and (c) is a diagram showing a display example of the enlarged image 660.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

[First Embodiment]
First, a configuration example of the system according to the present embodiment will be described with reference to the block diagram of FIG. 1 (b). As shown in FIG. 1 (b), the system according to the present embodiment has an image pickup unit 150 and an information processing device 100, and the image pickup unit 150 and the information processing device 100 are wirelessly and / or wirelessly connected to each other. It is configured to enable data communication with each other via the network of. Here, it is assumed that the image pickup unit 150 is a network camera.

First, the image pickup unit 150 will be described. The image pickup unit 150 captures a moving image and outputs an image of each frame in the moving image as a captured image. The image pickup unit 150 may be a device that periodically or irregularly captures a still image and outputs the still image as an captured image. In any case, the image pickup unit 150 outputs a sequence of captured images. The image pickup unit 150 can change the image pickup direction in the pan direction and / or the tilt direction based on the control from the information processing apparatus 100. In this way, the image pickup unit 150 is an image pickup device capable of changing the field of view. In FIG. 1B, the image pickup unit 150 is a separate device from the information processing device 100, but the image pickup unit 150 and the information processing device 100 are integrated to form one information processing device with an image pickup function. It may be configured.

Next, the information processing apparatus 100 will be described. The image acquisition unit 121 acquires the captured image output from the image pickup unit 150. The information acquisition unit 122 acquires information regarding the visual field of the imaging unit 150 at the time of imaging as visual field control information. The detection unit 123 detects a subject (referred to as a person in this embodiment) from the captured image acquired by the image acquisition unit 121. The motion calculation unit 124 obtains motion in the captured image of each subject detected by the detection unit 123. The attention degree calculation unit 125 calculates the degree of attention to the subject based on the visual field control information acquired by the information acquisition unit 122 and the “movement in the captured image of each subject” calculated by the motion calculation unit 124. .. The learning unit 127 updates (learns) the operation parameters of the attention level calculation unit 125. The selection unit 126 selects one or more subjects from the respective subjects based on the attention level calculated by the attention level calculation unit 125 for each subject detected by the detection unit 123.

Next, an example of the sequence of the captured images acquired by the image acquisition unit 121 will be described with reference to FIG. 2 (c). The captured image 250 is an captured image captured by the imaging unit 150 at time t1, and the captured image 250 includes a walking subject 251 and a subject 252. (X11, y11) indicates the position of the center of gravity (image coordinates) of the subject 251 in the captured image 250, and (x21, y21) indicates the position of the center of gravity (image coordinates) of the subject 252 in the captured image 250.

The captured image 260 is an captured image captured by the imaging unit 150 at time t2 (time after time t1) by changing the imaging direction (rotating the imaging unit 150) after imaging the captured image 250. The captured image 260 includes a walking subject 251 and a subject 252. (X12, y12) is the position of the center of gravity (image coordinates) of the subject 251 in the captured image 260, and (x22, y22) indicates the position of the center of gravity (image coordinates) of the subject 252 in the captured image 260. Next, the operation of the information processing apparatus 100 will be described according to the flowchart of FIG. 2A.

<Step S201>
The image acquisition unit 121 acquires the captured image output from the image pickup unit 150. The image acquisition unit 121 is not limited to acquiring the captured image from the image pickup unit 150. For example, the image acquisition unit 121 may acquire an captured image from a storage device that holds a group of captured images captured by the image pickup unit 150.

<Step S202>
The information acquisition unit 122 acquires "the amount of rotation of the image pickup unit 150 at the time of imaging the captured image acquired in step S201" as "field of view control information of the image pickup unit 150 at the time of imaging the captured image acquired in step S201". The amount of rotation of the image pickup unit 150 can be calculated from the angular velocity of the image pickup unit 150 measured by the gyro sensor of the image pickup unit 150. Assuming that the angular velocity of the imaging unit 150 at time t is w (t), the rotation amount Δθ of the imaging unit 150 between time t1 and time t2 can be calculated by calculating the following equation.

Therefore, the information acquisition unit 122 calculates the above equation using the “angular velocity of the image pickup unit 150 at each time” output from the gyro sensor of the image pickup unit 150, and “captures the captured image acquired in step S201”. The amount of rotation of the image pickup unit 150 at the time "is obtained.

The visual field control information of the image pickup unit 150 is not limited to the rotation amount of the image pickup unit 150. For example, the measurement result by the acceleration sensor of the image pickup unit 150, the measurement result by the distance measurement sensor of the image pickup unit 150, the aperture information of the image pickup unit 150, and the pan and tilt of the pan head on which the image pickup unit 150 is mounted. Is also good. When the image pickup unit 150 is a network camera (a camera that can be remotely controlled from the information processing device 100 via the network), the field of view control information of the image pickup unit 150 is the pan, tilt, and zoom control information of the image pickup unit 150. There may be.

Further, the information acquisition unit 122 acquires "the visual field control information attached to the captured image acquired in step S201" as "the visual field control information of the imaging unit 150 at the time of capturing the captured image acquired in step S201". You may do so. As described above, the method of acquiring the visual field control information by the information acquisition unit 122 is not limited to the specific acquisition method, and the information that can be used as the visual field control information is not limited to the above information.

<Step S203>
The detection unit 123 detects a subject from the captured image acquired in step S201. As a method of detecting a subject from a captured image, for example, there is the method described in Non-Patent Document 1 described above. In the method described in Non-Patent Document 1, gradient direction histogram features (Histograms of Oriented Gradients) are extracted from the image, and the extracted features are identified as human or not by using a model trained by a support vector machine. The method for detecting a subject from a captured image is not limited to the method disclosed in Non-Patent Document 1. For example, the feature amount to be extracted is not limited to the gradient direction histogram feature, but a Haar-like feature, an LBPH feature (Local Binary Pattern Histogram), or the like may be used, or a combination thereof may be used. Further, the model for identifying a person is not limited to the support vector machine, and an AdaBoost classifier, a random classification tree (Randomized Tree), or the like may be used.

Then, the detection unit 123 obtains the position information of the image area (subject area) of the subject and the likelihood with respect to the subject area for each subject detected from the captured image. When the subject area is a rectangular area, the "position information of the subject area" may be the image coordinates of the four corners of the subject area or the image coordinates of the two vertices facing each other. .. Further, the "position information of the subject area" may be the position of the center of gravity (image coordinates) of the subject area regardless of the shape of the subject area. The "likelihood with respect to the subject area" represents the degree of agreement (accuracy as a person) between the feature amount extracted from the subject area and the model for identifying the subject. Then, the detection unit 123 sets a subject corresponding to the subject region for which the likelihood of the predetermined value or more is obtained as a candidate subject.

<Step S204>
The motion calculation unit 124 calculates the motion of each candidate subject in the captured image of the candidate subject. The "movement of the candidate subject in the captured image" is obtained, for example, as the deviation of the position of the center of gravity of the candidate subject between the captured images. Taking FIG. 2C as an example, the movement of the subject 251 between the captured image 260 and the captured image 250 can be obtained by using the following equation.

V (x, y) = ((x12-x11) / (t2-t1), (y12-y11) / (t2-t1))
V (x, y) represents a vector (motion vector) representing "movement of the subject 251 between the captured image 260 and the captured image 250". The method for obtaining the movement of the subject in the captured image is not limited to a specific method, and for example, the velocity vector in each pixel such as the optical flow disclosed in Non-Patent Document 2 may be obtained.

<Step S205>
The attention degree calculation unit 125 calculates the attention degree to the candidate subject based on the visual field control information acquired in step S202 and the movement calculated for each candidate subject in step S204.

For example, the attention degree calculation unit 125 uses a trained model learned in advance by machine learning. The trained model can be configured by, for example, an NN (Neural Network) model. For example, as an NN model, when the movement of the candidate subject and the visual field control information of the imaging unit 150 are input, a learned model trained to output the degree of attention of the candidate subject is created in advance. Then, the attention degree calculation unit 125 inputs the visual field control information acquired in step S202 and the movement calculated for each candidate subject in step S204 into the trained model, and outputs the output of the trained model as “the candidate”. Calculated as "degree of attention to the subject".

Here, a specific example of the NN model used as the trained model will be described with reference to FIG. 3A. The NN model of FIG. 3A has three fully connected layers, an input layer 302, an intermediate layer 303, and an output layer 304, each layer having a plurality of neurons 305 different from each other. The rotation amount of the imaging unit 150 and the movement of the candidate subject are input to each neuron of the input layer 302, and these information input to the input layer 302 are transmitted to the intermediate layer 303 and the output layer 304, and are transmitted to the output layer 304. Outputs the degree of attention (estimated degree of attention 306) for the candidate subject. In FIG. 3A, there is only one fully connected layer, but there may be a plurality of fully bonded layers. Further, the NN model may be composed of only an input layer and an output layer without an intermediate layer.

As the trained model, a general-purpose trained model capable of calculating the degree of attention for various inputs may be used, or the trained model may be used properly according to the inputs. For example, there is a method of properly using the trained model according to the number of candidate subjects in the captured image. The trained model is not limited to the NN model, but may be composed of other machine learning models such as SVR (Support Vector Regression), and attention should be paid to the correlation between the time change of the candidate subject and the position of the imaging unit 150. It can also be calculated as a degree. Alternatively, as a recursive NN model, RNN (Recurrent Neural Network), LSTM (Long short-term memory), or the like may be used.

<Step S206>
The selection unit 126 selects one or more candidate subjects from the respective candidate subjects as the selected subject based on the degree of attention of each candidate subject calculated in step S205. For example, the selection unit 126 selects the candidate subject corresponding to the highest attention level among the attention levels of each candidate subject calculated in step S205 as the selected subject. Further, for example, the selection unit 126 selects a candidate subject corresponding to the attention level equal to or higher than the threshold value among the attention levels of the candidate subjects calculated in step S205 as the selection subject. Further, for example, if the attention level calculated in step S205 does not exceed the threshold value, the selection unit 126 may not select any candidate subject as the selected subject. Further, for example, the selection unit 126 may obtain the average value of the degree of attention for each candidate subject over a plurality of frames, and select the candidate subject corresponding to the average value having the largest value as the selection subject.

After selecting the selected subject, the selection unit 126 outputs information about the selected subject, but various forms can be considered for the information to be output and the output destination of the information, and the present invention is not limited to a specific form.

Next, the process performed by the information processing apparatus 100 for updating (learning) the operation parameters of the attention degree calculation unit 125 will be described with reference to the flowchart of FIG. 2 (b). In FIG. 2B, the same processing step as the processing step shown in FIG. 2A is assigned the same step number, and the description of the processing step will be omitted.

The information processing apparatus 100 performs processing according to the flowchart of FIG. 2A when selecting a selected subject from the captured image, and updates (learns) the operation parameters of the attention degree calculation unit 125 when updating (learning) the operation parameters of the attention degree calculation unit 125. Processing is performed according to the flowchart of FIG. 2 (b). The information processing apparatus 100 may shift the processing to step S1211 of FIG. 2B after the processing of step S206 described above.

<Step S1211>
The learning unit 127 calculates the difference (error) between the attention level of the candidate subject calculated in step S205 and the correct answer attention level set for the candidate subject. The method of calculating the error between the degree of attention and the correct answer attention is not limited to a specific method, but there are various methods such as the square of the difference between the degree of attention and the correct answer attention, the absolute value of the difference between the degree of attention and the correct answer attention, and so on. The method of calculation is applicable. Here, the setting of the correct answer attention level of the candidate subject will be described. For example, the learning unit 127 displays the captured image acquired in step S201 above on the display screen.

FIG. 3C shows an example of displaying the captured image on the display screen. The captured image 350 of FIG. 3C includes a candidate subject 351 and a candidate subject 352. In FIG. 3C, the user operates the mouse to move the cursor 353 to the vicinity of the candidate subject 351 in order to select the candidate subject 351 as the “desired candidate subject”, and drags the candidate subject 351 into the frame 354. The frame 354 is controlled so that the candidate subject 351 fits. Then, as shown in FIG. 3C, when the user operates the mouse to input a decision instruction while the candidate subject 351 is contained in the frame 354, the learning unit 127 sets the correct attention level of the candidate subject 351 to "1". (The initial value of the correct answer attention of all candidate subjects is "0"). By selecting a plurality of candidate subjects as desired candidate subjects, the correct answer attention level of each of the plurality of candidate subjects can be set to "1".

If the subject is selected and the reselection operation is accepted within a predetermined time, the correct answer attention level of the initially selected subject may be set to "0". Further, when a desired candidate subject is selected for the first captured image in the sequence of captured images acquired by the image acquisition unit 121, the candidate subjects detected from the subsequent captured images and the desired candidate are selected. Perform template matching with the subject. Then, the degree of attention to the correct answer of the candidate subject whose degree of agreement with the desired candidate subject is equal to or greater than the threshold value is set to "1". It should be noted that the captured image in which there is no candidate subject whose degree of coincidence with the desired candidate subject is equal to or greater than the threshold value may not be used for learning.

<Step S1212>
The learning unit 127 updates (learns) the operation parameters of the attention level calculation unit 125 based on the error calculated in step S1211. In the present embodiment, the learning unit 127 updates (learns) the operation parameters of the trained model used by the attention level calculation unit 125 based on the error calculated in step S1211.

For example, when the attention degree calculation unit 125 uses a trained model having the configuration shown in FIG. 3A, the learning unit 127 has an estimated attention level 306 output from the output layer 304 for the candidate subject and the candidate subject. Find the error between the correct answer and the attention level 316. Then, the learning unit 127 propagates the obtained error back to the fully connected layers (output layer 304, intermediate layer 303, input layer 302) in order, and updates the weights between neurons.

The parameter update amount may be weighted based on the evaluation value of the rating of the captured image. For example, there is a method of increasing the amount of parameter update as the evaluation value of the rating of the captured image is higher.

<Modification 1>
In the first embodiment, the information acquisition unit 122, the detection unit 123, the motion calculation unit 124, and the attention level calculation unit 125 are each independent functional units, but these functional units are integrated into one functional unit. Is also good. For example, the operations of these functional units may be collectively performed using a CNN (Convolutional Neural Network).

A configuration in which the functions of the information acquisition unit 122, the detection unit 123, the motion calculation unit 124, and the attention level calculation unit 125 are performed by using the CNN will be described with reference to FIG. 3 (b). Using the configuration shown in FIG. 3B, the functions of the information acquisition unit 122, the detection unit 123, the motion calculation unit 124, and the attention level calculation unit 125 are collectively performed.

The NN model having the configuration shown in FIG. 3B has three types of CNNs (CNN321, 322, 323). The CNN 321 is a model that estimates a feature amount 332 indicating the position and size of a candidate subject from the captured image 331 acquired by the image acquisition unit 121. CNN322 is a model for calculating an optical flow from a plurality of images as disclosed in Non-Patent Document 3 above. Further, CNN322 calculates the movement of the candidate subject and the movement of the background from the optical flow, and calculates the visual field control information as the feature amount 333 based on the movement of the background. The coupler 324 calculates the feature amount 334 by connecting the feature amount 332 and the feature amount 333. The CNN 323 is a model for calculating a map of the degree of attention of each region in the captured image 331 (attention map 335) from the feature amount 334. Then, in this modification, the selection unit 126 selects one or more selected subjects from the respective candidate subjects based on the degree of attention of each candidate subject in the attention level map 335. By doing so, it is possible to detect the subject with a learnable model.

<Modification 2>
When a selected subject is selected in the captured image of the frame of interest and the selected subject is tracked after the frame of interest, if the degree of attention of the selected subject is less than the threshold value in the captured image of a frame after the frame of interest. , The selected subject may be selected again from the other candidate subjects based on the degree of attention of the other candidate subjects other than the selected subject. By doing so, when tracking a once selected subject, if the tracking is mistakenly transferred to another subject in the middle, the correct subject can be reselected.

<Modification 3>
In the first embodiment, the selection unit 126 selects one or more selected subjects from the respective candidate subjects based on the degree of attention of each candidate subject, but the selection of the selected subject depends on the user operation. You may go. In this modification, a case where the information processing device 100 is an image pickup device including an image pickup unit 150 will be described as an example.

FIG. 4A shows an example of the appearance of the image pickup apparatus according to the present embodiment. The image pickup apparatus 450 according to the present embodiment includes a display unit 451 that displays various displays, a switch 452 and a dial 453 that are operated by the user to perform various operation inputs.

A display example on the display unit 451 is shown in FIG. 4 (b). In the following, it is assumed that the selection unit 126 performs the display control in the display unit 451. As shown in FIG. 4B, the display unit 451 displays the captured image 400 including the candidate subject 401 and the candidate subject 402. Further, as shown on the right side of FIG. 4B, the image pickup apparatus 450 is a table in which the attention level “0.6” obtained for the candidate subject 401 and the attention level “0.4” obtained for the candidate subject 402 are registered. Holds 420.

The selection unit 126 displays a frame 411 including the candidate subject 401 and a frame 412 including the candidate subject 402, but each of the frame 411 and the frame 412 has a degree of attention of the candidate subject 401 and attention of the candidate subject 402. The display format is according to the degree. In FIG. 4B, the frame 411 of the candidate subject 401 having a higher degree of attention than the candidate subject 402 is displayed as a solid line frame, and the frame 412 of the candidate subject 402 having a lower attention level than the candidate subject 401 is a dotted line. It is displayed as a frame. It should be noted that there are various display forms of the frame according to the degree of attention, and the display form is not limited to a specific display form. For example, the color of the frame, the thickness of the line, and the like may be controlled according to the degree of attention. Further, instead of the frame, a semi-transparent color rectangle may be superimposed on the candidate subject and displayed, and in that case, the display form of the rectangle is controlled according to the degree of attention. As described above, in addition to the frame, any information may be displayed as long as it is "information for notifying the user of the candidate subject".

When the user presses the switch 452, the selection unit 126 selects the candidate subject 401 (selection target) surrounded by the frame 411, which is a solid line frame, as the selection subject. When the user turns the dial 453 to one side, the frame 411 is displayed as a dotted line frame, and the frame of the candidate subject 402 which is "the candidate subject having the next highest attention after the candidate subject currently surrounded by the solid line frame". 412 is displayed as a solid line. That is, by operating the dial 453, the user can switch the selection target in the order according to the degree of attention (the order of high attention may be high or the order of low attention may be low). By selecting a selected subject in this way, when there are a plurality of candidate subjects having a high degree of attention, the user's desired candidate subject can be selected as the selected subject.

<Effect of the first embodiment>
According to the first embodiment, even when there are a plurality of candidate subjects in the captured image and the movement amount of the subject desired by the user is small or the contrast is not the maximum, the movement of the candidate subject and the field of view control of the image pickup unit are controlled. The desired subject can be selected based on the information. Further, according to the first embodiment, the subject can be selected for various movements of the candidate subject and the visual field control information of the imaging unit.

In the image pickup unit 150 exemplified as a network camera, it is conceivable that the position of the subject to be noted changes depending on the field of view. For example, consider the case where the image pickup unit 150 is set as a security camera in a store. In this case, the position of the subject to be noted changes depending on whether the image pickup unit 150 has the entrance / exit as the image pickup direction or the vicinity of the product shelf as the image pickup direction, so that the image pickup direction and the position of the subject to be noted correlate with each other. There is a relationship. In this way, by inputting the visual field control information which is the imaging direction, it is possible to obtain an output for selecting a subject to be noted with higher accuracy.

[Second Embodiment]
Each of the following embodiments including the present embodiment describes the difference from the first embodiment, and is the same as the first embodiment unless otherwise specified below. In the first embodiment, the selected subject is selected based on the movement of the candidate subject and the visual field control information. In such a first embodiment, when a subject near a specific subject having a high degree of attention is selected, it is necessary to reselect the subject every time the subject near the specific subject changes.

In the present embodiment, a specific candidate subject is specified as a point of interest from each candidate subject based on the degree of attention of each candidate subject, and the position of each candidate subject changes relative to the point of interest. The tracking degree is calculated based on this. The tracking degree is the degree to which the candidate subject is close to the point of interest.

In the present embodiment, a case where a sequence of captured images of a soccer game is input to the information processing apparatus 100 will be described as an example. In this case, the information processing apparatus 100 selects a ball in the captured image as a point of interest, and changes in the relative position of each candidate subject (candidate subject other than the point of interest) in the captured image with respect to the ball. The tracking degree is calculated based on the above, and the selected subject is selected based on the tracking degree. This makes it easier to select a subject that is close to a particular point of interest.

Next, an example of the captured image captured by the imaging unit 150 will be described with reference to FIG. 5 (c). The captured image 550 captured by the image pickup unit 150 includes a subject 551 and a subject 552 moving in the direction of the arrow, and a ball 553 moving in the direction of the arrow. (X1, y1) indicates the position of the center of gravity (image coordinates) of the subject 551 in the captured image 550, and (x2, y2) indicates the position of the center of gravity (image coordinates) of the subject 552 in the captured image 550. Further, (xb, yb) indicates the position of the center of gravity (image coordinates) of the ball 553 in the captured image 550.

Next, an example of the functional configuration of the system according to the present embodiment will be described with reference to the block diagram of FIG. 5A. The information processing apparatus 500 has a configuration in which the attention point calculation unit 501 and the tracking degree calculation unit 502 are added to the information processing apparatus 100.

The attention point calculation unit 501 selects a specific candidate subject from the respective candidate subjects as the attention point based on the attention level of each candidate subject obtained by the attention degree calculation unit 125. The tracking degree calculation unit 502 calculates the tracking degree for the candidate subject based on the change in the relative position of the candidate subject (candidate subject other than the attention point) with respect to the attention point.

Next, the operation of the information processing apparatus 500 will be described with reference to the flowchart of FIG. 5 (b). In FIG. 5B, the same processing steps as those in FIG. 2A are assigned the same step numbers, and the description of the processing steps will be omitted. In the flowchart of FIG. 5B, after the processing in step S205, the process proceeds to step S1501.

<Step S1501>
The attention point calculation unit 501 identifies the attention degree having the largest value among the attention degrees of each candidate subject calculated in step S205, and selects the candidate subject of the specified attention degree as the attention point. In the following, in order to give a specific explanation, it is assumed that the attention of the ball 553 is the highest among the attention of each of the three candidate subjects of the subject 551, the subject 552, and the ball 553 in FIG. 5 (c). As a result, it is assumed that the ball 553 is selected as the point of interest.

<Step S1502>
The selection unit 126 determines whether or not to select the point of interest as the selected subject. There are various judgment methods for judging whether or not to select a point of interest as a selected subject, and the judgment method is not limited to a specific judgment method. In the present embodiment, the detection unit 123 performs attribute classification (classification of the subject attribute) of the object in parallel with object detection (subject detection) described in Non-Patent Document 2 above. Then, if the attribute of the attention point is "person", the selection unit 126 determines that the attention point is selected as the selected subject, and the process proceeds to step S206. On the other hand, when the attribute of the attention point is not "person", the selection unit 126 determines that the attention point is not selected as the selected subject, and the process proceeds to step S1503.

The user may decide whether or not to select the point of interest as the selected subject. For example, the user may set in advance setting information indicating whether or not to select the attention point as the selected subject in the information processing apparatus 500. Further, for example, in step S1502, the user may be inquired whether or not to select the attention point as the selected subject. Then, if there is an operation input to "select the attention point as the selected subject" from the user, the process proceeds to step S206, and if there is an operation input to "do not select the attention point as the selected subject" from the user, the process proceeds. Goes to step S1503.

<Step S1503>
The tracking degree calculation unit 502 calculates the tracking degree for each of the subject 551 and the subject 552 based on the change in the position relative to the ball 553. The relative position of the subject 551 with respect to the ball 553 is (x1-xb, y1-yb), and the relative position of the subject 552 with respect to the ball 553 is (x2-xb, y2-yb).

Therefore, the tracking degree calculation unit 502 obtains the difference of (x1-xb, y1-yb) between the captured images as "change in the relative position of the subject 551 with respect to the ball 553". Further, the tracking degree calculation unit 502 obtains the difference of (x2-xb, y2-yb) between the captured images as "change in the relative position of the subject 552 with respect to the ball 553".

Then, the tracking degree calculation unit 502 obtains "a change in the relative position of the subject 551 with respect to the ball 553" within a certain period of time. Then, the tracking degree calculation unit 502 uses the reciprocal of the distance between the average of changes within a certain period (the average relative position of the subject 551 with respect to the ball 553) and the position of the ball 553 as the tracking degree with respect to the subject 551. demand.

Similarly, the tracking degree calculation unit 502 obtains "a change in the relative position of the subject 552 with respect to the ball 553" within a certain period of time. Then, the tracking degree calculation unit 502 uses the reciprocal of the distance between the average of changes within a certain period (the average relative position of the subject 552 with respect to the ball 553) and the position of the ball 553 as the tracking degree with respect to the subject 552. demand.

Note that the tracking degree calculation unit 502 may obtain the tracking degree for the candidate subject by using the trained model in the same manner as the attention degree calculation unit 125. In this case, the tracking degree calculation unit 502 obtains the output of the trained model in which the "change in the relative position of the candidate subject with respect to the point of interest" is input as the tracking degree for the candidate subject. This trained model is an NN model trained to output the tracking degree for the candidate subject when "change in the relative position of the candidate subject with respect to the point of interest" is input.

Then, in the present embodiment, when the process proceeds from step S1501 to step S206, the same process as in the first embodiment is performed in step S206. On the other hand, when the process proceeds from step S1503 to step S206, in step S206, the selection unit 126 selects one or more candidate subjects from the respective candidate subjects based on the tracking degree of each candidate subject other than the point of interest. Select as the selected subject. It is the same as step S206 in the first embodiment except that the tracking degree is used instead of the attention degree.

<Effect of the second embodiment>
According to the second embodiment, when the degree of attention of a specific object is high, an object near the specific object can be selected as a selection subject.

[Third Embodiment]
In the first embodiment, a method of calculating the degree of attention of a candidate subject based on the visual field control information of the image pickup unit 150 at the time of imaging and selecting the selected subject based on the degree of attention has been described. In the first embodiment, since the visual field control information of the image pickup unit 150 that has captured the captured image for selecting the selected subject is acquired, the selected subject can be selected from the captured image. However, if the captured image for selecting the selected subject has been captured in the past and the visual field control information of the imaging unit 150 at the time of capturing the captured image is not obtained, in the first embodiment, the captured image Cannot select the selected subject from. In the present embodiment, as the visual field control information of the image pickup unit 150, information related to the user's browsing operation for the captured images captured in the past is used.

An example of the functional configuration of the system according to the present embodiment will be described with reference to the block diagram of FIG. 6A. As shown in FIG. 6A, the system according to the present embodiment includes an information processing apparatus 600, an operation unit 601 operated by a user to perform various operation inputs to the information processing apparatus 600, and an image taken in the past. It has an external storage device 602 that holds the captured image group.

First, the operation unit 601 will be described. The operation unit 601 has a user interface such as a keyboard, a mouse, and a touch panel, and various instructions can be input to the information processing apparatus 600 by the user operating the operation unit 601. Further, the operation unit 601 has a display screen capable of displaying an captured image or the like.

Next, the external storage device 602 will be described. The external storage device 602 is a memory device such as a hard disk drive device or a USB memory, and the external storage device 602 stores a group of captured images captured in the past.

Next, the information processing apparatus 600 will be described. The information processing device 600 has the same configuration as the information processing device 100 described above, but the information acquisition unit 122 acquires the field control information from the operation unit 601 and the image acquisition unit 121 stores the captured image externally. The point obtained from the apparatus 602 is different from the first embodiment. The user operates the operation unit 601 to perform various browsing operations on the captured image stored in the external storage device 602, and the information acquisition unit 122 uses the information related to this browsing operation as the visual field control information. Obtained from the operation unit 601. The information related to the viewing operation for the captured image includes, for example, information related to operations such as an operation for enlarging the captured image by the user and an operation for moving the viewing range in the captured image (scrolling operation in the up, down, left, and right directions) by the user. include.

In the information processing device 600, the information acquisition unit 122 acquires the field control information from the operation unit 601, the image acquisition unit 121 acquires the captured image from the external storage device 602, and the information processing device 600 is used for viewing the captured image as the field control information. It operates in the same manner as the information processing apparatus 100, except that such information is used.

FIG. 6B shows an example of displaying a partial image 650 (viewing range in the captured image) of the captured image read from the external storage device 602 by the image acquisition unit 121 on the display screen of the operation unit 601. The partial image 650 includes a subject 651,652,653. The user can operate the operation unit 601 to scroll the captured image in the window so that the range to be viewed in the captured image fits in the window. Further, the user operates the operation unit 601 to move the cursor 654 in order to display the enlarged image 660 (FIG. 6 (c)) in which the partial area 655 of the partial image 650 displayed in the window is enlarged in the window. The partial area 655 is specified by performing a drag operation using the tool. As a result, as shown in FIG. 6C, an enlarged image 660 obtained by enlarging the image in the partial region 655 is displayed in the window. The image display control according to such a user operation is performed by the operation unit 601. Then, the information acquisition unit 122 acquires the information related to such a user operation from the operation unit 601 as the visual field control information. In the present embodiment, scroll information indicating which partial area of the captured image is to be displayed in the window as the scroll operation performed by the user by operating the operation unit 601 is used as the visual field control information. It shall be used.

The average attention level (average attention level) in the sequence of captured images may be obtained for each candidate subject, and the selected subject may be selected according to the obtained average attention level. At that time, when the enlarged image 660 obtained by enlarging the image in the partial area 655 of the captured image is displayed in the window, the magnified image 660 is enlarged according to the magnified image (magnification rate with respect to the image in the partial area 655). The weight of the candidate subject in the captured image is controlled. Therefore, the average attention level is the average of the weighted attention levels weighted for each captured image.

<Modification 1>
In the third embodiment, it is assumed that the external storage device 602 stores the image group captured by only one image pickup unit 150, but the image group captured by two or more image pickup units 150 is stored. May be. In that case, the user can view the image captured by any imaging unit 150.

It is assumed that the user operates the operation unit 601 while viewing the image captured by the image pickup unit A and inputs an instruction to view the image captured by the image pickup unit B. At this time, the information acquisition unit 122 obtains the amount of movement MA from the image pickup range of the image pickup unit A to the image pickup range of the image pickup unit B.

The detection unit 123 recognizes each subject in the captured image. The motion calculation unit 124 obtains the amount of movement MB from the position of the subject T in the image captured by the image pickup unit A to the position of the subject T in the image captured by the image pickup unit B.

The attention degree calculation unit 125 inputs the movement amount MA and the movement amount MB into the trained model trained to output the “attention degree of the subject T” corresponding to the movement amount MA and the movement amount MB. The output of the trained model is obtained as the "attention degree of the subject T". The method of obtaining the "attention level of the subject T" from the movement amount MA and the movement amount MB is not limited to a specific method, and for example, as in the first embodiment, the movement amount and the subject in the imaging range. A method of obtaining the degree of attention of the subject T by using the correlation with the movement amount of T may be adopted. By doing so, it is possible to select the subject that the viewer is paying attention to from the browsing information of the plurality of captured images captured by the plurality of imaging units.

<Effect of the third embodiment>
According to the third embodiment, even when the visual field control information that can be acquired from the image pickup unit is limited, such as the recorded data of the surveillance camera, the attention level is calculated based on the image viewing operation by the viewer. The subject can be selected based on the degree of attention.

<Example of application to other fields>
For example, even in the case of selecting a subject of interest based on a browsing operation in a virtual viewpoint image, the above-described embodiment and modification can be applied in the same manner. A virtual viewpoint image is an image generated by performing synchronous imaging from different positions using a plurality of image pickup devices and using the obtained multi-viewpoint images. By such a technique, a virtual viewpoint image is displayed on a three-dimensional space. Images from any viewpoint can be acquired. In addition, in a goggle-type video reproduction device such as VR goggles, even when the attention level of the candidate subject is calculated based on the user's line-of-sight information in addition to the movement information of the goggles, the above embodiment And variants can be applied in the same way. By doing so, it is possible to display annotations related to a person or an object having a high degree of attention and to reproduce a tracking image.

[Fourth Embodiment]
The functional unit of the information processing device 100 shown in FIG. 1 (b), the functional unit of the information processing device 500 shown in FIG. 5 (a), and the functional unit of the information processing device 600 shown in FIG. 6 (a) are hardware. It may be implemented by hardware or software (computer program). In the latter case, the computer device capable of executing the software can be applied to the information processing device 100, the information processing device 500, and the information processing device 600.

An example of such a hardware configuration of a computer device will be described with reference to the block diagram of FIG. 1 (a). The configuration shown in FIG. 1A is an example of the configuration of a computer device applicable to the information processing device 100, the information processing device 500, and the information processing device 600, and can be appropriately changed / modified.

The CPU 101 executes various processes using computer programs and data stored in the ROM 102 and the RAM 103. As a result, the CPU 101 controls the operation of the entire computer device, and also executes or controls various processes described as those performed by the information processing device 100, the information processing device 500, and the information processing device 600.

The ROM 102 stores the setting data of the information processing device 100, the information processing device 500, and the information processing device 600. Further, the ROM 102 stores computer programs and data related to the activation of the information processing device 100, the information processing device 500, and the information processing device 600. Further, the ROM 102 stores computer programs and data related to the basic operations of the information processing device 100, the information processing device 500, and the information processing device 600.

The RAM 103 has an area for storing computer programs and data loaded from the ROM 102 and the external storage device 104. Further, the RAM 103 has an area for storing data received from the outside (imaging unit 150, operation unit 601, external storage device 602, etc.) via the I / F 107. Further, the RAM 103 has a work area used by the CPU 101 to execute various processes. As described above, the RAM 103 can appropriately provide various areas.

The external storage device 104 is a non-volatile memory such as a hard disk drive device, a USB memory, and a flash memory. The external storage device 104 stores computer programs and data for causing the CPU 101 to execute or control various processes described as those performed by the OS (operating system), the information processing device 100, the information processing device 500, and the information processing device 600. Has been done. The computer programs and data stored in the external storage device 104 are appropriately loaded into the RAM 103 according to the control by the CPU 101, and are processed by the CPU 101. The external storage device 104 can also be used as the above-mentioned external storage device 602.

Further, the external storage device 104 can also be realized by a medium (recording medium) and an external storage drive for realizing access to the media. As such a medium, for example, a flexible disk (FD), a CD-ROM, a DVD, an MO, and the like are known. Further, the external storage device 104 may be a computer device such as a server device connected to the computer device via a network.

The display unit 105 has a liquid crystal screen, an organic EL screen, a touch panel screen, and the like, and displays the processing result by the CPU 101 with images, characters, and the like. The display unit 105 may be a projection device such as a projector that projects an image or characters. Further, the display unit 105 may be a display device connected to a computer device via a network. The display unit 105 can be used as a display device (display screen) that has appeared in the above description.

The operation unit 106 is a user interface such as a keyboard, a mouse, and a touch panel screen, and various instructions can be input to the CPU 101 by the user's operation. The operation unit 106 can also be used as the operation unit 601 described above.

The configuration of the display unit 105 and the operation unit 106 is not limited to this, and other configurations may be used. For example, the operation unit 106 may be a pen tab red, or the display unit 105 and the operation unit 106 may be combined to form a tablet.

The I / F 107 is a communication interface for performing data communication with the outside (imaging unit 150, operation unit 601, external storage device 602, etc.), and the computer device is with the outside via the I / F 107. Data communication can be performed.

The CPU 101, ROM 102, RAM 103, external storage device 104, display unit 105, operation unit 106, and I / F 107 are all connected to the system bus 108.

In addition, the numerical values, processing timing, processing order, data (information) acquisition source / output destination, screen configuration, and its operation method used in each of the above embodiments and modifications are for specific explanation. It is given as an example, and is not intended to be limited to such an example.

Further, a part or all of each of the above-described embodiments and modifications may be used in combination as appropriate. Further, a part or all of each of the above-described embodiments and modifications may be selectively used.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiment, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to publicize the scope of the invention.

100: Information processing device 121: Image acquisition unit 122: Information acquisition unit 123: Detection unit 124: Motion calculation unit 125: Attention calculation unit 126: Selection unit 127: Learning unit 150: Imaging unit

Claims (14)

An acquisition means for acquiring an image captured by the image pickup unit, and
Selection to select one or more subjects as the selected subject based on the movement of the subject in the captured image acquired by the acquisition means and the information related to the field of view of the imaging unit or the viewing operation for the captured image. An information processing device characterized by being provided with means. Further provided with a calculation means for obtaining the degree of attention to the subject in the captured image based on the movement and the information.
The information processing according to claim 1, wherein the selection means selects one or more subjects from the subjects in the captured image as the selection subject based on the degree of attention obtained by the calculation means for each subject. Device. The information processing apparatus according to claim 2, wherein the calculation means obtains the output of the trained model in which the movement and the information are input as the degree of attention. The information according to claim 2 or 3, wherein the selection means reselects the selected subject based on the attention of a subject other than the selected subject when the attention of the selected subject becomes less than the threshold value. Processing equipment. One of claims 2 to 4, further comprising display control means for displaying information for notifying the user of the subject based on the degree of attention obtained by the calculation means for each subject. The information processing device described in. The information processing apparatus according to claim 5, wherein the display control means displays information for notifying the user of the subject in a display form according to the degree of attention of the subject. The information processing apparatus according to claim 5, wherein the selection means switches the selection target among the respective subjects in an order according to the degree of attention, and selects the selection target as the selection subject. The selection means selects a specific subject as a point of interest among the respective subjects based on the degree of attention obtained for each subject by the calculation means, and selects the point of interest and a subject other than the point of interest. The information processing apparatus according to claim 2, wherein one or more subjects are selected as selected subjects from subjects other than the point of interest based on the relative change in the position of the information processing apparatus. The information related to the visual field of the image pickup unit includes the rotation amount of the image pickup unit, the measurement result by the acceleration sensor of the image pickup unit, the measurement result by the distance measuring sensor of the image pickup unit, the aperture information of the image pickup unit, and the image pickup unit. The information processing according to any one of claims 1 to 8, wherein the pan and tilt of the mounted cloud stand, and the pan, tilt, and zoom control information of the image pickup unit are used. Device. The information processing apparatus according to claim 2, wherein the attention level obtained for each subject by the calculation means is the average of the attention levels of the subject in a plurality of captured images. The amount of movement from the imaging range of the imaging unit A to the imaging range of the imaging unit B and the amount of movement from the position of the subject in the image captured by the imaging unit A to the position of the subject in the image captured by the imaging unit B are acquired. Acquisition method and
The amount of movement from the imaging range of the imaging unit A to the imaging range of the imaging unit B, and the amount of movement from the position of the subject in the image captured by the imaging unit A to the position of the subject in the image captured by the imaging unit B. An information processing apparatus comprising:, a selection means for selecting one or more subjects as a selection subject based on the above. It is an information processing method performed by an information processing device.
Acquires the captured image captured by the imaging unit and obtains it.
It is characterized in that one or more subjects are selected as the selected subject based on the movement of the subject in the acquired image and the information related to the field of view of the image pickup unit or the browsing operation for the captured image. Information processing method to do. It is an information processing method performed by an information processing device.
The amount of movement from the imaging range of the imaging unit A to the imaging range of the imaging unit B and the amount of movement from the position of the subject in the image captured by the imaging unit A to the position of the subject in the image captured by the imaging unit B are acquired. death,
The amount of movement from the imaging range of the imaging unit A to the imaging range of the imaging unit B, and the amount of movement from the position of the subject in the image captured by the imaging unit A to the position of the subject in the image captured by the imaging unit B. An information processing method characterized in that one or more subjects are selected as selected subjects based on. A computer program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 11.

Images