JP2021083015A - Reproduction control device and reproduction control program - Google Patents

Abstract

To more properly support browsing of a user with control related to reproduction.SOLUTION: A reproduction control device 20 comprises an image data acquisition unit 211, a region detection unit 212, a feature amount extraction unit 214, a scene detection unit 216 and a reproduction control unit 217. The image data acquisition unit 211 acquires a plurality of pieces of image data that are chronically continuous. The region detection unit 212 detects a target region including a prescribed target from each image of the plurality of pieces of image data. The feature amount extraction unit 214 extracts the feature amount from each of the plurality of pieces of image data on the basis of the change in the images between the plurality of pieces of image data and whether or not the changed region is the target region. The scene detection unit 216 detects the image data corresponding to a prescribed scene by inputting the feature amount to a learning model. The reproduction control unit 217 controls reproduction of the plurality of pieces of image data on the basis of information indicating the image data corresponding to the prescribed scene detected by the scene detection unit 216.SELECTED DRAWING: Figure 3

Description

The present invention relates to a reproduction control device and a reproduction control program.

Conventionally, there is known a technique for controlling playback according to an operation of a user who browses a moving image.
For example, Patent Document 1 discloses that the reproduction speed of a moving image is changed stepwise based on the duration of a contact operation with respect to a user's touch panel and the pressing force of the contact operation. As a result, the user can control the playback more intuitively than when operating a general user interface such as a fast-forward button or a seek bar.

Japanese Patent No. 6483305

However, in the method of controlling the reproduction according to the operation content of the user as described above, it is necessary for the user to perform various operations for the reproduction control, which is complicated for the user. Further, for example, in a moving image for the first time to be viewed, it is not easy for the user to specify in which part of the moving image a predetermined scene (for example, a scene for viewing the moving image) is included.

The present invention has been made in view of such a situation. Then, an object of the present invention is to support the user's browsing more appropriately by controlling the reproduction.

In order to solve the above problems, the reproduction control device according to the embodiment of the present invention is
An image data acquisition means for acquiring a plurality of image data that are continuous in time, and
An area detecting means for detecting a target area including a predetermined target from the image of each of the plurality of image data, and an area detecting means.
A feature amount extraction means for extracting a feature amount from each of the plurality of image data based on a change in an image between the plurality of image data and whether or not the changing region is the target area.
A scene detection means for detecting image data corresponding to a predetermined scene by inputting the feature amount into the learning model, and
A reproduction control means for controlling the reproduction of the plurality of image data based on the information indicating the image data corresponding to the predetermined scene detected by the scene detection means.
It is characterized by having.

According to the present invention, it is possible to more appropriately support the user's browsing by controlling the reproduction.

It is a block diagram which shows an example of the whole structure of the reproduction control system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the structure of the wearable camera which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the structure of the reproduction control apparatus which concerns on one Embodiment of this invention. It is a schematic diagram explaining the detection of the target area and the gaze point in the process by the reproduction control apparatus which concerns on one Embodiment of this invention. It is a schematic diagram explaining the moving distance of the gaze point in the process by the reproduction control apparatus which concerns on one Embodiment of this invention. It is a schematic diagram explaining the movement amount of the background in the processing by the reproduction control apparatus which concerns on one Embodiment of this invention. It is a schematic diagram explaining the movement amount of the moving part in the processing by the reproduction control apparatus which concerns on one Embodiment of this invention. It is a schematic diagram which shows an example of the user interface at the time of reproduction in the processing by the reproduction control apparatus which concerns on one Embodiment of this invention. It is a flowchart explaining the flow of the photographing process executed by the wearable camera which concerns on one Embodiment of this invention. It is a flowchart explaining the flow of the learning process executed by the reproduction control apparatus which concerns on one Embodiment of this invention. It is a flowchart explaining the flow of the reproduction control processing executed by the reproduction control apparatus which concerns on one Embodiment of this invention.

Hereinafter, an example of the embodiment of the present invention will be described with reference to the accompanying drawings.

[System configuration]
FIG. 1 is a block diagram showing an overall configuration of the reproduction control system S according to the present embodiment. As shown in FIG. 1, the reproduction control system S includes a wearable camera 10 and a reproduction control device 20. Further, FIG. 1 also shows a user U who wears the wearable camera 10.

The wearable camera 10 and the reproduction control device 20 are connected to each other so as to be able to communicate with each other. Communication between the devices may be performed in accordance with an arbitrary communication method, and the communication method is not particularly limited. Further, the communication connection may be a wired connection or a wireless connection. Further, communication between the devices may be performed directly or via a network including a relay device. In this case, the network is realized by, for example, a LAN (Local Area Network), a network such as the Internet or a mobile phone network, or a network in which these are combined.

The wearable camera 10 is a device having a function of capturing an image (hereinafter, referred to as a “field-of-view image”) of a space corresponding to the field of view of the user U. The wearable camera 10 is realized by, for example, a glasses-type wearable device.

Further, the wearable camera 10 also measures the gazing point, which is a portion of the visual field that the user U is gazing at, at the same time as capturing the visual field image. Further, the wearable camera 10 generates image data including the captured visual field image and the measured information on the gazing point of the user U (for example, coordinate values of two-dimensional coordinates corresponding to the position of the gazing point). Further, the wearable camera 10 repeats the process for generating such image data to generate a moving image composed of a plurality of temporally continuous image data showing a change in the visual field image and the gazing point of the user U. To do. Then, the wearable camera 10 transmits the moving image composed of the plurality of image data to the reproduction control device 20.

The reproduction control device 20 is a device that controls the reproduction of the moving image received from the wearable camera 10. The reproduction control device 20 is realized by, for example, a personal computer or a server device.
As a specific processing content, the reproduction control device 20 acquires a plurality of time-continuous image data from the wearable camera 10. Further, the reproduction control device 20 detects a target area including a predetermined target from the images of each of the plurality of image data. Further, the reproduction control device 20 extracts a feature amount from each of the plurality of image data based on the change of the image between the plurality of image data and whether or not the changing region is the target region. .. Further, the reproduction control device 20 detects image data corresponding to a predetermined scene (for example, a scene to be viewed in a moving image) by inputting this feature amount into the learning model. Then, the reproduction control device 20 controls the reproduction of a plurality of image data based on the information indicating the image data corresponding to the detected predetermined scene.

In this way, the wearable camera 10 can generate a moving image composed of a plurality of image data showing changes in the visual field image and the gazing point of the user U. Further, the reproduction control device 20 detects a predetermined scene based on the feature amount extracted from a plurality of image data in the moving image and the learning model, and also determines a plurality of predetermined scenes based on whether or not the scene is a predetermined scene. It is possible to control the playback of a moving image consisting of the image data of.
Therefore, according to the reproduction control system S according to the present embodiment, it is possible to more appropriately support the user's browsing by controlling the reproduction.

Since such browsing support is provided, the playback control system S solves the problem that the user needs to perform various operations for playback control as described above, which is complicated for the user. can do. In addition, according to the playback control system S, in the video or the like that is viewed for the first time as described above, a predetermined scene (for example, a scene that is the purpose of viewing the video) is included in any part of the video. It is possible to solve the problem that it is not easy to identify whether or not the system is used.

Such a reproduction control system S can be used for various purposes. In the following, as an example of a suitable use of the regeneration control system S, a case where the user U is a surgeon who performs an operation as a predetermined operation will be described as an example. Then, based on the three feature quantities of (1) the movement of the line of sight of the user U, (2) the background change in the visual field image of the user U, and (3) the movement of the hand which is the movement part of the user U in this operation. It is assumed that the reproduction control system S is used for detecting an incision scene, which is a predetermined scene, by performing machine learning.

In the incision scene, the movement of the line of sight of the user U is small because the work is focused on the affected area, the background does not change much because the user U does not move the head, and the movement of the entire hand is small because it is a delicate work by the hand. it is conceivable that. That is, since these three features are highly related to the incision scene, they are considered to be suitable for the use of detecting the incision scene. The operation, which is a predetermined operation, may be performed by the user U alone, but in the following description, it is assumed that the operation is performed as a collaborative work between the user U and the assistant. Therefore, in the above (3), the movement of the hand, which is the movement part of the assistant, is also extracted as a feature amount.

The application of using the reproduction control system S in the moving image of this operation will be described in more detail. One of the methods of transmitting medical technology is to refer to surgical videos. In particular, young surgeons have limited opportunities to experience surgery as a surgeon, so surgical videos from a first-person perspective that correspond to the surgeon's field of view are useful as teaching materials to supplement practical training in surgery. However, surgical videos are often lengthy. For example, in tumor removal surgery in breast surgery, the recording time is often about two hours. It takes a lot of time to identify a predetermined scene (here, an incision scene as an example) that is the purpose of viewing the video from such a long video. This is because the surgical video includes scenes that are not essential in the surgery, such as preparation scenes and tidying up scenes.

Therefore, by using the reproduction control system S as described above, an incision scene that is the purpose of viewing the video is detected from the video of the surgery that tends to take a long time, and the detected incision scene is used as another scene (for example,). , Preparation scenes and tidying up scenes), so that the user who is the viewer can browse in a more easily visible manner. As a result, the user who is a viewer can easily browse the incision scene without performing a complicated operation for playback control.

Further, again, this is only an example of suitable applications, and is not intended to limit the applications in which the reproduction control system S can be used. That is, the reproduction control system S can be used for controlling the reproduction of any moving image other than this. Further, when the moving image for controlling playback includes a work, this work may be a work by a single worker or a collaborative work by a plurality of workers.

In the following, in order to clarify the explanation, a user (corresponding to the user U in FIG. 1) who wears the wearable camera 10 and performs an operation and an assistant thereof are referred to as “workers”. On the other hand, a user who browses the moving image of the operation reproduced by the reproduction control device 20 is referred to as a "viewer".

[Wearable camera configuration]
Next, the configuration of the wearable camera 10 will be described with reference to the block diagram of FIG. As shown in FIG. 2, the wearable camera 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a communication unit 14, a sensor unit 15, and a storage. A unit 16, an input unit 17, an imaging unit 18, and an eye tracking unit 19 are provided. Each of these parts is connected by a signal line and sends and receives signals to and from each other.

The CPU 11 executes various processes (for example, a shooting process described later) according to a program recorded in the ROM 12 or a program loaded from the storage unit 16 into the RAM 13.
Data and the like necessary for the CPU 11 to execute various processes are also appropriately stored in the RAM 13.

The communication unit 14 performs communication control for the CPU 11 to perform communication with another device (for example, the reproduction control device 20).
The sensor unit 15 is composed of an acceleration sensor and a gyro sensor, and measures the movement of an operator wearing the wearable camera 10. Based on the measurement result of the sensor unit 15, the CPU 11 can perform processing such as correction of the deviation between the image pickup unit 18 and the operator after the calibration.

The storage unit 16 is composed of a semiconductor memory such as a DRAM (Dynamic Random Access Memory) and stores various data.
The input unit 17 is composed of various buttons, a touch panel, and the like, and inputs various information according to a user's instruction operation.

The image pickup unit 18 is composed of an image pickup device including a lens, an image pickup element, and the like, and captures a field image.
The eye tracking unit 19 is composed of a light emitting element such as an LED (Light Emitting Diode) and an image pickup device for eye tracking, and measures a gazing point. Specifically, the eye tracking unit 19 generates a reflection point of light on the cornea of the worker by causing the light emitting element to emit light, and captures an image of the worker's eyeball with an image pickup device for eye tracking. .. Then, the eye tracking unit 19 analyzes the captured image of the eyeball to calculate the coordinate values of the two-dimensional coordinates corresponding to the position of the gazing point as information indicating the gazing point of the operator.

The imaging unit 18 and the eye tracking unit 19 are arranged at positions suitable for taking a visual field image and measuring the gazing point while the operator is wearing the wearable camera 10. For example, the lens of the imaging unit 18 is arranged at the bridge portion of the eyeglasses in the wearable camera 10. Further, for example, the light emitting device of the eye tracking unit 19 and the imaging device for eye tracking are arranged around the lens of the eyeglasses in the wearable camera 10.

In the wearable camera 10, the "shooting process" is performed by the cooperation of each of these parts.
Here, the shooting process is a series of processes in which the wearable camera 10 generates a moving image composed of a plurality of image data continuously connected in time based on the visual field image and the information indicating the position of the gazing point.

When this shooting process is executed, as shown in FIG. 2, the visual field image shooting unit 111, the gazing point measurement unit 112, the image data generation unit 113, and the image data transmission unit 114 function in the CPU 11. ..
Further, an image data storage unit 161 is provided in one area of the storage unit 16.
Data necessary for realizing processing is appropriately transmitted and received between these functional blocks at appropriate timings, even if not specifically mentioned below.

The field-of-view image capturing unit 111 uses the imaging unit 18 to capture a field-of-view image at a predetermined cycle (that is, a predetermined frame rate). Then, the field-of-view image photographing unit 111 outputs the field-of-view image obtained by the photographing to the image data generation unit 113.

The gazing point measuring unit 112 uses the eye tracking unit 19 to obtain coordinate values of two-dimensional coordinates corresponding to the gazing point position in a predetermined period (that is, a predetermined frame rate) similar to that taken by the visual field image capturing unit 111. Is calculated. Then, the gazing point measurement unit 112 outputs the coordinate values corresponding to the calculated gazing point positions to the image data generation unit 113.

The image data generation unit 113 associates the field image input from the field image capturing unit 111 with the coordinate values corresponding to the position of the gazing point input from the gazing point measuring unit 112 (that is, frame by frame). By synthesizing), image data including information on the gazing point is generated. Then, the image data generation unit 113 stores the generated image data in the image data storage unit 161.
The field-of-view image capturing unit 111, the gazing point measurement unit 112, and the image data generation unit 113 repeat the process for generating such image data while the work by the operator is continuing, so that the user U can generate the image data. Generates a plurality of temporally continuous image data showing changes in the visual field image and the gazing point.

The image data transmission unit 114 converts a plurality of temporally continuous image data generated by the image data generation unit 113 and stored in the image data storage unit 161 into a moving image data format, and is a playback control device. Send to 20. The process of converting the plurality of image data into the format of moving image data may be performed by the reproduction control device 20 that has received the plurality of image data.

[Configuration of playback control device]
Next, the configuration of the reproduction control device 20 will be described with reference to the block diagram of FIG. As shown in FIG. 3, the reproduction control device 20 includes a CPU 21, a ROM 22, a RAM 23, a communication unit 24, a storage unit 25, an input unit 26, an output unit 27, and a drive 28. .. Each of these parts is connected by a signal line and sends and receives signals to and from each other.

The CPU 21 executes various processes (for example, learning process and reproduction control process described later) according to the program recorded in the ROM 22 or the program loaded from the storage unit 25 into the RAM 23.
Data and the like necessary for the CPU 21 to execute various processes are also appropriately stored in the RAM 23.

The communication unit 24 controls communication for the CPU 21 to communicate with another device (for example, the wearable camera 10).
The storage unit 25 is composed of a semiconductor memory such as a DRAM (Dynamic Random Access Memory) and stores various data.

The input unit 26 is composed of various buttons and a touch panel, or an external input device such as a mouse and a keyboard, and inputs various information according to a user's instruction operation.
The output unit 27 is composed of a display, a speaker, or the like, and outputs an image or sound.
A removable medium (not shown) made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 28. The program read from the removable media by the drive 28 is installed in the storage unit 25 as needed.

In the reproduction control device 20, the "learning process" and the "reproduction control process" are performed by the cooperation of each of these parts.
Here, the learning process shows the input data including the feature amount extracted from the moving image data received from the wearable camera 10 by the playback control device 20 and a predetermined scene (here, an incision scene) acquired from the viewer. It is a series of processes for constructing a learning model (including updating the learning model) by performing machine learning using a set with a label as teacher data.
Further, in the reproduction control process, the reproduction control device 20 detects a predetermined scene based on the feature amount extracted from a plurality of image data in the moving image and the learning model constructed by the learning process, and also determines a predetermined scene. It is a series of processes for controlling the reproduction of a moving image composed of a plurality of image data based on whether or not it is a scene.

When these learning processes and reproduction control processes are executed, as shown in FIG. 3, in the CPU 21, the image data acquisition unit 211, the area detection unit 212, the gazing point detection unit 213, the feature amount extraction unit 214, and so on. Works.
Further, in one area of the storage unit 25, a moving image data storage unit 251 and a learning model storage unit 252 are provided.
Data necessary for realizing processing is appropriately transmitted and received between these functional blocks at appropriate timings, even if not specifically mentioned below.

The image data acquisition unit 211 acquires the moving image data obtained by converting a plurality of image data from the wearable camera 10 by receiving the moving image data. Then, the image data acquisition unit 211 stores the video data obtained by converting the acquired plurality of image data in the video data storage unit 251. The point that the reproduction control device 20 may perform the process of converting the image data into the format of the moving image data is as described above in the description of the image data transmission unit 114.

The area detection unit 212 performs image recognition using an existing method such as edge detection for each of the visual field images (that is, each frame) in the moving image data stored in the moving image data storage unit 251. Detects the target area, which is the area including the moving part of the worker (here, the worker's hand).

The gazing point detection unit 213 includes the image data generation unit 113 in the image data at the time of image data generation from each of the field images (that is, each frame) in the moving image data stored in the moving image data storage unit 251. , The information of the gazing point of the operator (here, the coordinate value indicating the position of the gazing point) is detected.

The detection of the target area and the detection of the gazing point information will be described with reference to FIG. FIG. 4 is a schematic diagram illustrating the detection of the target area and the gazing point.
As shown in FIG. 4, the field of view image 31, which is an example of the field of view image, includes images of captured objects such as the moving part 32, the moving part 33, the scalpel 34, and the marking 35. Further, in the visual field image 31, the boundary 36 of the target area and the gazing point 37 are also shown.

The field-of-view image 31 is a field-of-view image of a scene in which the user U, who is a worker, is making an incision with the assistance of an assistant who is a worker.
The moving part 32 is the hand of the moving part of the user U (surgeon) who is an operator. On the other hand, the moving part 33 is the hand of the moving part of the assistant who is the worker.

The scalpel 34 is a scalpel used by a user U (surgeon) who is an operator to make an incision in a patient. Marking 35 is a line drawn on the patient by a skin marker to clarify the surgical site.

The boundary 36 of the target area is the boundary between the target area detected by the area detection unit 212 and the non-target area which is another area. In this example, since the operating portion 32 and the operating portion 33 are included, the inside of the boundary 36 of the target region is detected as the target region, and the outside is detected as the non-target region. In this example, the target region is detected as one circular region, but it is detected as a plurality of regions corresponding to each operating part according to the environment in which the present embodiment is implemented. Alternatively, it may be detected as a region having a shape other than a circular shape.

The gazing point 37 is a point corresponding to a coordinate value indicating the position of the gazing point detected by the region detection unit 212. This corresponds to the gazing point that the worker U (surgeon) was actually gazing at when the visual field image was taken.

The area detection unit 212 and the gazing point detection unit 213 output the target area detected in this way and the gazing point information to the feature amount extraction unit 214.

The feature amount extraction unit 214 is a feature amount of each moving image data (that is, each frame) in the moving image data based on the detection results of the area detecting unit 212 and the gazing point detecting unit 213, changes between the moving image data, and the like. Is extracted.

As the first feature amount, extraction of the feature amount based on the movement of the line of sight (that is, the movement of the gazing point) of the user U (surgeon) who is the operator will be described with reference to FIG. FIG. 5 is a schematic diagram illustrating the moving distance of the gazing point. First, it is assumed that the gazing point is detected at the position indicated as the gazing point 42-n in the field image 41-n which is the field image of the nth frame (n is an integer value of 1 or more). Next, it is assumed that the gazing point moves and the gazing point is detected at the position indicated as the gazing point 42-m in the visual field image 41-m which is the visual field image of the mth frame (m = n + 1). In this case, the distance from the gazing point 42-n to the gazing point 42-m is the moving distance of the gazing point. In this case, the feature amount extraction unit 214 extracts the first feature amount by calculating it by the following formula shown as, for example, <calculation formula of the feature amount based on the movement of the gazing point>.

<Calculation formula for features based on movement of gazing point>
Euclidean distance / unit time However, the Euclidean distance is the positive square root of the sum of squares of the differences between the components of the coordinate values of the gazing point 42-n and the gazing point 42-m, and the unit time is the frame at the time of shooting the field image. The spacing between adjacent frames corresponding to the rate.

As the second feature amount, extraction of the feature amount based on the background change in the visual field image of the user U (surgeon) who is the operator will be described with reference to FIG. FIG. 6 is a schematic diagram illustrating the amount of movement of the background. First, in the non-target region 43-n, which is the non-target region (that is, the background) in the field image of the nth frame (n is an integer value of 1 or more), the operating table is photographed at the position indicated as the object 44-n. Suppose. Next, since the orientation of the head of the user U (surgeon) who is the operator has changed, in the non-target area 43-m, which is the non-target area of the visual field image content of the mth frame (m = n + 1), It is assumed that the operating table is photographed at the position shown as the object 44-m.

In this case, in order to calculate the second feature amount, the feature amount extraction unit 214 first calculates a movement vector (corresponding to an arrow in the figure) indicating the movement of an object (here, an operating table) between frames. To do. The calculation of this movement vector can be performed based on, for example, the Lucas-Kanade method of optical flow. Further, for the detection of the feature points to be tracked in this case, an existing method such as corner detection can be used. Here, in the present embodiment, the movement vectors of all the calculated feature points are not used as they are as feature quantities, but are used as feature quantities based on whether or not the background moves significantly. Therefore, the feature amount extraction unit 214 extracts the second feature amount by calculating the average value of all the movement vectors calculated between the frames. Here, the average value of all movement vectors between frames in the non-target area is used as the second feature amount, but the average value of all movement vectors between frames in both the target area and the non-target area is used as the second feature. It may be set as the feature amount of.

As a third feature amount, extraction of the feature amount based on the movement of the moving part (here, the hand) of the user U (surgeon) who is the operator will be described with reference to FIG. 7. FIG. 7 is a schematic diagram illustrating the amount of movement of the moving portion. First, it is assumed that in the target area 45-n, which is the target area in the field image of the nth frame (n is an integer value of 1 or more), the hand, which is the moving part, is photographed at the position indicated as the moving part-46n. Next, since the hand of the user U (surgeon) who is the operator has moved, in the target area 45-m, which is the target area in the visual field image of the mth frame (m = n + 1), the moving part is -46 m. It is assumed that the hand, which is the moving part, is photographed at the indicated position.

In this case, in order to calculate the third feature amount, the feature amount extraction unit 214 first calculates a movement vector (corresponding to an arrow in the figure) indicating the movement of the motion part (here, the hand) between frames. To do. The calculation of this movement vector can be performed based on the Lucas-Kanade method of optical flow, for example, in the same manner as in the second feature quantity. However, depending on the type of moving part, it may not be possible to sufficiently detect the feature points. In such a case, the movement vector may be calculated based on the Gunnar-Farneback method of optical flow for all the pixels corresponding to the hand. In any case, the feature amount extraction unit 214 calculates the average value of all the movement vectors calculated for the movement part between the frames with the third feature amount in the same way as the second feature amount. Extract by doing.

Then, the feature amount extraction unit 214 outputs each of these three feature amounts extracted by calculation. The output destination is the learning unit 215 in the case of the learning process, and the scene detection unit 216 in the case of the reproduction control process.

The learning unit 215 uses machine learning as teacher data for a set of input data including three feature quantities extracted by the feature quantity extraction unit 214 and a label indicating a predetermined scene (here, an incision scene) acquired from a viewer. Build a learning model (including updating the learning model) by performing.
Here, the three feature amounts of each image data in the moving image data to be learned are acquired by being input from the feature amount extraction unit 214 as described above.

The label is generated by referring to the moving image to be learned by the viewer in advance and performing an operation for labeling the image data corresponding to a predetermined scene (here, an incision scene). For example, in the case of an incision scene, an operation of labeling the corresponding image data from the moment when the scalpel is cut to the moment when the scalpel is separated from the affected part is performed. In response to this operation, information indicating the correct answer (for example, the value "1") is given to each of the corresponding image data from the moment when the scalpel is cut to the moment when the scalpel is separated from the affected part, and the other image data is incorrect. (For example, the value "0") is given. By this labeling process, the learning unit 215 can acquire a label for each of the image data. This labeling process may be performed by the reproduction control device 20, or may be performed by another device, and the result may be acquired by the reproduction control device 20.

The learning unit 215 generates teacher data by combining the three feature quantities acquired in this way with the corresponding labels. Then, the learning unit 215 uses the teacher data to perform, for example, supervised machine learning. In this case, the learning unit 215 performs machine learning by, for example, a neural network configured by combining perceptrons. Specifically, the features included in the teacher data are given to the input layer of the neural network as input data, and the weighting for each perceptron is changed so that the output of the output layer of the neural network is the same as the label. Repeat learning while doing. For example, after outputting by a method called forward propagation, a weighted value is used to reduce the output error of each perceptron by a technique called backpropagation (also called backpropagation). Repeat adjusting.
In this way, the learning unit 215 inductively acquires a learning model for learning the characteristics of the teacher data and estimating the result from the input.

The machine learning method is not necessarily limited, and for example, a general neural network containing only fully connected layers may be used, or a recurrent neural network such as RNN (Recurrent Neural Network) may be used.

Then, when the predetermined condition for ending the machine learning is satisfied, the learning unit 215 stores the constructed learning model in the learning model storage unit 252. Predetermined conditions for terminating machine learning can be set arbitrarily, but for example, the error between the output and the label is less than or equal to the predetermined reference, and the number of repetitions of weighting adjustment reaches the predetermined number. , The predetermined condition can be that a predetermined time has passed since the start of machine learning. It should be noted that constructing a learning model includes not only creating a new learning model but also updating an existing learning model with new teacher data.

The scene detection unit 216 is based on three feature quantities extracted by the feature quantity extraction unit 214 and a learning model constructed by the learning unit 215 and stored in the learning model storage unit 252, and a predetermined scene (here, here). Incision scene) is detected. Here, the three feature amounts of each image data in the moving image data to be reproduced and controlled are acquired by being input from the feature amount extraction unit 214 as described above.

The scene detection unit 216 gives the three feature quantities acquired in this way as input data to the input layer of the learning model, and determines a predetermined scene (here, an incision scene) based on the output of the output layer of the neural network. ) Is detected. For example, if the output of the output layer is information indicating a correct answer (for example, a value "1" or "a value close to 1 above a predetermined threshold value"), the scene detection unit 216 puts the image data in a predetermined scene. Detect as corresponding image data.
On the other hand, if the output of the output layer is information indicating an incorrect answer (for example, a value "0" or "a value close to 0 below a predetermined threshold value"), the scene detection unit 216 can obtain the predetermined image data. It is not detected as image data corresponding to the scene. That is, it is detected as image data corresponding to other scenes.

Then, the scene detection unit 216 performs this detection process on all the image data in the moving image data, and also provides information indicating which image data is the image data corresponding to the detected predetermined scene. Add to video data. Further, the scene detection unit 216 outputs the moving image data to which the information is added in this way to the reproduction control unit 217 and stores the moving image data storage unit 251.

In the reproduction of the moving image data to which the scene detection unit 216 has added information, the reproduction control unit 217 determines whether or not the image data to be reproduced is the image data corresponding to a predetermined scene based on the information added by the scene detection unit 216. Is determined, and control related to reproduction is performed based on the determination result. Specifically, when the reproduction control unit 217 continuously reproduces a plurality of image data included in the moving image data, the reproduction mode of the image data corresponding to a predetermined scene (hereinafter, "first aspect"). The mode of reproduction of other image data (that is, image data corresponding to other scenes) (hereinafter referred to as "second mode") is different from that of the other mode.

As a premise, since the predetermined scene is, for example, a scene intended by the viewer for viewing, it is desirable to allow the user to view the scene in a manner that is easier to see than other scenes.
Therefore, the reproduction control unit 217 makes the reproduction speed in the first aspect slower than the reproduction speed in the second aspect, for example. For example, the reproduction speed in the first aspect may be a constant speed along the frame rate at the time of shooting, or a reproduction speed slower than that (so-called slow reproduction). On the other hand, the reproduction speed in the second aspect is set to a reproduction speed faster than the constant speed along the frame rate at the time of shooting (so-called fast forward). This makes it possible for the viewer to view a predetermined scene more carefully than other scenes.

In addition, the reproduction control unit 217 expands and reproduces a part of the image data corresponding to a predetermined scene, for example, when reproducing in the first aspect. On the other hand, when the reproduction is performed in the second aspect, no particular processing such as enlargement is performed. As a result, the user can view a predetermined scene in more detail than other scenes. In this case, the area to be expanded may be, for example, a target area detected by the area detection unit 212, an area around the gazing point detected by the gazing point detection unit 213, or an area around the moving part. It can be the area around the tool (here, the scalpel) used by the worker.

The reproduction control unit 217 may perform both the different reproduction speeds and the enlargement in this way. In addition, for example, as the first aspect, a text indicating that the scene is a predetermined scene may be displayed, or a sound indicating that the scene is a predetermined scene may be output. Further, for example, as the first aspect, a text such as an explanation corresponding to a predetermined scene (for example, a text explaining the method of incision in the incision scene) may be displayed.

FIG. 8 is a schematic view showing an example of a user interface at the time of reproduction, which is accompanied by reproduction control by such a reproduction control unit 217. As shown in FIG. 8, the reproduction screen 51 includes a reproduction area 52, a seek bar 53, a slider 54, a predetermined scene portion 55, and an operation icon group 56.

In the playback area 52, a playback image of a moving image to be controlled for playback is displayed. The seek bar 53 is used to adjust the playback position of the moving image according to the operation of the viewer. The slider 54 indicates the current playback location. The predetermined scene portion 55 indicates a portion corresponding to the detected predetermined scene in the seek bar 53. In the figure, a predetermined scene portion 55 is represented by hatching. The operation icon group 56 is an icon corresponding to a so-called stop button, a so-called fast-forward button, and a so-called rewind button.

The viewer can browse the reproduced image of the moving image reproduced in different modes depending on whether or not it is a predetermined scene by referring to the reproduction area 52 only by performing the reproduction start instruction operation. Further, since the predetermined scene portion 55 is displayed, the viewer can easily reach the predetermined scene when operating the slider 54 or the operation icon group 56. Therefore, the viewer does not need to perform complicated operations to reach a predetermined scene as in the conventional case. That is, according to the present embodiment, it is possible to more appropriately support the browsing of the user who is the viewer by controlling the reproduction.

[Shooting process]
Next, with reference to FIG. 9, the flow of the photographing process executed by the wearable camera 10 will be described. FIG. 9 is a flowchart illustrating a flow of shooting processing executed by the wearable camera 10. The shooting process is executed in response to a shooting start instruction operation from a user such as a worker who starts the work.

In step S11, the field of view image capturing unit 111 uses the imaging unit 18 to capture a field of view image at a predetermined cycle (that is, a predetermined frame rate).
In step S12, the gazing point measuring unit 112 uses the eye tracking unit 19 to perform two dimensions corresponding to the gazing point position at a predetermined period (that is, a predetermined frame rate) similar to that taken by the visual field image capturing unit 111. Calculate the coordinate value of the coordinates.

In step S13, the image data generation unit 113 associates the field image input from the field image capturing unit 111 with the coordinate values corresponding to the position of the gazing point input from the gazing point measuring unit 112 on a frame-by-frame basis. By doing (that is, synthesizing), image data including information on the gazing point is generated.

In step S14, the image data generation unit 113 determines whether or not there has been a shooting end instruction operation from a user such as an operator who has completed the work. If there is a shooting end instruction operation, it is determined as Yes in step S14, and the process proceeds to step S15. On the other hand, if there is no shooting end instruction operation, it is determined as No in step S14, and the process is repeated again from step S11.

In step S15, the image data transmission unit 114 converts a plurality of time-continuous image data generated by the image data generation unit 113 into a moving image data format and transmits the image data to the reproduction control device 20. This ends this process.

[Learning process]
Next, the flow of the learning process executed by the reproduction control device 20 will be described with reference to FIG. FIG. 10 is a flowchart illustrating a flow of learning processing executed by the reproduction control device 20. The learning process is executed in accordance with a learning start instruction operation from a user such as a viewer.

In step S21, the image data acquisition unit 211 acquires the moving image data obtained by converting a plurality of image data from the wearable camera 10 by receiving the moving image data.
In step S22, by performing image recognition for each of the visual field images (that is, each frame) in the moving image data, it is an area including the moving part of the worker (here, the worker's hand). Detect the target area.

In step S23, the gazing point detection unit 213 includes the gazing point of the operator from each of the field images (that is, each frame) in the moving image data, which the image data generating unit 113 includes in the image data at the time of image data generation. Information (here, coordinate values indicating the position of the gazing point) is detected.
In step S24, the feature amount extraction unit 214 sets each moving image data (that is, each frame) in the moving image data based on the detection results of the area detecting unit 212 and the gazing point detecting unit 213, changes between the moving image data, and the like. Each feature is extracted.

In step S25, the learning unit 215 acquires a label indicating a predetermined scene (here, an incision scene) generated based on the operation of the viewer.
In step S26, the learning unit 215 generates teacher data by combining the feature amount and the corresponding label, and performs machine learning using the teacher data.

In step S27, the learning unit 215 determines whether or not a predetermined condition for terminating machine learning is satisfied. The specific contents of the predetermined condition for terminating the machine learning are as described above in the explanation of the learning unit 215. When the predetermined condition for ending the machine learning is satisfied, it is determined as Yes in step S27, and the process proceeds to step S28. On the other hand, if the predetermined condition for ending the machine learning is not satisfied, it is determined as No in step S27, and the process repeats step S26 again.

In step S28, the learning unit 215 builds a learning model (including updating the learning model) based on the result of machine learning. This ends this process.

[Playback control processing]
Next, with reference to FIG. 11, the flow of the reproduction control process executed by the reproduction control device 20 will be described. FIG. 11 is a flowchart illustrating the flow of the reproduction control process executed by the reproduction control device 20. The playback control process is executed in response to a playback start instruction operation from a user such as a viewer.

The processing contents from step S31 to step S34 and the processing contents from step S21 to step S24 are the same except that the moving image data to be processed replaces the moving image data to be learned from the moving image data to be controlled to play back the moving image. Therefore, the duplicate description will be omitted.

In step S35, the scene detection unit 216 detects a predetermined scene (here, an incision scene) based on the feature amount extracted by the feature amount extraction unit 214 and the learning model constructed by the learning unit 215. Then, this detection process is performed on all the image data in the moving image data.
In step S36, the scene detection unit 216 adds information indicating which image data corresponds to the detected predetermined scene to the moving image data.

In step S37, the reproduction control unit 217 reproduces the moving image data to which the scene detection unit 216 adds information indicating which image data corresponds to the predetermined scene. Although step S36 and step S37 may be executed continuously, step S37 may be executed after the end of step S36 in accordance with a playback start instruction operation from a user such as a viewer.

In step S38, the reproduction control unit 217 determines whether or not the image data in the moving image data to be reproduced is the image data corresponding to a predetermined scene. If the image data corresponds to a predetermined scene, it is determined to be Yes in step S38, and the process proceeds to step S39. On the other hand, if the image data does not correspond to a predetermined scene (that is, the image data corresponds to another scene), No is determined in step S38, and the process proceeds to step S40.

In step S39, the reproduction control unit 217 reproduces the image data corresponding to the predetermined scene in the first aspect.
In step S40, the reproduction control unit 217 reproduces the image data corresponding to another scene in the second aspect.

In step S41, 218 determines whether or not the moving image has ended by playing the moving image to the end. When the moving image is finished, it is determined as Yes in step S41, and this process is finished. On the other hand, if the moving image is not finished, it is determined as No in step S41, and the process is repeated again from step S38.

According to the shooting process, the learning process, and the reproduction control process described above, it is possible to more appropriately support the user's browsing by controlling the reproduction.
For example, according to these processes, an incision scene to be viewed in a video is detected from a video of an operation that tends to take a long time, and the detected incision scene is used as another scene (for example, a preparation scene or tidying up). Make it possible to browse in a manner that is easier for the user who is the viewer than the screen). As a result, the user who is a viewer can easily browse the incision scene without performing a complicated operation for playback control. In addition, rather than simply repeating machine learning based on general-purpose clues that can be identified by image recognition, such as a bicycle or a person in image data, a gaze point that is appropriate for detecting a predetermined scene, etc. Based on the feature quantity, a predetermined scene can be detected by machine learning in a shorter period of time.

[Modification example]
Although the embodiment of the present invention has been described above, this embodiment is merely an example and does not limit the technical scope of the present invention. The present invention can take various other embodiments without departing from the gist of the present invention, and can be modified in various ways such as omission and substitution. In this case, these embodiments and modifications thereof are included in the scope and gist of the invention described in the present specification and the like, and are included in the scope of the invention described in the claims and the equivalent scope thereof.
As an example, the embodiment of the present invention described above may be modified as in the following modification.

<First modification>
In the above-described embodiment, a learning model is constructed and a predetermined scene is detected by using three feature quantities that are considered to appropriately represent the characteristics of a predetermined scene (here, an incision scene) in surgery. It was. Not limited to this, other feature quantities may be added and used depending on what kind of scene the predetermined scene to be detected is, or other feature quantities may be used instead. Good.

For example, in the above-described embodiment, the worker's hand is used as the moving part, but the feature amount may be extracted and used by using other parts of the worker such as fingers and feet as the moving part. In addition, a tool (for example, a scalpel) used by an operator may be used as a moving part to extract a feature amount and use the feature amount.
In addition, for example, in order to consider the surrounding environment of the place where the work is performed, the shape and color transition of the affected area, etc., the feature amount is extracted from the change of the color information and the brightness information indicated by each pixel. May be used.

In addition, for example, in order to detect the scene of collaborative work with higher accuracy, the number of moving parts (for example, the number of hands) may be extracted as a feature amount and used. In collaborative work, it is highly likely that the number of detected hands will be three or more. Therefore, by setting the number of moving parts such as hands as a feature amount, it is possible to detect collaborative work with higher accuracy. In addition, a wearable camera 10 is attached to each worker who performs collaborative work, and a feature amount is extracted from each image data of the visual field image of each worker taken by each wearable camera 10 and used. May be good. That is, the feature amount may be extracted from a plurality of field images and used. For example, in collaborative work, it is highly likely that the gaze points of each worker will be in the vicinity. Therefore, by extracting a feature amount from a plurality of visual field images and using the feature amount, it is possible to detect collaborative work with higher accuracy. Further, in this case, machine learning (or setting) is performed on which of the visual field images of each worker should be reproduced according to the detected scene, and which of the visual field images of each worker should be reproduced is machined. It may be switched based on the learning result (or the setting content).

In addition, for example, when there are a plurality of types of scenes to be detected as predetermined scenes (for example, an incision scene and a suture scene), a plurality of types of labeling may be performed according to each scene. .. In this case, if it is known that a plurality of types of scenes are performed in a predetermined order, the predetermined order may be used as one of the feature quantities. For example, since it can be seen from the surgical plan that the suturing scene is performed after the incision scene is performed, the type of work that is likely to be performed in each time zone is set as one of the features based on this order. You may use it. Alternatively, when the likelihood value for each scene is output in the output of the learning model, weighting is performed so that the likelihood is high for the types of work that are likely to be performed in each time zone. You may do so. That is, information such as an operation plan in which the scenes indicate a predetermined order may be used as a feature amount or for weighting the output likelihood.

<Second modification>
The user may be able to construct a learning model and detect a predetermined scene by weighting each extracted feature amount with an arbitrary magnification. For example, a user interface for adjusting the degree of weighting, such as an extracted feature amount and a slider corresponding to the extracted feature amount, is prepared. Then, according to the operation of the user using this user interface, which feature amount is weighted and how much is set. Then, each feature amount is weighted according to the setting to construct a learning model and detect a predetermined scene. In addition to the above-mentioned three feature amounts, the feature amounts that can be weighted include, for example, the presence / absence of the detected moving part, the size of the detected moving part, and the distance of each detected feature amount from the screen center. It may be the distance between the detected motion site and the gazing point, and the like.

<Third modification example>
In the above-described embodiment, it is assumed that the wearable camera 10 performs a shooting process to generate moving image data. Not limited to this, the shooting process may be performed by another device to generate moving image data. For example, the imaging process may be performed by a medical device such as an endoscope to generate moving image data. That is, the moving image data targeted for the reproduction control in the present embodiment may be the moving image data generated by shooting with a device other than the wearable camera 10. Alternatively, for example, the wearable camera 10 (or another device that performs shooting processing) and the playback control device 20 may be integrated.

As described above, the reproduction control device 20 according to the present embodiment includes an image data acquisition unit 211, an area detection unit 212, a feature amount extraction unit 214, a scene detection unit 216, and a reproduction control unit 217. ..
The image data acquisition unit 211 acquires a plurality of image data that are continuous in time.
The area detection unit 212 detects a target area including a predetermined target from the images of each of the plurality of image data.
The feature amount extraction unit 214 extracts the feature amount from each of the plurality of image data based on the change of the image between the plurality of image data and whether or not the changing area is the target area.
The scene detection unit 216 detects the image data corresponding to a predetermined scene by inputting the feature amount into the learning model.
The reproduction control unit 217 controls the reproduction of a plurality of image data based on the information indicating the image data corresponding to the predetermined scene detected by the scene detection unit 216.
As described above, the reproduction control device 20 detects a predetermined scene based on the feature amount extracted from the plurality of image data in the moving image and the learning model, and based on whether or not the scene is a predetermined scene. , It is possible to control the playback of a moving image composed of a plurality of image data.
Therefore, according to the reproduction control device 20, it is possible to more appropriately support the user's browsing by controlling the reproduction.

When the reproduction control unit 217 continuously reproduces a plurality of image data, the reproduction mode of the image data corresponding to the predetermined scene detected by the scene detection unit 216 and the reproduction mode of the other image data. To make it different.
As a result, the user can view a predetermined scene in a manner that is easier to see than other scenes.

When the reproduction control unit 217 continuously reproduces a plurality of image data, the reproduction speed of the image data corresponding to the predetermined scene detected by the scene detection unit 216 is slower than the reproduction speed of the other image data. To do.
As a result, the user can browse a predetermined scene more carefully than other scenes.

When a plurality of image data are continuously reproduced, the reproduction control unit 217 expands and reproduces a part of the image data corresponding to a predetermined scene detected by the scene detection unit 216.
As a result, the user can view a predetermined scene in more detail than other scenes.

A predetermined scene is a scene for viewing a plurality of image data to be continuously reproduced, and is a scene in which a plurality of users collaborate with each other.
A predetermined target is a part of each of a plurality of users who perform collaborative work.
The plurality of image data are image data obtained by photographing a space corresponding to the field of view of any user who performs collaborative work.
As a result, the user who browses the image can browse the image corresponding to the field of view of the user who performs the worker when the collaborative work which is the purpose of browsing is being performed.

The reproduction control device 20 further includes a gazing point detection unit 213.
The gazing point detection unit 213 detects the gazing point of the user who visually recognizes the imaging target when capturing a plurality of image data.
The feature amount extraction unit 214 further extracts the feature amount from each of the plurality of image data based on the change in the gazing point of the user who visually recognizes the shooting target between the plurality of image data.
As a result, a predetermined scene can be detected with high accuracy in consideration of an index of a change in the user's gaze point.

The reproduction control device 20 further includes a learning unit 215.
The learning unit 215 constructs a learning model by performing machine learning using a set of input data including a feature amount and a label indicating image data corresponding to a predetermined scene as teacher data, thereby constructing a plurality of learning models in a moving image. A learning model for detecting a predetermined scene can be constructed based on the feature amount extracted from the image data of.

A predetermined scene is a plurality of scenes performed in a predetermined order.
The teacher data also includes information indicating a predetermined order.
As a result, it is possible to construct a learning model capable of performing learning based on information indicating a predetermined order (for example, a surgical plan indicating the order of surgical operations) and detecting a predetermined scene with higher accuracy.

[Realization of functions by hardware and software]
The function of executing a series of processes according to the above-described embodiment can be realized by hardware, software, or a combination thereof. In other words, it suffices if the function of executing the above-mentioned series of processes is realized in any one of the reproduction control systems S, and the mode in which this function is realized is not particularly limited.

For example, when the function of executing the above-mentioned series of processes is realized by a processor that executes arithmetic processing, the processor that executes this arithmetic processing is composed of various processing units such as a single processor, a multiprocessor, and a multicore processor. In addition to the above, the present invention includes a combination of these various processing units and a processing circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

Further, for example, when the function of executing the above-mentioned series of processes is realized by software, the programs constituting the software are installed in the computer via a network or a recording medium. In this case, the computer may be a computer in which dedicated hardware is incorporated, or a general-purpose computer capable of performing a predetermined function by installing a program (for example, a general-purpose personal computer or the like). Electronic devices in general). Further, the step of describing the program may include only the processes performed in time series in the order thereof, but may include the processes executed in parallel or individually. Further, the steps for describing the program may be executed in any order within a range that does not deviate from the gist of the present invention.

The recording medium on which such a program is recorded may be provided to the user by being distributed separately from the computer main body, or may be provided to the user in a state of being incorporated in the computer main body in advance. In this case, the storage medium distributed separately from the computer itself is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versailles Disc), a Blu-ray (registered trademark) Disc (Blu-ray Disc), or the like. The magneto-optical disk is composed of, for example, MD (Mini Disc) or the like. The recording medium provided to the user in a state of being preliminarily incorporated in the computer body is, for example, the ROM 12 of FIG. 2 in which the program is recorded, the ROM 22 of FIG. 3, the storage unit 16 of FIG. 2, or the storage of FIG. It is composed of a hard disk and the like included in the unit 25.

10 Wearable camera, 20 Playback control device, 11,21 CPU, 12,22 ROM, 13,23 RAM, 14,24 communication unit, 15 sensor unit, 16,25 storage unit, 17,26 input unit, 18 imaging unit, 19 Eye tracking unit, 27 output unit, 28 drive, 111 field image capturing unit, 112 gazing point measurement unit, 113 image data generation unit, 114 image data transmission unit, 161 image data storage unit, 211 image data acquisition unit, 212 area Detection unit, 213 gazing point detection unit, 214 feature amount extraction unit, 215 learning unit, 216 scene detection unit, 217 playback control unit, 251 video data storage unit, 217 learning model storage unit, S playback control system, U user

Claims (9)

An image data acquisition means for acquiring a plurality of image data that are continuous in time, and
An area detecting means for detecting a target area including a predetermined target from the image of each of the plurality of image data, and an area detecting means.
A feature amount extraction means for extracting a feature amount from each of the plurality of image data based on a change in an image between the plurality of image data and whether or not the changing region is the target area.
A scene detection means for detecting image data corresponding to a predetermined scene by inputting the feature amount into the learning model, and
A reproduction control means for controlling the reproduction of the plurality of image data based on the information indicating the image data corresponding to the predetermined scene detected by the scene detection means.
A reproduction control device characterized by comprising. When the plurality of image data are continuously reproduced, the reproduction control means reproduces the image data corresponding to the predetermined scene detected by the scene detecting means, and reproduces the other image data. The reproduction control device according to claim 1, wherein the aspect is different from that of the reproduction control device. When the plurality of image data are continuously reproduced, the reproduction control means sets the reproduction speed of the image data corresponding to the predetermined scene detected by the scene detection means from the reproduction speed of the other image data. The reproduction control device according to claim 1 or 2, wherein the reproduction control device is also slowed down. When the plurality of image data are continuously reproduced, the reproduction control means expands and reproduces a part of the image data corresponding to the predetermined scene detected by the scene detection means. The reproduction control device according to any one of claims 1 to 3. The predetermined scene is a scene for viewing the plurality of image data to be continuously reproduced, and is a scene in which a plurality of users collaborate with each other.
The predetermined target is a part of each of a plurality of users who perform the collaborative work.
The plurality of image data are image data obtained by photographing a space corresponding to the field of view of any user performing the collaborative work.
The reproduction control device according to any one of claims 1 to 4, wherein the reproduction control device is characterized. Further provided with a gaze point detecting means for detecting the gaze point of the user who visually recognizes the shooting target when the plurality of image data are taken.
The feature amount extraction means further extracts a feature amount from each of the plurality of image data based on a change in the gazing point of the user who visually recognizes the photographing target between the plurality of image data.
The reproduction control device according to any one of claims 1 to 5, wherein the reproduction control device is characterized. It is characterized by further providing a learning means for constructing the learning model by performing machine learning using a set of input data including the feature amount and a label indicating image data corresponding to the predetermined scene as teacher data. The reproduction control device according to any one of claims 1 to 6. The predetermined scene is a plurality of scenes performed in a predetermined order.
The reproduction control device according to claim 7, wherein the feature amount also includes information indicating the predetermined order. An image data acquisition function that acquires multiple image data that are continuous in time,
An area detection function that detects a target area including a predetermined target from the images of each of the plurality of image data, and an area detection function.
Feature extraction that extracts features from each of the plurality of image data based on the change in the image between the plurality of image data and whether or not the changing region is a region included in the target region. Features and
A scene detection function that detects image data corresponding to a predetermined scene by inputting the feature amount into the learning model, and
A playback control function that controls playback of the plurality of image data based on information indicating image data corresponding to the predetermined scene detected by the scene detection function, and
A playback control program characterized by realizing a computer.

Images