JP2023117955A - Information processing device, imaging apparatus, imaging system, method, and program - Google Patents

Abstract

To provide a user with information on the status of imaging at appropriate timing.SOLUTION: An imaging apparatus sets an area to an image area of an image captured by the imaging apparatus. When at least a portion of a subject in the image area is included in the area, the imaging apparatus notifies a user of an alert. The imaging apparatus controls the size of the area on the basis of the speed at which the imaging apparatus changes its imaging direction.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device, an imaging device, an imaging system, a method, and a program.

2. Description of the Related Art In recent years, when an imaging device takes an image of a scene such as a lecture or sports, automatic imaging is performed by the imaging device. In automatic photography, for example, a subject of interest is detected, and the pan/tilt/zoom (hereinafter referred to as PTZ) of the imaging device is controlled so that the detected subject is always captured in the image captured by the imaging device. . In the future, in addition to simple automatic shooting such as tracking of a detected subject, automatic shooting that realizes the shooting composition intended by the user by a network model using machine learning is expected.

On the other hand, there is a case where the cameraman (user) wants to change the PTZ of the imaging device by his/her own operation to realize a desired photographing composition. As described above, there are use cases in which an object is photographed by manual operation instead of automatic photographing. In this case, the information processing device that controls automatic photography monitors the state of the user's operation related to imaging by the imaging device, and intervenes in the user's operation according to the imaging state. For example, a user may perform an operation of capturing an image of a subject at the edge of an imaging range of an imaging device in order to realize a shooting composition intended by the user. Here, when the subject moves out of the imaging range of the imaging device, the information processing device images the subject prior to the user's operation. A user and an information processing apparatus jointly perform an operation related to image capturing by an image capturing apparatus, so that a higher quality image can be captured.

Therefore, the information processing apparatus presents the user with information according to the imaging situation according to the user's operation, thereby making it possible to perform imaging with an imaging composition more intended by the user. For example, when a user performs an operation of capturing an image of a subject at the end of the imaging range of the imaging device, the information processing device presents to the user whether or not it is possible to capture an image in response to unexpected movement of the subject. As a result, imaging can be performed without intervention of the information processing device while the user is operating the imaging device. Japanese Patent Laid-Open No. 2004-100003 discloses notifying a user that a subject has reached a warning boundary of an imaging range.

JP 2010-147712 A

However, when the information processing apparatus notifies the user of information related to the imaging situation, the timing at which the information should be notified to the user differs depending on the operating speed of the PTZ of the imaging apparatus, the moving speed of the subject, and the like. For example, when the PTZ operation speed of the imaging device is high, presentation of information related to imaging conditions to the user by the information processing device can be delayed. On the other hand, when the PTZ operation speed of the imaging device is low, it is necessary to expedite the presentation of information related to imaging conditions to the user by the information processing device. In the conventional technology, since the information processing apparatus cannot provide the user with information related to the imaging situation at an appropriate timing, it has sometimes been impossible to image the subject with a desired imaging composition while continuing the user's operation.

As described above, there is a problem that the user cannot be provided with information regarding the imaging situation at an appropriate timing.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a user with information related to an imaging situation at an appropriate timing.

In order to achieve the object of the present invention, an information processing apparatus according to one embodiment of the present invention has the following configuration. That is, an information processing apparatus for controlling an image capturing apparatus that captures an image of a subject based on a user's operation includes setting means for setting an area for an image area of a captured image captured by the image capturing apparatus; notification means for notifying a warning when at least part of the area is included in the area, wherein the setting means controls the size of the area based on the speed at which the imaging device changes the imaging direction. characterized by

Advantageous Effects of Invention According to the present invention, it is possible to provide a user with information related to imaging conditions at appropriate timing.

FIG. 2 is a diagram showing an example of the hardware configuration of an information processing apparatus; 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment; FIG. The figure which shows an example of a notification frame and an automatic control frame. 6 is a flowchart for explaining notification processing according to the first embodiment; 4A and 4B are diagrams for explaining sizes of a notification frame and an automatic control frame according to the performance of an imaging device; FIG. 4 is a flowchart for explaining from notification processing to automatic control processing; 4A and 4B are diagrams for explaining the positional relationship between an imaging range and a subject according to the first embodiment; FIG. FIG. 11 is a block diagram showing an example of the functional configuration of an information processing apparatus according to the second embodiment; 9 is a flowchart for explaining notification processing according to the second embodiment; FIG. 5 is a diagram showing an example of a notification frame and an automatic control frame that are set according to the distance to the subject; FIG. 11 is a block diagram showing an example of a functional configuration of an information processing apparatus according to a third embodiment; 4 is a flowchart for explaining from notification processing to automatic control processing; FIG. 10 is a diagram for explaining an example of a notification frame and an automatic control frame according to the moving speed of a subject; FIG. 11 is a block diagram showing an example of a functional configuration of an information processing apparatus according to a fourth embodiment; FIG. 11 is a diagram for explaining selection of a subject on an image according to the fourth embodiment; FIG. 14 is a flowchart for explaining notification processing according to the fourth embodiment; Schematic diagram for explaining from notification processing to automatic control processing.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
The information processing device sets a region for an image region of a captured image captured by the imaging device, and notifies a warning when at least part of a subject appearing in the image region is included in the region. The information processing device controls the size of the area based on the speed at which the imaging device changes the imaging direction. Note that the present embodiment can be used as an imaging device equipped with the functions of an information processing device. Also, the present embodiment can be used as an imaging system in which an information processing device, an imaging device, and a controller are combined.

The image area is an imaging range in which an image of a subject can be captured by an imaging device. In this embodiment, the image area is simply referred to as an image. External devices are user terminals and controllers. User terminals include, for example, smart phones, tablets, notebook PCs, and the like. The imaging direction of the imaging device is the panning and tilting direction of the imaging device. The warning is given by a method of displaying a text message on the screen of the external device or the information processing device, but it may also be given by, for example, a method of sounding a warning sound or a method of vibrating the terminal or the like.

FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing apparatus; The information processing apparatus 100 includes a CPU 101 , a RAM 102 , a ROM 103 , a video reception interface 104 , a video output interface 105 and a network interface 106 .

The CPU 101 is a processor that comprehensively controls each unit in the information processing apparatus 100 . The CPU 101 implements each function of the information processing apparatus 100 by executing arithmetic processing and various programs using control programs stored in the ROM 103 . Note that the information processing apparatus 100 may have one or a plurality of dedicated hardware different from the CPU 101 , and at least part of the processing by the CPU 101 may be executed by the dedicated hardware. For example, dedicated hardware includes ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

The RAM 102 is a read/write volatile memory. A RAM 102 is used as a main memory of the CPU 101 and as a temporary storage area such as a work area.

The ROM 103 is a read-only nonvolatile memory. The ROM 103 has a storage area for storing a boot program, a control program, parameters to be set in each part of the information processing apparatus 100, and the like.

A video reception interface 104 is an interface for receiving a video signal. The video reception interface 104 includes, for example, HDMI (High Definition Multimedia Interface) and SDI (Serial Digital Interface).

A video output interface 105 is an interface for transmitting a video signal. The video output interface 105 includes, for example, HDMI (High Definition Multimedia Interface) and SDI (Serial Digital Interface).

The network interface 106 is an interface unit that performs communication between the information processing apparatus 100 and peripheral devices. The network interface 106 is, for example, a wired network such as a LAN (Local Area Network). The information processing apparatus 100 transmits and receives various types of information to and from peripheral devices via the network interface 106 .

The internal bus 107 is a communication path for exchanging data, control information, and the like. Since each unit in the information processing apparatus 100 is connected by the internal bus 107, it is possible to transmit and receive data, control information, and the like.

The imaging device 110 is a device that captures an image of a subject, and includes, for example, a PTZ camera. The imaging device 110 has a mechanism capable of PTZ operation, and controls the PTZ based on an operation instruction from the information processing device 100 to automatically photograph a subject. The PTZ control instruction is transmitted from the information processing device 100 to the imaging device 110 via the network hub 130, which will be described later. Although one imaging device 110 is shown in FIG. 1 , there may be two or more imaging devices 110 .

The controller 120 is a device that receives various data inputs from the user, and includes, for example, a keyboard, mouse, pointer, and buttons. The controller 120 further includes a display that displays various data, and includes, for example, a touch panel, a liquid crystal display, and an organic EL display. The user can control PTZ operations and the like of the imaging device 110 via the controller 120 . Input of PTZ operation by the user is transmitted to the imaging device 110 via the network hub 130 as a control signal for the imaging device 110 . In this embodiment, it is assumed that the controller 120 is connected to the imaging device 110 via the information processing device 100 . Therefore, the control signal transmitted from the controller 120 is transmitted to the imaging device 110 via the information processing device 100 . Note that the control signal may be transmitted directly to the imaging device 110 without going through the information processing device 100 .

The network hub 130 is a device for consolidating a plurality of cables connected to each node in a network or the like. Since the information processing device 100, the imaging device 110, and the controller 120 are interconnected by the network hub 130, they can transmit and receive data, control information, and the like.

FIG. 2 is a block diagram illustrating an example of the functional configuration of the information processing apparatus according to the first embodiment; Functions of the information processing apparatus are realized by the CPU 101 executing a control program in the ROM 103 . Note that the description of the configuration described with reference to FIG. 1 is omitted.

The input unit 201 acquires images from the imaging device 110 via the image reception interface 104 .

The analysis unit 202 analyzes the video acquired from the imaging device 110 . The analysis unit 202 specifies the two-dimensional coordinates (X, Y) as the position on the image of the subject appearing in each still image (hereinafter referred to as image) that constitutes the video.

The acquisition unit 203 acquires the information of the imaging device 110 (corresponding to the speed of changing the imaging direction) from the imaging device 110 via the network hub 130 . The information of the imaging device 110 is information about the operation speed during PTZ operation, and includes, for example, an index indicating how quickly a designated PTZ position can be reached. Also, the information of the imaging device 110 may further include imaging resolution and frame rate. The imaging resolution indicates how many points constitute one image output from the imaging device 110 . The frame rate represents the number of images that the imaging device 110 can output in one second. As the numerical value of the frame rate increases, the image from the imaging device 110 becomes smoother.

The control unit 204 performs PTZ control on the imaging device 110 and control such as image quality change at the time of imaging. The control unit 204 performs various controls of the imaging device 110 based on the control protocol.

The transfer unit 205 transmits an operation instruction for the imaging device 110 received from the controller 120 to the imaging control unit 206 .

The imaging control unit 206 receives from the transfer unit 205 an operation instruction for the imaging apparatus 110 , image analysis information from the analysis unit 202 , and area setting information from the setting unit 207 . Generate control signals. The imaging control unit 206 transmits the PTZ control signal to the control unit 204 .

The setting unit 207 sets an area for notifying the controller 120 or the like of a warning (notification area) and the image capturing direction of the image capturing apparatus 110 by the information processing apparatus 100 based on the information of the image capturing apparatus 110 (the speed at which the image capturing direction is changed). Set the control start area (automatic control area) on the image. The automatic control area is the area outside the notification area on the image area. As described above, an image (image area) is an imaging range in which an image of a subject can be captured by the imaging device 110 . For example, the setting unit 207 refers to the information of the imaging device 110 and a table of the size of the area and the automatic control area associated with the information of the information of the imaging device 110 to set the size of the notification area and the automatic control area. set. Here, the notification is a notification that warns the user that the subject will no longer appear in the image. The notification refers to, for example, displaying a text message such as "May be lost" on the screen of the user's terminal and the controller 120, but is not limited to this.

The determination unit 208 determines whether or not to notify the controller 120 and the terminal of the user based on the analysis information of the image by the analysis unit 202 and the area (notification area or automatic control area) set on the image by the setting unit 207. determine whether

The reception unit 209 transmits the user operation information for the imaging device 110 transmitted from the controller 120 to the transfer unit 205 and the display control unit 210 via the network hub 130 .

The display control unit 210 controls display of results of processing various data by the information processing apparatus 100 .

The synthesizing unit 211 generates a synthetic image by superimposing notification information generated by the display control unit 210 on each of the still images forming the image captured by the imaging device 110 .

The output unit 212 outputs the synthesized image generated by the synthesizing unit 211 to at least one of the controller 120 and the user's terminal via the video output interface 105 . Note that the output unit 212 may output the composite image to a screen included in the information processing apparatus 100 .

FIG. 3 is a diagram showing an example of a notification frame and an automatic control frame. FIG. 3(a) is a diagram showing an example of a notification frame and an automatic control frame.

An image 301 is an image captured by the imaging device 110 . The image 301 includes a notification frame 302 (illustrated by dashed lines) and an automatic control frame 303 (illustrated by solid lines). Notification frame 302 is a frame for notifying controller 120 of a warning when at least part of subject 304 is positioned outside notification frame 302 . The automatic control frame 303 is a frame in which the information processing apparatus 100 starts controlling the imaging direction of the imaging device 110 when at least part of the subject 304 is positioned outside the automatic control frame 303 . A region between the notification frame 302 and the automatic control frame 303 corresponds to a notification region used when notifying the controller 120 of a warning. An area outside the automatic control frame 303 corresponds to an area (automatic control area) where the information processing apparatus 100 starts controlling the imaging direction of the imaging device 110 . The automatic control frame 303 is positioned outside the notification frame 302 on the image 301 . Note that the distance between the notification frame 302 and the automatic control frame 303 may be a predetermined distance, or a distance may be set according to imaging conditions or the like.

A subject 304 is a subject captured by the imaging device 110, and includes, for example, people, automobiles, and animals. In FIG. 3A, the subject 304 is located outside the range of the image 301. In FIG. The user can control the PTZ position of the imaging device 110 via the controller 120 to project the subject 304 in the image 301 . Various operations according to the position of the subject 304 on the image 301 will be described below using an example of capturing an image while tracking the subject 304 .

FIG. 3(b) shows an image in which the subject is imaged at a normal position. In the image 305, the subject 304 appears to the right of the center of the image 305 (indicated by +). This indicates a state in which the imaging device 110 normally images the subject 304 based on the user's operation via the controller 120 . At this time, the information processing device 100 does not control the imaging direction of the imaging device 110 (see the right diagram in FIG. 3B). In the right diagram of FIG. 3B , the controller 120 transmits a control signal related to the accepted user's operation to the imaging device 110 via the information processing device 100 . Here, the transmission path of the control signal from the information processing device 100 or the controller 120 to the imaging device 110 is indicated by an arrow.

FIG. 3(c) shows an image in which a subject is captured near the boundary of the notification frame. In image 306 , subject 304 appears at a position near the boundary of notification frame 302 . Here, if the subject 304 moves to the right side of the image 306 , the subject 304 will no longer appear in the image 306 . Therefore, the output unit 212 notifies a warning on the screen of the controller 120 in order to correct the user's operation of the imaging device 110 related to imaging. For example, the output unit 212 notifies the screen of the controller 120 of the message “There is a possibility of being lost” in a rectangular area on the image 306 . Accordingly, the user recognizes that an operation to change the imaging direction of the imaging device 110 (for example, an operation to pan the imaging device 110 to the right of the image 306) is required. Note that in the imaging situation as shown in FIG. 3C, priority is given to the user's operation to change the imaging direction of the imaging device 110 . Therefore, the information processing device 100 suppresses the control of the imaging direction of the imaging device 110 (see the right diagram of FIG. 3(c)). In the right diagram of FIG. 3C , the controller 120 transmits a control signal related to the accepted user's operation to the imaging device 110 via the information processing device 100 . Here, the transmission path of the control signal from the information processing device 100 or the controller 120 to the imaging device 110 is indicated by an arrow. The information processing device 100 also transmits a warning signal to the controller 120 .

FIG. 3(d) shows an image of a subject captured on the boundary of the automatic control frame. In the image 307 , the subject 304 appears at a position on the boundary of the automatic control frame 303 and part of the subject 304 is positioned outside the automatic control frame 303 . Here, when the subject 304 moves to the right side of the image 307 , the subject 304 disappears from the image 307 . Therefore, the information processing apparatus 100 changes the imaging direction of the imaging device 110 before the user performs an operation to change the imaging direction of the imaging device 110 (see the right diagram in FIG. 3D). That is, the change control of the imaging direction of the imaging device 110 by the information processing device 100 has priority over the user's operation. Accordingly, the information processing apparatus 100 prevents imaging in which the subject 304 is lost from the image 307 . While the user's operation is disabled, the output unit 212 notifies the screen of the controller 120 of the message "Forcibly operate the camera" within a rectangular area on the image 307 . In the right diagram of FIG. 3( d ), the controller 120 suppresses transmission of the control signal related to the received user's operation to the imaging device 110 via the information processing device 100 . On the other hand, the information processing device 100 transmits a control signal for changing the imaging direction to the imaging device 110 . Here, the transmission path of the control signal from the information processing device 100 or the controller 120 to the imaging device 110 is indicated by an arrow.

As described above, the controller 120 is notified or the information processing apparatus 100 changes the imaging direction of the imaging apparatus 110 according to the position of the subject 304 with respect to the notification frame 302 and the automatic control frame 303 set in the image. As a result, the imaging of the subject 304 can be continued while the user's operation to change the imaging direction of the imaging device 110 is maintained as much as possible.

FIG. 4 is a flowchart for explaining processing for notifying a user according to the first embodiment. This process is implemented by the CPU 101 executing a program in the ROM 103 .

In S<b>401 , the acquisition unit 203 acquires the performance of the imaging device 110 from the imaging device 110 . The performance of the imaging device 110 is the pan/tilt operation speed of the imaging device 110 .

In S<b>402 , the setting unit 207 sets the notification frame 302 and the automatic control frame 303 based on the performance of the imaging device 110 .

In S<b>403 , the input unit 201 receives the image captured by the imaging device 110 . In the processing after S403, each still image (hereinafter referred to as image) forming the video is processed.

In S404, the analysis unit 202 analyzes the image and identifies the position (coordinates) of the subject 304 on the image.

In step S<b>405 , the determination unit 208 determines whether the position of the subject 304 is outside the notification frame 302 based on the position (coordinates) of the subject 304 on the image and the position (coordinates) of the notification frame 302 . judge. Also, the position of the subject 304 includes the position of at least a portion of the subject 304 (eg, head, hands, feet, and torso). If the determination unit 208 determines that the position of the subject 304 is outside the notification frame 302 (Yes in S405), the process proceeds to S406. If the determination unit 208 determines that the position of the subject 304 is not outside the notification frame 302 (No in S405), the process proceeds to S407.

In S406, the display control unit 210 generates display information (warning) indicating that the imaging condition of the imaging device 110 is not good based on the user's operation. The display information (warning) is, for example, a message "There is a possibility of being lost." The synthesizing unit 211 generates a synthetic image in which a warning is superimposed on the image. The output unit 212 notifies the controller 120 of the composite image.

In S407, the output unit 212 determines whether or not the imaging of the subject by the imaging device 110 has ended. When the output unit 212 determines that the imaging has not ended (No in S407), the process returns to S402. When the output unit 212 determines that the imaging has ended (Yes in S407), it ends the processing. The determination condition regarding the end of imaging may be, for example, whether or not a control signal indicating the end of imaging by the user has been received.

Here, a supplementary description will be given of the processing when the setting unit 207 sets the notification frame 302 and the automatic control frame 303 in S402. FIG. 5 is a diagram for explaining sizes of the notification frame and the automatic control frame according to the performance of the imaging device. It is assumed that the sizes of the screens in FIGS. 5A and 5B are the same.

FIG. 5A shows a notification frame and an automatic control frame when the pan/tilt operation speed of the imaging device is fast. FIG. 5(a) shows a notification frame 501 (illustrated by dashed lines), an automatic control frame 502 (illustrated by solid lines), and a distance 503 (illustrated by arrows). A distance 503 is the distance from the center position of the image to the position of the subject 504 .

FIG. 5B shows the notification frame and the automatic control frame when the pan/tilt operation speed of the imaging device is slow. FIG. 5(b) shows a notification frame 511 (illustrated by a dashed line), an automatic control frame 512 (illustrated by a solid line) and a distance 513 (illustrated by an arrow). A distance 513 is the distance from the center position of the image to the position of the subject 504 .

The imaging apparatus 110 with a fast panning/tilting operation speed in FIG. 5A can quickly change the panning/tilting even if the subject 504 exceeds the notification frame 501 , and thus can continue imaging the subject 504 . Therefore, the area of the notification frame 501 is wider than the area of the notification frame 511 . Also, the area of the automatic control frame 502 is wider than the area of the automatic control frame 512 . The length of distance 503 is longer than distance 513 . In this manner, the size of the area of the notification frame and the automatic control frame and the length of the distance can be changed according to the pan/tilt operation speed of the imaging device 110 . Accordingly, it is possible to continue imaging the subject while maintaining the user's operation related to imaging of the imaging device 110 as much as possible.

FIG. 6 is a flowchart for explaining from notification processing to automatic control processing.

In S<b>601 , the information processing apparatus 100 acquires the performance of the imaging device 110 from the imaging device 110 . The performance of the imaging device 110 is the pan/tilt operation speed of the imaging device 110 .

In step S<b>602 , the setting unit 207 sets the notification frame 302 and the automatic control frame 303 based on the performance of the imaging device 110 .

In S<b>603 , the input unit 201 receives the image captured by the imaging device 110 . In the processing after S603, each still image (hereinafter referred to as image) forming the video is processed.

In S604, the analysis unit 202 analyzes the image and identifies the position (coordinates) of the subject 304 on the image.

In step S<b>605 , the determination unit 208 determines whether the position of the subject 304 is outside the notification frame 302 based on the position (coordinates) of the subject 304 on the image and the position (coordinates) of the notification frame 302 . judge. If the determination unit 208 determines that the position of the subject 304 is outside the notification frame 302 (Yes in S605), the process proceeds to S606. If the determination unit 208 determines that the position of the subject 304 is not outside the notification frame 302 (No in S605), the process proceeds to S609.

In step S<b>606 , the determination unit 208 determines whether the position of the subject 304 is outside the automatic control frame 303 based on the position (coordinates) of the subject 304 on the image and the position (coordinates) of the automatic control frame 303 . determine whether If the determination unit 208 determines that the position of the subject 304 is outside the automatic control frame 303 (Yes in S606), the process proceeds to S608. If the determination unit 208 determines that the position of the subject 304 is not outside the automatic control frame 303 (No in S606), the process proceeds to S609.

In S607, the display control unit 210 generates display information (warning) indicating that the imaging condition of the imaging device 110 based on the user's operation is not good. The display information (warning) is, for example, a message "There is a possibility of being lost." The synthesizing unit 211 generates a synthetic image in which a warning is superimposed on the image. The output unit 212 notifies the screen of the user's terminal of the composite image.

In S<b>608 , the control unit 204 controls the imaging direction (panning or tilting direction) of the imaging device 110 so that the subject 304 is positioned inside the notification frame 302 .

In S609, the output unit 212 determines whether or not the imaging of the subject by the imaging device 110 has ended. When the output unit 212 determines that the imaging has not ended (No in S609), the process returns to S603. When the output unit 212 determines that the imaging has ended (Yes in S609), it ends the process. The determination condition regarding the end of imaging may be, for example, whether or not a control signal indicating the end of imaging by the user has been received.

Effects of the present invention will be described with reference to FIG. FIG. 7 is a diagram for explaining the positional relationship between the imaging range and the subject according to the first embodiment.

An image 700 shows an image in which the imaging device 110 captures an object in an appropriate imaging direction based on user's operation. At this time, no warning message is displayed on the image 700 .

An image 701 shows a composite image in which display information (warning) indicating that the imaging condition of the imaging device 110 based on user's operation is not good is superimposed on the captured image. The display information (warning) is, for example, a message "There is a possibility of being lost." Next, by checking the screen on which the image 701 is displayed, the user recognizes that the imaging condition of the imaging device 110 based on the user's operation is not good. Here, the user pans the imaging device 110 rightward so that the subject is positioned closer to the center of the image 701 (indicated by a ten) so that the subject is not lost from the imaging range. Although an example of displaying a text message as a warning on the screen of the user's terminal has been described, the present invention is not limited to this, and for example, the information in the notification frame 302 may be superimposed and displayed on the screen in advance. You can stay. As a result, the user can capture an image of the subject while confirming the quality of the image capturing situation of the user while operating the image capturing apparatus 110 .

An image 702 shows an image of a subject captured by the imaging device 110 after the imaging position of the imaging device 110 is panned to the right (the panning direction is indicated by an arrow) based on the user's operation. As a result, the position of the subject on the image 702 is closer to the center (near the center) than the position of the subject on the image 701 . As a result, the user can continue imaging while maintaining the imaging composition desired by the user while preventing the subject from being lost from the imaging range.

As described above, according to the first embodiment, the size of the area used for notifying the user of the warning is set based on the performance of the imaging device (the speed at which the imaging direction is changed). According to the first embodiment, the imaging status is notified to the user at appropriate timing while the imaging device is performing imaging based on the user's operation. As a result, it is possible to continue imaging by the imaging device based on the user's operation while preventing the subject from being lost from the imaging range.

(Second embodiment)
The second embodiment further controls the sizes of the notification frame and the automatic control frame set based on the performance of the imaging device, based on the distance between the imaging device 110 and the subject. For example, the movement of the subject on the image when the distance between the imaging device 110 and the subject is short is relatively greater than the movement of the subject on the image when the distance between the imaging device 110 and the subject is long. Become. The second embodiment uses the distance between the imaging device 110 and the subject to provide the user with more accurate notification when the imaging device 110 performs imaging based on the user's operation. 2nd Embodiment demonstrates the difference with 1st Embodiment.

FIG. 8 is a block diagram illustrating an example of a functional configuration of an information processing apparatus according to the second embodiment; Note that blocks that are the same as the functional blocks described with reference to FIG. 2 are assigned the same numbers, and description thereof is omitted.

A distance information acquisition unit 801 acquires distance information between the imaging device 110 and the subject that has already been calculated by the imaging device 110 . For example, as an active method, the imaging device 110 irradiates an object with an infrared ray, an ultrasonic wave, a laser, or the like, and calculates the distance to the object based on the time until the reflected wave returns or the angle of the reflected wave. do. Alternatively, the imaging device 110 calculates the distance to the subject based on the blurring of the image of the subject in the image as a passive method (for example, DFD (depth from defocus)).

FIG. 9 is a flowchart for explaining notification processing according to the second embodiment.

In step S<b>901 , the acquisition unit 203 acquires the performance (pan/tilt operation speed) of the imaging device 110 from the imaging device 110 .

In S<b>902 , the setting unit 207 sets the notification frame 302 and the automatic control frame 303 based on the performance of the imaging device 110 . Here, the notification frame 302 and the automatic control frame 303 are temporary frames set based on the performance of the imaging device 110 . The sizes of the notification frame 302 and the automatic control frame 303 are updated by processing described later.

In S<b>903 , the input unit 201 receives the image captured by the imaging device 110 . In the processing after S903, each still image (hereinafter referred to as image) forming the video is processed.

In S904, the analysis unit 202 analyzes the image and identifies the position (coordinates) of the subject 304 on the image.

In S<b>905 , the distance information acquisition unit 801 acquires distance information between the imaging device 110 and the subject 304 from the imaging device 110 .

In S<b>906 , the setting unit 207 updates the notification frame 302 and automatic control frame 303 set in S<b>902 based on the distance information between the imaging device 110 and the subject 304 . When different distance information is used for each still image (image), the setting unit 207 updates the notification frame 302 and the automatic control frame 303 to have different sizes for each image. The details of this process will be described later.

In step S<b>907 , the determination unit 208 determines whether the position of the subject 304 is outside the notification frame 302 based on the position (coordinates) of the subject 304 on the image and the position (coordinates) of the notification frame 302 . judge. If the determination unit 208 determines that the position of the subject 304 is outside the notification frame 302 (Yes in S907), the process proceeds to S908. If the determination unit 208 determines that the position of the subject 304 is not outside the notification frame 302 (No in S907), the process proceeds to S909.

In S908, the display control unit 210 generates display information (warning) indicating that the imaging condition of the imaging device 110 based on the user's operation is not good. The display information (warning) is, for example, a message "There is a possibility of being lost." The synthesizing unit 211 generates a synthetic image in which a warning is superimposed on the image. The output unit 212 notifies the screen of the user's terminal of the composite image.

In S909, the output unit 212 determines whether or not the imaging of the subject by the imaging device 110 has ended. When the output unit 212 determines that the imaging has not ended (No in S909), the process returns to S903. If the output unit 212 determines that the imaging has ended (Yes in S909), it ends the process. The determination condition regarding the end of imaging may be, for example, whether or not a control signal indicating the end of imaging by the user has been received.

Here, specific processing of S906 will be described with reference to FIG. FIG. 10 is a diagram showing an example of a notification frame and an automatic control frame that are set according to the distance to the subject.

FIG. 10A is a diagram showing an example when the distance between the imaging device and the subject is long. The upper diagram of FIG. 10A shows a subject 304 on the image, a notification frame 1001, an automatic control frame 1002, and a distance 1003. FIG. The lower diagram of FIG. 10( a ) shows the distance 1004 between the imaging device 110 and the object 304 . Since distance 1004 is longer than distance 1014, which will be described later, the amount of relative movement of object 304 in the image is small. Therefore, the notification frame 1001 and the automatic control frame 1002 are set as large frames within the image. Here, the setting unit 207 updates the sizes of the notification frame 302 and the automatic control frame 303 by referring to a table that associates the distance information between the imaging device 110 and the subject 304 with the sizes of various frames. .

FIG. 10B is a diagram showing an example when the distance between the imaging device and the subject is short. The upper diagram of FIG. 10B shows the subject 304 on the image, the notification frame 1011, the automatic control frame 1012, and the distance 1013. FIG. The lower diagram of FIG. 10B shows the distance 1014 between the imaging device 110 and the subject 304 . Since distance 1014 is shorter than distance 1004, the amount of relative movement of subject 304 in the image is greater. Therefore, the notification frame 1011 and the automatic control frame 1012 are set as small frames within the image.

As described above, according to the second embodiment, the sizes of the notification frame and the automatic control frame on the image are updated based on the distance information between the imaging device and the subject. As a result, it is possible to set a notification frame or the like with higher accuracy according to the imaging situation, and it is possible to notify the user of the imaging situation at an appropriate timing.

(Third embodiment)
The third embodiment further controls the size of the notification frame and the automatic control frame based on the feature information (moving speed) of the subject. For example, a fast-moving subject moves out of the image (imaging range) of the imaging device in a short time compared to a slow-moving subject. Therefore, in the third embodiment, the moving speed information of the subject is used to update the sizes of the notification frame and the automatic control frame that are set based on the performance of the imaging device, so that the user can obtain a more accurate imaging situation. to notify. 3rd Embodiment demonstrates the difference with 1st Embodiment and 2nd Embodiment.

FIG. 11 is a block diagram showing an example of the functional configuration of an information processing apparatus according to the third embodiment; Note that blocks that are the same as the functional blocks described in FIG. 2 are assigned the same numbers, and the description thereof is omitted.

A recognition unit 1101 performs recognition processing on a subject specified in an image. Here, the recognition process is, for example, the process of identifying whether the subject is a person, an automobile, or an animal. The recognition unit 1101 may use various techniques when identifying the subject, and may use, for example, a network model obtained by machine learning.

The information acquisition unit 1102 acquires information on the recognized subject. For example, the subject information is information about the maximum moving speed when the subject moves in space. It is also assumed that subject information is registered in advance as a database in a storage unit (not shown) of the information processing apparatus 100 .

FIG. 12 is a flowchart for explaining from notification processing to automatic control processing.

In S<b>1201 , the acquisition unit 203 acquires the performance (pan/tilt operation speed) of the imaging apparatus 110 .

In step S<b>1202 , the setting unit 207 sets the notification frame 302 and the automatic control frame 303 based on the performance of the imaging device 110 . Here, the notification frame 302 and the automatic control frame 303 are temporary frames set based on the performance of the imaging device 110 . The sizes of the notification frame 302 and the automatic control frame 303 are updated by processing described later.

In S<b>1203 , the input unit 201 receives the image captured by the imaging device 110 . In the processing after S1203, each still image (hereinafter referred to as image) forming the video is processed.

In S1204, the analysis unit 202 analyzes the image and identifies the position (coordinates) of the subject 304 on the image.

In S1205, the recognition unit 1101 identifies the subject 304 in the image. In this embodiment, the subject 304 is a person or an automobile, but is not limited to these.

In S<b>1206 , the information acquisition unit 1102 acquires information on the identified subject 304 from the storage unit (not shown) of the information processing apparatus 100 . The information on the subject 304 is information on the maximum moving speed when the subject 304 moves in space.

In S<b>1207 , the setting unit 207 updates the sizes of the notification frame 302 and automatic control frame 303 set in S<b>1202 based on the information about the subject 304 . The details of this process will be described later.

In S<b>1208 , the determination unit 208 determines whether the position of the subject 304 is outside the notification frame 302 based on the position (coordinates) of the subject 304 on the image and the position (coordinates) of the notification frame 302 . judge. If the determination unit 208 determines that the position of the subject 304 is outside the notification frame 302 (Yes in S1208), the process proceeds to S1209. If the determination unit 208 determines that the position of the subject 304 is not outside the notification frame 302 (No in S1208), the process proceeds to S1210.

In S1209, the display control unit 210 generates display information (warning) indicating that the imaging condition of the imaging device 110 based on the user's operation is not good. The display information (warning) is, for example, a message "There is a possibility of being lost." The synthesizing unit 211 generates a synthetic image in which a warning is superimposed on the image. The output unit 212 notifies the screen of the controller 120 of the composite image.

In S1210, the output unit 212 determines whether or not the imaging of the subject by the imaging device 110 has ended. When the output unit 212 determines that the imaging has not ended (No in S1210), the process returns to S1203. When the output unit 212 determines that the imaging has ended (Yes in S1210), it ends the process. The determination condition regarding the end of imaging may be, for example, whether or not a control signal indicating the end of imaging by the user has been received.

Here, specific processing of S1207 will be described with reference to FIG. FIG. 13 is a diagram illustrating an example of a notification frame and an automatic control frame according to the moving speed of the subject.

FIG. 13A is a diagram illustrating an example of a notification frame and an automatic control frame when the subject is a person. FIG. 13A shows a subject 304 on an image, a notification frame 1301, an automatic control frame 1302, and a distance 1303. FIG. A subject 304 is a person.

FIG. 13B is a diagram illustrating an example of the notification frame and the automatic control frame when the subject is an automobile. FIG. 13B shows a subject 1314 on the image, a notification frame 1311, an automatic control frame 1312, and a distance 1313. FIG. A subject 1314 is an automobile.

Comparing the moving speeds of subject 304 (person) and subject 1314 (car), subject 1314 moves faster than subject 304 . Therefore, the setting unit 207 sets the notification frame 1311 and the automatic control frame 1312 on the image in which the subject 1314 moving fast is displayed to be smaller than the notification frame 1311 and the automatic control frame 1312 on the image in which the subject 304 is displayed. do.

As described above, according to the third embodiment, the size of the notification frame or the like is further controlled based on subject information (for example, moving speed). As a result, it is possible to set a more accurate notification frame according to the characteristics of the subject, and to notify the user of the imaging situation at an appropriate timing.

(Fourth embodiment)
The fourth embodiment will explain the setting of the notification frame and the automatic control frame when a plurality of subjects are shown in the image. In the fourth embodiment, notification processing is performed in consideration of imaging conditions according to the positions of a plurality of subjects on an image. 4th Embodiment demonstrates the difference with 1st Embodiment, 2nd Embodiment, and 3rd Embodiment.

FIG. 14 is a block diagram illustrating an example of a functional configuration of an information processing apparatus according to the fourth embodiment; Note that blocks that are the same as the functional blocks described with reference to FIGS. 2 and 11 are given the same numbers and their description is omitted.

A management unit 1401 (corresponding to storage control means) stores information on the recognized subject in a memory. Here, the subject information is information indicating whether or not the subject is to be tracked. Based on the subject information, the information processing apparatus 100 performs notification or imaging control of the imaging apparatus 110 by the information processing apparatus 100 only when the specified subject is the imaging target. It should be noted that whether or not the subject specified in the image is to be captured is determined based on subject selection information by the user. Subject selection information is information that the user has selected a subject via the controller 120 , and is recorded in the memory by the management unit 1401 via the reception unit 209 and the display control unit 210 .

FIG. 15 is a diagram illustrating selection of a subject on an image according to the fourth embodiment. An example of setting whether or not a subject is to be imaged will be described with reference to FIG. 15 .

FIG. 15 shows a subject 304 on an image, a notification frame 1502, displays 1503, and 1504. FIG. As shown in FIG. 15 , when subject 304 is positioned adjacent to notification frame 1502 , output unit 212 outputs a composite image including display 1503 on the screen of controller 120 . Here, if the management unit 1401 does not store information about the subject 304, the display control unit 210 superimposes a display 1504 on the composite image so that the user can select whether or not the subject 304 is to be captured. to display.

By selecting a display 1504 on the screen via the controller 120 (or touching the screen), the user can set the subject 304 on the composite image as a non-imaging target. Then, the storage unit (not shown) of the information processing apparatus 100 stores information (corresponding to the determination result) regarding the subject 304 as a non-imaging target. After that, the management unit 1401 stores information on whether or not the subject 304 is to be tracked together with information on the subject.

FIG. 16 is a flowchart illustrating notification processing according to the fourth embodiment.

In S<b>1601 , the acquisition unit 203 acquires the performance (pan/tilt operation speed) of the imaging device 110 from the imaging device 110 .

In step S<b>1602 , the setting unit 207 sets the notification frame 302 and the automatic control frame 303 based on the performance of the imaging device 110 .

In S<b>1603 , the input unit 201 receives the image captured by the imaging device 110 . In the processing after S1603, each still image (hereinafter referred to as an image) forming the video is processed.

In S1604, the analysis unit 202 analyzes the image and identifies the number of subjects 304 in the image. In subsequent processing, the processing is repeated according to the number of subjects 304 .

In S1605, the recognition unit 1101 identifies the subject 304 in the image.

In S<b>1606 , the information acquisition unit 1102 acquires subject information from the management unit 1401 .

In step S1607, the determination unit 208 determines whether or not the subject 304 in the image is an imaging target based on comparison between the identified subject 304 in the image and subject information. If the determination unit 208 determines that the subject 304 in the image is not the imaging target (No in S1607), the process proceeds to S1611. If the determination unit 208 determines that the subject 304 in the image is the imaging target (Yes in S1607), the process advances to S1608.

In S1608, the analysis unit 202 identifies the position of the subject 304 to be imaged on the image.

In step S<b>1609 , the determination unit 208 determines whether the position of the subject 304 is outside the notification frame 302 based on the position (coordinates) of the subject 304 on the image and the position (coordinates) of the notification frame 302 . judge. If the determining unit 208 determines that the position of the subject 304 is outside the notification frame 302 (Yes in S1609), the process proceeds to S1610. If the determination unit 208 determines that the position of the subject 304 is not outside the notification frame 302 (No in S1609), the process proceeds to S1611.

In S1610, the display control unit 210 generates display information (warning) indicating that the imaging condition of the imaging device 110 is not good based on the user's operation. The display information (warning) is, for example, a message "There is a possibility of being lost." The synthesizing unit 211 generates a synthetic image in which a warning is superimposed on the image. The output unit 212 notifies the screen of the controller 120 of the composite image.

In S<b>1611 , the output unit 212 determines whether or not processing related to imaging conditions for all subjects in the image has been completed, based on the number of subjects 304 in the image. When the output unit 212 determines that the processing related to the imaging status of all subjects on the image has not been completed (No in S1611), the process returns to S1607. When the output unit 212 determines that the processing related to the imaging status for all subjects on the image has been completed (Yes in S1611), the process proceeds to S1612.

In S1612, the output unit 212 determines whether the imaging of the subject by the imaging device 110 has ended. When the output unit 212 determines that the imaging has not ended (No in S1212), the process returns to S1203. When the output unit 212 determines that the imaging has ended (Yes in S1212), it ends the process. The determination condition regarding the end of imaging may be, for example, whether or not a control signal indicating the end of imaging by the user has been received.

Here, an example of a shooting control method by the information processing apparatus when a plurality of shooting targets are present in the image will be described with reference to FIG. 17 . FIG. 17 is a schematic diagram for explaining from notification processing to automatic control processing.

FIG. 17(a) shows an image of two subjects. FIG. 17A shows a subject 1701, a subject 1702, a notification frame 1703, and an automatic control frame 1704 on an image. A subject 1701 is a person. A subject 1702 is, for example, a ball.

FIG. 17(b) shows an image in which two subjects are captured adjacent to the notification frame. A subject 1701 and a subject 1702 are positioned at both ends inside the notification frame 1703 . So far, the determination unit 208 has determined whether or not to issue a warning notification to the controller 120 based on whether or not at least part of the subject is positioned outside the notification frame 1703 . Here, the determination unit 208 may determine whether or not to notify the controller 120 of the warning based on whether or not at least part of the subject is positioned adjacent to the notification frame 1703 . . Therefore, the display control unit 210 generates display information (warning) under each of the two subjects on the image. The display information (warning) is, for example, a message "There is a possibility of being lost." The synthesizing unit 211 generates a synthetic image in which two warnings are superimposed on the image. The output unit 212 notifies the controller 120 of the composite image. In FIG. 17(b), a message "may be lost" is displayed below the subject 1701 and subject 1702, respectively.

FIG. 17(c) shows an image in which one of the two subjects is positioned outside the automatic control frame. A subject 1702 is positioned adjacent to an automatic control frame 1704 . Here, when both the subject 1701 and the subject 1702 are to be imaged, the control unit 204 performs zoom control of the image capturing apparatus 110 to zoom out the image. Note that the control unit 204 suppresses zoom control of the imaging device 110 when neither the subject 1701 nor the subject 1702 is positioned adjacent to the automatic control frame 1704 .

FIG. 17(d) is a diagram illustrating how two subjects are zoomed out. The image in FIG. 17(d) shows the image after zooming out, and the subject 1701 and the subject 1702 in the image are displayed in a smaller size than the subjects in FIGS. 17(a) to 17(c). . Also, the subject 1701 and the subject 1702 are positioned away from the notification frame 1703 . As a result, even when one of the two subjects is adjacent to the automatic control frame 1704, the subject 1701 and the subject 1702 can be imaged within the image (imaging range).

As described above, according to the fourth embodiment, it is possible to allow the user to select an imaging target when a plurality of subjects are present in an image. As a result, it is possible to continue imaging so as to achieve the imaging composition intended by the user, and to notify the user of the imaging status at an appropriate timing.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: information processing device, 101: CPU, 102: RAM, 103: ROM, 104: video reception interface, 105: video output interface, 106: network interface, 107: internal bus, 110: imaging device, 120: controller

Claims (16)

setting means for setting an area for an image area of a captured image captured by the imaging device;
notification means for notifying a warning when at least part of a subject appearing in the image area is included in the area;
The setting means controls the size of the area based on the speed at which the imaging device changes the imaging direction.
An information processing device characterized by: further comprising acquisition means for acquiring characteristics of the subject;
The setting means further controls the size of the area based on the features of the subject.
The information processing apparatus according to claim 1, characterized by: The characteristics of the subject include the distance between the subject and the imaging device and the maximum moving speed of the subject,
The setting means further controls the size of the region based on at least one of a distance between the subject and the imaging device and a maximum moving speed of the subject.
3. The information processing apparatus according to claim 2, characterized by: The setting means further sets an automatic control region for controlling the imaging direction of the imaging device based on the control of the information processing device to the image region,
The automatic control area is an area outside the area on the image area,
4. The information processing apparatus according to any one of claims 1 to 3, characterized by: Further comprising control means for controlling the imaging direction of the imaging device,
When at least a part of the subject is included in the automatic control area, the control means controls the imaging direction of the imaging device so that the subject appears in the center of the image area.
5. The information processing apparatus according to claim 4, characterized by: determining means for determining whether or not the subject appearing in the image area is a predetermined subject, based on a user's operation;
further comprising storage control means for storing the result of determination by the determination means in a memory;
The information processing apparatus according to any one of claims 1 to 5, characterized in that: identification means for identifying a plurality of subjects appearing in the image area;
determining means for determining whether or not the plurality of identified subjects are the defined subjects based on the determination result and the plurality of subjects identified by the identification section;
7. The information processing apparatus according to claim 6, characterized by: The notification means notifies the warning when at least part of the specified subject is included in the area.
8. The information processing apparatus according to claim 7, characterized by: zoom control means for controlling zooming of the imaging device based on whether or not at least part of the specified subject is included in an automatic control area that is an area outside the area;
The zoom control means controls the zoom of the imaging device when at least part of the specified subject is included in the automatic control area.
9. The information processing apparatus according to claim 7 or 8, characterized by: The zoom control means suppresses control of the zoom of the imaging device when at least part of the specified subject is not included in the automatic control area.
10. The information processing apparatus according to claim 9, characterized by: The imaging direction is the pan and tilt direction of the imaging device,
The information processing apparatus according to any one of claims 1 to 10, characterized in that: The notification means notifies the warning to at least one of a user's terminal and a controller that controls the imaging device;
The information processing apparatus according to any one of claims 1 to 11, characterized in that: An imaging device that controls an imaging direction based on a user's operation,
imaging means for imaging a subject;
setting means for setting an area for an image area of a captured image captured by the imaging means;
notification means for notifying a warning when at least part of the subject is included in the area;
The setting means controls the size of the area based on the speed at which the imaging means changes the imaging direction.
An imaging device characterized by: an imaging device;
a controller that controls the imaging direction of the imaging device;
an information processing apparatus according to any one of claims 1 to 12;
An imaging system comprising: A method executed by an information processing device, comprising:
a setting step in which the setting means of the information processing device sets a region for an image region of a captured image captured by an imaging device;
a notification step in which the notification means of the information processing device notifies a warning when at least part of the subject appearing in the image area is included in the area;
The setting means of the information processing device controls the size of the region based on the speed at which the imaging device changes the imaging direction in the setting means.
A method characterized by: A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 12.

Images