JP2021111902A - Information processing device, information processing method, and program - Google Patents

Abstract

To more appropriately suppress blurring of a captured image.SOLUTION: A control unit 703 is configured to acquire positional information, an estimated result of a distance between the imaging unit 701 and a subject, and a measurement result of an amount of swing of the frame including at least an angle swing of the frame in a tilt direction, the positional information being according to a position with respect to a frame of an imaging unit 701, which is supported by the frame of a vessel 100. The control unit 703 is configured to control a posture of the imaging unit 701 so that a change in the direction of an optical axis of the imaging unit 701 in accordance with the swing of the frame of the vessel 100 is corrected based on the positional information, the estimated result of the distance, and the measurement result of the amount of swing.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to information processing devices, information processing methods, and programs.

Conventionally, there has been known a technique for correcting blurring that becomes apparent in an image due to shaking of the image pickup apparatus (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2006-74402 Japanese Unexamined Patent Publication No. 7-218960

An object to be solved by the present invention is to more preferably suppress blurring of a captured image.

The information processing apparatus according to the present invention includes position information according to the position of an imaging unit supported by a predetermined housing with respect to the housing, an estimation result of a distance between the imaging unit and a subject, and at least the housing. The acquisition means for acquiring the measurement result of the amount of shaking of the housing including the angular shaking in the predetermined rotation direction of the body, the position information, the estimation result of the distance, and the measurement result of the amount of shaking. Based on the above, the control means for controlling the posture of the imaging unit is provided so that the change in the direction of the optical axis of the imaging unit in response to the shaking of the housing is corrected.

It is explanatory drawing for demonstrating the blur appearing in an image. It is explanatory drawing for demonstrating the blur appearing in an image. It is explanatory drawing for demonstrating the blur appearing in an image. It is explanatory drawing for demonstrating shift blur. It is explanatory drawing for demonstrating shift blur. It is explanatory drawing for demonstrating shift blur. It is a block diagram which showed an example of the functional structure of an image pickup apparatus. It is a block diagram which showed another example of the functional structure of an image pickup apparatus. It is explanatory drawing for demonstrating the correction of the blur appearing in an image. It is a figure which showed an example of the hardware configuration of an information processing apparatus.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted. Further, in the following description, in order to make it easier to understand the features of the technique according to the embodiment of the present invention, the distance between the lens surface and the image sensor in the image pickup unit is set between the subject and the lens surface. Assume that it is negligibly short compared to the distance.

<Technical Thought>
With reference to FIGS. 1 to 6, the basic concept of the technique for correcting the blurring that becomes apparent in the image according to the image pickup result due to shaking or the like by the image pickup apparatus according to the present embodiment will be described.

For example, FIG. 1 is a conceptual diagram showing an example of the installation conditions of the image pickup device according to the present embodiment, and is a state in which the image pickup device 101 according to the present embodiment is installed on the ship 100 (in other words, the housing of the ship 100). The state in which the image pickup apparatus 101 is supported) is schematically shown. Specifically, FIG. 1 is a conceptual diagram showing a situation in which a ship 100 is navigating on a sea surface 103 on a plane perpendicular to the traveling direction of the ship 100. In the example shown in FIG. 1, the image pickup device 101 is installed at a position higher than the center of gravity 102 of the ship 100 by a height H in order to enable imaging of another ship farther away.

Here, with reference to FIG. 2, when the line-of-sight distance d by the image pickup apparatus 101 is used, the relationship between the line-of-sight distance d and the height H will be outlined. In the example shown in FIG. 2, the center of the earth is O and the radius of the earth is R. Further, when the height of the position A where the image pickup device 101 is installed from the sea surface is H', the position that can be imaged by the image pickup device 101 is limited to the position B separated by the distance d. Under the above conditions, the height H'is represented by the conditional expression shown in the following equation (1-1).

In equation (1-1), the radius R of the earth is 6400 km, and the braking distance of large vessels is generally said to be about 5 km. Therefore, if the line-of-sight distance d is double that, 10 km, the required height The H'is about 8 m. Further, since the center of gravity of the ship is below the waterline, H> H'. Therefore, in this case, the height H is required to be, for example, a height of about a dozen meters.

Here, when a right-handed coordinate system is constructed based on the y-axis with the direction from the center of the earth toward the center of gravity of the ship as the positive direction and the z-axis with the direction of travel of the ship 100 as the positive direction, The coordinate system 104 shown in FIG. 1 is obtained. In the following, in order to make the explanation easier to understand, the position of the center of gravity of the ship 100 is set as the origin of the coordinate system 104. In this case, the xy coordinate of the image pickup apparatus 101 is (0, H). Further, in the example shown in FIG. 1, it is assumed that the optical axis 105 of the image pickup apparatus 101 is oriented in the negative direction along the x-axis as shown by the arrow. That is, in the example shown in FIG. 1, it is assumed that the imaging device 101 is installed so that the imaging direction is a negative direction along the x-axis.

Here, with reference to FIG. 3, a case where the ship 100 is shaken due to waves will be considered. In the example shown in FIG. 3, the ship 100 receives waves from a direction lateral to the traveling direction, and rotation in a plane perpendicular to the traveling direction, so-called rolling or rolling, occurs. The situation is schematically shown. In order to make the explanation easier to understand, the imaging direction and the direction of shaking are fixed, but the application destination of the technique according to the present embodiment is not necessarily limited. That is, the imaging direction and the direction of the shaking are not particularly limited as long as it is possible to absorb the influence of the shaking based on the control of the posture of the imaging device 101 along the desired rotation direction such as the tilt direction. As a specific example, even if the imaging direction of the imaging device 101 is along the z-axis, for example, when the shaking direction is the rotation direction centered on the x-axis, details will be described below. It is possible to apply the technique according to the present embodiment.

Here, when the angle 302 formed by the straight line 301 passing through the center of gravity 102 of the ship 100 and the image pickup device 101 and the y-axis is φ, the gyro disposed in the image pickup device 101 or the ship 100. It is assumed that a detector such as a sensor that detects shaking outputs an angle φ. This blurring of angle φ is also referred to as “angle blurring” hereafter.
In the example shown in FIG. 3, since the optical axis 105 of the image pickup device 101 is tilted by an angle φ, the optical axis 105 is oriented toward the sea surface, and the image corresponding to the image pickup result by the image pickup device 101 is greatly blurred. It becomes. For example, 303 schematically shows the optical axis 105 in which the angle blur has occurred. Under such a situation, for example, PT control is performed in the tilt direction so that the optical axis 303 in which the angle blur occurs becomes the optical axis shown in 304, so that the image pickup result by the image pickup apparatus 101 can be obtained. The effect of shaking that becomes apparent in the image is absorbed (correction of blurring).

Further, as the blur that becomes apparent in the image according to the image pickup result by the image pickup device 101 due to the shaking of the ship 100, in addition to the angle blur shown by the angle φ, the image pickup device 101 translates along the y-axis direction. There is a blur that becomes apparent when the position of the optical axis 105 is deviated by Δh. This blur of Δh is also referred to as “shift blur” hereafter.

Here, with reference to FIGS. 4, 5, and 6 together with FIG. 3, the change in the position on the optical sensor on which the image of the subject is formed due to the shift blur will be described below.
First, FIG. 4 will be described. FIG. 4 is a schematic diagram schematically showing the positional relationship between each of the lens 401 and the image sensor (optical sensor) 403 of the image pickup device 101 and the subject 402. In the example shown in FIG. 4, it is assumed that the lens 401 and the subject 402 face each other in a state of being separated by a distance l.
In the present disclosure, the term "face-to-face" means that the surface of the subject to be imaged (hereinafter, also referred to as "imaging surface") is substantially perpendicular to the optical axis of the imaging device. In the following, in order to make the explanation easier to understand, the optical axes of the image sensor 403 and the lens 401 are substantially parallel to the x-axis, and the y-axis is abbreviated with respect to the image pickup surface of the subject 402 and the sensor surface of the image sensor 403. A parallel coordinate system shall be set.
That is, in the example shown in FIG. 4, the direction (for example, the vertical direction) in which the translational component of the sway due to waves or the like is directed, which is considered as shift blur (in other words, is subject to correction of shift blur described later), is the y-axis. The coordinate system is set so that it substantially matches the direction. Further, the x-axis direction corresponds to the direction perpendicular to the translational component (for example, the horizontal direction).

It is assumed that the image pickup device 403 is disposed with respect to the lens 401 along the x-axis by a focal length f. Further, the size of the sensor surface of the image sensor 403 is assumed to be 2h in height and 2w in width when the direction along the y-axis is the height direction. When the height above the optical axis of the subject 402 is t1, the image height t2 on the sensor surface of the image sensor 403 is represented by the conditional expression shown in the following equation (1-2).

That is, when the image height t2 is smaller than the height h of the sensor surface of the image sensor 403, the subject is imaged in the image without being removed from the frame.

Next, FIG. 5 will be described. FIG. 5 is a schematic view schematically showing a situation in which the image sensor rotates by an angle φ around an axis substantially orthogonal to each of the x-axis and the y-axis with the center of the sensor surface of the image sensor 403 as the center of rotation. .. The lens 501 and the subject 502 shown in FIG. 5 correspond to the lens 401 and the subject 402 shown in FIG. 4, respectively. In the example shown in FIG. 5, the image height t2'of the subject 502 on the sensor surface of the image sensor 403 is represented by the conditional expression shown in the following equation (1-3).

Here, the optical axis passing through the center of the rotated lens 501 and substantially perpendicular to the sensor surface of the image sensor 403 is defined as x', and the axis rotated by an angle φ with respect to the y-axis is defined as y'. Further, the intersection of the line segment drawn from the end point of the subject 502 so as to be perpendicular to the optical axis x'and the optical axis x'is u, and the distance between the intersection u and the center of the lens 501. Is l', and the distance between the end point of the drawn line segment and the intersection u is t'1. In this case, the distances l'and t'1 can be calculated based on the conditional expressions shown below as equations (1-4) and (1-5), respectively.

In the above equations (1-4) and (1-5), the above-mentioned approximation based on the relationship of l >> f is applied. Substituting the distances l'and t'1 based on the above equations (1-4) and (1-5) into the above equation (1-3), the conditional expression shown below as the equation (1-6) is obtained. Will be converted.

Here, focusing on the case where the image height t2 represented by the equation (1-2) changes to the image height t'2 represented by the equation (1-6) due to the rotational fluctuation of the angle φ, the angle φ The amount of change Δt in the image height due to rotational shaking is expressed by the conditional expression shown in the following equation (1-7).

Further, FIG. 6 schematically shows a situation in which the position of the image pickup apparatus is shifted by Δh along the y-axis due to the translational component in the y-direction (for example, the vertical direction) of the shaking caused by waves or the like. It is a schematic diagram. Specifically, FIG. 6 shows that the optical axis of the image pickup element 403 is located downward (−y direction) by Δh from the x-axis from a position substantially coincident with the x-axis, and with respect to the x-axis. It shows the situation where it has moved to a position that roughly coincides with the x'axis that is almost parallel. In this case, the image height t ″ 2 of the subject is expressed by the conditional expression shown by the following equation (1-8).

Here, when the amount of change in the image height of the subject due to the shift movement is Δt2 = t''2-t2, fΔh / l is the height of the sensor surface of the image sensor 403 based on the equation (1-8). When it is sufficiently smaller than 2h, Δt2 can be ignored. That is, it is not necessary to consider the fluctuation of the image height with respect to the shift movement. The amount of change Δt2 in the image height in this case is expressed by the conditional expression shown in the following equation (1-9).

If it is not necessary to consider macro photography and the like, the condition of f << 1 is generally satisfied. Therefore, if Δh is not large, the conditional expression shown in Eq. (1-9) is sufficiently satisfied.

On the other hand, from Eq. (1-7), the conditions under which the amount of change in image height Δt due to rotation can be ignored are limited. For example, when the sensor size of the image sensor is full size (36 mm × 24 mm) and the focal length f is 50 mm, the order of the focal length f and the order of the sensor size (2h) are about the same. It becomes difficult to ignore the amount of change Δt in the image height. In general, when an ultra-wide-angle lens is used and the shaking angle φ is relatively small, the amount of change Δt in the image height due to the shaking can be ignored.

In the example shown in FIG. 3, the image pickup apparatus 101 cancels the change in image height with respect to rotation by controlling the posture along the tilt direction so that the angle φ becomes substantially equal to 0 in the equation (1-7). It will be possible to go out. Here, the amount of change in image height related to shift movement, which is generally negligible, is estimated for the case where the image pickup device 101 is installed on the ship 100 as shown in FIG. Specifically, as shown in FIG. 3, when the ship 100 is tilted by an angle φ with respect to the center of gravity 102, the shift movement amount of the imaging device 101 is a conditional expression shown by the following equation (1-10). expressed.

Here, the height H is on the order of 10 m from the formula (1-1) in order to install the image pickup device 101 on the ship 100 so as to be able to see 10 km ahead. For example, when the order is substantially equal to the distance l to the subject, the order O (Δt) of the amount of change in the image height due to the shift movement is expressed by the conditional expression shown in the following equation (1-11). The order is sufficient for the sensor size (2h).

Therefore, as shown in FIG. 3, even if the posture of the image pickup apparatus 101 is rotated by an angle φ in the tilt direction and the optical axis is returned to the direction indicated by 304, the change in image height due to the shift movement can be ignored. Due to the difficulty, the subject may be out of the frame. The present embodiment is characterized in that the posture of the image pickup apparatus 101 is further rotated by an angle θ in the tilt direction to direct the optical axis in the direction indicated by 305 in order to reduce the possibility that the subject is out of the frame due to shift blur. .. Specifically, the image height changed by the shift movement is shown by the equation (1-8), and by substituting this image height into t2 of the equation (1-6), t'2 becomes t2 as a result. Therefore, the angle θ may be set as the following equation (1-12) so as to satisfy the conditional equation.

When the above equation (1-12) is solved with respect to the angle θ, the angle θ is expressed by the conditional equation shown below as the equation (1-13).

In the following description, the angle θ represented by the above equation (1-13) is also referred to as a “shift correction angle θ” for convenience.

As described above, with reference to FIGS. 1 to 6, the outline of the basic concept of the technique for correcting the blurring that becomes apparent in the image according to the image pickup result due to shaking or the like by the image pickup apparatus according to the present embodiment has been described.

<Structure example 1>
As a configuration example 1, an example of the functional configuration of the image pickup apparatus according to the present embodiment will be described with reference to FIG. 7. The image pickup device 101 includes an image pickup unit 701, a drive unit 702, a control unit 703, a distance estimation unit 704, a shake measurement unit 705, and an installation position setting unit 706. The image pickup device 101 is installed on the ship 100 as shown in FIG. 1, and the subject is subject to shaking due to waves or the like based on the shift correction angle θ in addition to the inclination φ of the ship 100. It is assumed that the attitude is controlled in the tilt direction so as not to come off the frame.

The imaging unit 701 images a subject via an optical system such as a lens, and outputs image data (for example, still image or moving image data) according to the imaging result. The image pickup unit 701 may store the image data according to the image pickup result in the storage area in the image pickup apparatus 101, or may output the image data to another apparatus via the network. Further, the imaging unit 701 outputs information regarding the focal length f of the lens (optical system) at the time of imaging the subject to the control unit 703.

The drive unit 702 receives an instruction regarding the target coordinates and the target angle of view from the control unit 703, directs the image pickup unit 701 to the target coordinates, and controls so that the angle of view of the lens of the image pickup unit 701 becomes the target angle of view. .. The drive unit 702 can be realized by a drive device such as an actuator for realizing such an operation or a control device (for example, an IC or the like) that controls the operation of the drive device.

The control unit 703 acquires information from the sway measurement unit 705 according to the measurement result of the inclination φ of the ship 100. The control unit 703 acquires information from the installation position setting unit 706 according to the calculation result of the distance H between the center of gravity of the ship 100 and the image pickup device 101 according to the installation position of the image pickup device 101. The control unit 703 acquires information from the distance estimation unit 704 according to the estimation result of the distance l between the subject and the image pickup device 101 (imaging unit 701). The control unit 703 acquires information regarding the focal length f of the lens of the image pickup unit 701 from the image pickup unit 701.
The control unit 703 acquires the above-mentioned information on the inclination φ, the distance H, the distance l, and the focal length f from the shaking measurement unit 705, the installation position setting unit 706, the distance estimation unit 704, and the imaging unit 701, respectively. The process to be performed corresponds to an example of "acquisition process".
Then, the control unit 703 determines the tilt drive angle φ1 related to the control of the posture of the image pickup apparatus 101 along the tilt direction based on the inclination φ, the distance H, the distance l, and the focal length f. The tilt drive angle φ1 is expressed by a conditional expression shown below as an equation (1-14) based on, for example, the inclination φ of the ship 100 and the shift correction angle θ.

The shift correction angle θ is calculated based on the equation (1-13). Further, from the equations (1-9) and (1-10), when Δt2 = fH (1-cosφ) / l is equal to or less than a predetermined threshold value, the influence of the shift movement is limited, so that the shift correction is performed. The angle θ = 0 may be set. Here, the threshold value is determined based on the sensor size as shown in the equation (1-11), and may be, for example, 10 ^ (-2).

The distance estimation unit 704 estimates the distance l between the subject imaged by the image pickup unit 701 and the image pickup unit 701 (imaging device 101), and outputs information according to the estimation result of the distance l to the control unit 703. do. The method is not particularly limited as long as the distance l can be estimated.
As a specific example, the output of the radar provided on the ship (for example, the reception result of the reflected wave of a predetermined signal transmitted to the target) may be used for estimating the distance l. If the subject is a ship, the output from the AIS (Automatic Identification System) receiver provided on the ship may be used to estimate the distance l. Further, when a Distress signal is received by VHF (Very High Frequency) / HF (High Frequency), DSC (Digital Selective Calling) information may be used for estimating the distance l. Further, by analyzing the image according to the image pickup result by the image pickup unit 701, the result of the analysis may be used for estimating the distance l. Specifically, an object whose size is known or can be estimated from the subjects captured in the image may be specified, and the distance l may be estimated based on the image height and the focal length f of the object.

The sway measuring unit 705 measures the inclination φ (in other words, the angular sway of the ship 100) of the ship 100 by using a sensor that can detect a change in posture such as a gyro sensor, and obtains the result of the measurement. The corresponding information is output to the control unit 703.

The installation position setting unit 706 calculates the distance H between the center of gravity of an object supporting the image pickup device 101 such as a ship 100 and the image pickup device 101, and controls information according to the calculation result. Output to 703.
As a specific example, the installation position setting unit 706 may receive information regarding the altitude of the position where the image pickup apparatus 101 is installed (supported position) as input, and calculate the distance H based on the altitude. In this case, the method of acquiring information regarding the altitude of the position where the imaging device 101 is installed is not particularly limited. For example, the information regarding the altitude of the position where the image pickup device 101 is installed may be information based on an instruction from the user via a predetermined input device, or the altitude by an altimeter or a barometer provided in the image pickup device 101. It may be information based on the measurement result of or atmospheric pressure. Further, the installation position setting unit 706 may generate information regarding the distance between the center of gravity of a general ship and the waterline in advance and use the information for calculating the distance H.
The information according to the calculation result of the distance H described above is the "position information" according to the position of the image pickup device 101 (imaging unit 701) supported by the housing of an object such as the ship 100 with respect to the housing. Corresponds to one example.

As described above, as the configuration example 1, an example of the functional configuration of the image pickup apparatus according to the present embodiment has been described with reference to FIG. 7.

<Structure example 2>
As a configuration example 2, another example of the functional configuration of the image pickup apparatus according to the present embodiment will be described with reference to FIGS. 8 and 9. In the following description, when the imaging device according to the configuration example 2 showing an example of the configuration in FIG. 8 is explicitly distinguished from the imaging device 101 according to the configuration example 1 showing an example of the configuration in FIG. 7. Is also referred to as "imaging apparatus 200" for convenience.

First, FIG. 8 will be described. The image pickup device 200 is different from the image pickup device 101 shown in FIG. 7 in that it includes an image processing unit 207, and is substantially the same as the image pickup device 101 in other components. Therefore, in the following, an example of the configuration of the image pickup apparatus 200 will be described focusing on a portion different from the imaging apparatus 101 shown in FIG. 7, and detailed description of the other portions will be omitted.

The image processing unit 207 acquires an image (for example, a still image or a moving image) according to the imaging result from the imaging unit 701, performs a predetermined correction process on the image, and then outputs the image to a predetermined output destination. do.

As described with reference to FIG. 3, the image pickup device 101 according to the configuration example 1 described above has the image pickup device 101 by an angle φ1 based on the equation (1-14) when the ship 100 is tilted by an angle φ. By controlling the posture in the tilt direction, the influence of the shaking of the ship 100 is reduced. In this case, since the inclination of the ship 100 is canceled out, the deviation from the state in which the subject and the image sensor of the image pickup unit 701 are facing each other becomes the shift correction angle θ described above. In this case, since the subject and the image sensor of the image sensor 701 are not facing each other, the image of the subject imaged on the image sensor is distorted so that a square object is captured as a trapezoid. Will occur. The image processing unit 207 corrects the distortion of the subject image that appears in the image.

As a method of correcting the distortion of the subject image that appears in the image, for example, a method using geometric transformation may be applied. Here, an example of geometric transformation will be described with reference to FIG. FIG. 9 represents a state in which the sensor surface of the image sensor included in the image pickup device is tilted with respect to the image pickup surface of the subject as shown in FIG. 5 based on a new coordinate system in which the optical axis is a new x-axis. It is a figure. The angle φ shown in FIG. 5 is a shift correction angle θ based on the equation (1-13), and it is assumed that the imaging device images the subject from below by the shift correction angle θ.

In the example shown in FIG. 9, 901 schematically shows the imaging surface of the subject. In the coordinate system newly set above, the image pickup surface 901 of the subject is tilted by an angle θ with respect to the optical axis of the image pickup apparatus. 902 schematically shows a line orthogonal to the imaging surface 901 and passing through the center of the lens 903 of the imaging apparatus.
Assuming that the intersection of the imaging surface 901 and the line 902 is P and the center of the lens 903 is O, the length of the line segment PO is l. Further, one end point of the imaging surface 901 is R, the intersection of the imaging surface 901 and the x-axis is Q, and the intersection of the line perpendicular to the x-axis from the end point R and the x-axis is defined as the intersection. Let it be Q'. Assuming that the length of the line segment RP is t1, the length of the line segment Q'O is represented by t1sinθ + lcosθ, and the length of the line segment RQ'is represented by t1cosθ-lsinθ. In this case, the image height of the line segment RQ is t2. At this time, since the line segment RQ is projected onto the line segment RQ', the image on the sensor surface of the image sensor is reduced by the ratio α represented by the conditional expression shown in the following equation (2-1). NS.

Therefore, as a simpler correction method, assuming that the distance between the point to be converted and the center is r, there is a method of converting the pixel value of the distance r from the center to the pixel value of the distance r × α from the center. Be done. Here, the ratio α is a value determined by the equation (2-1). Further, the pixel value of a pixel having no conversion source may be converted by using a bicubic interpolation method or the like. Further, the correction based on the above ratio α may be applied to a part of the regions. For example, a correction based on the ratio α may be applied to a region where the distance r exceeds a predetermined threshold value.

Further, when the image height t2 in a tilted system is obtained in order to approximate the conversion more strictly, for example, the image height t2 is expressed by the conditional expression shown in the following equation (2-2). ..

Here, assuming that the conditional expression shown in the following equation (2-3) is satisfied as a condition that the shift correction angle θ is small and the distance l is not too long as compared with the image height t2, the image height t2 is expressed by the equation (2). -2) Only the first term on the right side.

Then, when the ratio α is obtained so as to satisfy the conditional expression shown by the following equation (2-4), the image height is converted from t2 to t1 by the ratio α. The ratio α is represented by the conditional expression shown below as the equation (2-5).

That is, assuming that the distance between the point to be converted and the center of the lens is r, it is possible to convert the pixel value of the distance r from the center as the pixel value of the distance r × α from the center. Become. Here, the ratio α is a value calculated with the image height t2 as the distance r in the equation (2-5). Further, the pixel value of a pixel having no conversion source may be converted by using a bicubic interpolation method or the like.

By applying the image processing unit 207 that executes the above processing, the posture of the image pickup apparatus 101 is controlled in the tilt direction in order to reduce the possibility that the subject is framed out due to the shaking of the ship 100, which is apparent in the image. It is possible to reduce the distortion of the subject.

<Supplement>
The functional configuration described with reference to FIGS. 7 and 8 is merely an example, and does not necessarily limit the application destination of the technology according to the present embodiment. For example, it corresponds to a portion related to image imaging such as an imaging unit 701, a portion related to posture control of an imaging device (imaging unit 701) such as a driving unit 702, and a control unit 703 that controls these operations. It may be realized by devices that are different from each other. In this case, the device provided with the portion corresponding to the control unit 703 corresponds to an example of the “information processing device”.

Here, with reference to FIG. 10, an example of a hardware configuration for realizing a device (information processing device) including a portion corresponding to the control unit 703 will be described. In the following description, a device including a portion corresponding to the control unit 703 will also be referred to as an "information processing device 1000" for convenience.

As shown in FIG. 10, the information processing apparatus 1000 includes a CPU (Central Processing Unit) 1011, a ROM (Read Only Memory) 1012, and a RAM (Random Access Memory) 1013. Further, the information processing device 1000 includes an auxiliary storage device 1014, an output device 1015, an input device 1016, and a communication I / F 1017. The CPU 1011, ROM 1012, RAM 1013, auxiliary storage device 1014, output device 1015, input device 1016, and communication I / F 1017 are connected to each other via a bus 1018.

The CPU 1011 is a central processing unit that controls various operations of the information processing device 1000. For example, the CPU 1011 may control the operation of the entire information processing apparatus 1000. The ROM 1012 stores a control program, a boot program, and the like that can be executed by the CPU 1011. The RAM 1013 is the main storage memory of the CPU 1011 and is used as a work area or a temporary storage area for developing various programs.

The auxiliary storage device 1014 stores various data and various programs. The auxiliary storage device 1014 is realized by a storage device that can temporarily or continuously store various data such as an HDD (Hard Disk Drive) and a non-volatile memory represented by an SSD (Solid State Drive). ..

The output device 1015 is a device that outputs various information, and is used for presenting various information to the user. In the present embodiment, the output device 1015 is realized by a display device such as a display. The output device 1015 presents information to the user by displaying various display information. However, as another example, the output device 1015 may be realized by an acoustic output device that outputs sounds such as voice and electronic sounds. In this case, the output device 1015 presents information to the user by outputting sounds such as voice and telegraph. Further, the device applied as the output device 1015 may be appropriately changed depending on the medium used for presenting information to the user.

The input device 1016 is used for receiving various instructions from the user. In the present embodiment, the input device 1016 includes an input device such as a mouse, a keyboard, and a touch panel. However, as another example, the input device 1016 may include a sound collecting device such as a microphone and collect sound spoken by the user. In this case, the collected voice is subjected to various analysis processes such as acoustic analysis and natural language processing, so that the content indicated by the voice is recognized as an instruction from the user. Further, the device applied as the input device 1016 may be appropriately changed depending on the method of recognizing the instruction from the user. Further, a plurality of types of devices may be applied as the input device 1016.

The communication I / F 1017 is used for communication with an external device via a network. The device applied as the communication I / F 1017 may be appropriately changed according to the type of the communication path and the applicable communication method.

The CPU 1011 expands the program stored in the ROM 1012 or the auxiliary storage device 1014 into the RAM 1013, and executes this program to execute the functional configuration of the image pickup device shown in FIGS. 7 and 8 (particularly, the image pickup unit 701 and the drive unit 702). In addition, the processing of the imaging apparatus shown in FIGS. 7 and 8 (particularly, the processing of the control unit 703) is realized with the execution of the program.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or recording medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Further, the program code itself installed in the computer in order to realize the processing according to the present embodiment corresponds to an example of the embodiment of the present invention. Further, based on the instruction included in the program read by the computer, the OS (Operating System) or the like running on the computer performs a part or all of the actual processing, and the processing also performs the function of the above-described embodiment. May be realized.

Further, in the above-described embodiment, it is assumed that the image pickup device is mainly installed on the ship, and the influence of the blurring that becomes apparent in the image according to the image pickup result by the image pickup device due to the shaking of the ship is reduced. An example of the case has been described. On the other hand, in an environment where shaking can occur constantly, if the situation is such that the blur that becomes apparent in the image according to the imaging result due to the shaking is corrected, the application of the technique according to this embodiment is particularly applicable. Not limited.

101 Imaging device 701 Imaging unit 702 Drive unit 703 Control unit 704 Distance estimation unit 705 Measuring unit 706 Installation position setting unit

Claims (12)

Includes position information according to the position of the imaging unit supported by the housing with respect to the housing, an estimation result of the distance between the imaging unit and the subject, and at least angular fluctuation of the housing in a predetermined rotation direction. The measurement result of the amount of shaking of the housing, the acquisition means for acquiring, and
Based on the position information, the estimation result of the distance, and the measurement result of the amount of shaking, the change in the direction of the optical axis of the imaging unit according to the shaking of the housing is corrected. A control means for controlling the posture of the image pickup unit and
Information processing device. In the control means, the measurement result of the amount of shaking is φ, and the shift correction angle according to the deviation in the direction perpendicular to the optical axis of the imaging unit caused by the shaking of the housing is θ. The information processing apparatus according to claim 1, wherein the information processing apparatus controls the posture of the imaging unit along the rotation direction based on the angle φ1 represented by the conditional expression shown below.

Let H be the distance between the image pickup unit and the center of shaking according to the position information, and consider the distance between the image pickup unit and the subject according to the estimation result of the distance as a correction target. The information processing apparatus according to claim 2, wherein the shift correction angle θ is calculated based on the conditional expression shown below when the distance in the direction perpendicular to the translational component of the shaking is l.

The distance l is estimated based on the measurement result of the distance to the subject by a radar that transmits a predetermined signal toward the target and measures the distance to the target based on the reflected wave, according to claim 3. Information processing equipment. The housing is a housing of a ship.
The distance l is estimated based on the information acquired by the AIS (Automatic Identification System) of the ship.
The information processing device according to claim 3. The housing is a housing of a ship.
The distance l is estimated based on DSC (Digital Selective Calling).
The information processing device according to claim 3. The focal length is f, the distance between the imaging unit according to the position information and the center of shaking is H, the measurement result of the amount of shaking is φ, and the imaging unit according to the estimation result of the distance. Of the distance between the subject and the subject, when the distance in the direction perpendicular to the translational component of the shaking to be considered as the correction target is l, the shaking of the housing represented by the following conditional expression is obtained. The information processing apparatus according to any one of claims 2 to 6, wherein the shift correction angle θ is set to 0 when the amount of change Δt2 in the image height of the imaging unit is equal to or less than a threshold value.

The information processing apparatus according to claim 7, wherein the threshold value is determined according to the size of an image pickup device included in the image pickup unit. 2. The information processing apparatus according to any one of 8 to 8. The control means controls the posture of the image pickup unit by controlling the operation of the drive unit that changes the posture of the image pickup unit so that the direction of the optical axis changes along the rotation direction. The information processing apparatus according to any one of 1 to 9. It is an information processing method executed by an information processing device.
Position information according to the position of the imaging unit supported by the housing with respect to the housing, the estimation result of the distance between the imaging unit and the subject, and at least the angular fluctuation of the housing in the predetermined rotation direction. The measurement result of the amount of shaking of the housing including, and the acquisition step to acquire
Based on the position information, the estimation result of the distance, and the measurement result of the amount of shaking, the change in the direction of the optical axis of the imaging unit according to the shaking of the housing is corrected. A control step that controls the posture of the image pickup unit,
Information processing methods, including. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 10.

Images