WO2018066705A1

WO2018066705A1 - Smartphone

Info

Publication number: WO2018066705A1
Application number: PCT/JP2017/036536
Authority: WO
Inventors: 神山好則
Original assignee: 神山好則
Priority date: 2016-10-09
Filing date: 2017-10-06
Publication date: 2018-04-12
Also published as: JPWO2018066705A1; JP2018061220A; JP6142060B1

Abstract

Provided is a smartphone of which a body can be moved to operate an imaging device to position an imaging frame, and which can prevent a "handshake" state. The smartphone 1 is provided with: an imaging device 2 which captures an image of an object to be captured; a sensor 5 which measures the position of the imaging device 2 with respect to three axial directions thereof or a quantity related to the position; and a control device 6 which controls the operation of the imaging device 2 using at least one of three sensor values that are output from the sensor 5 and respectively corresponding to the three axial directions. The sensor 5 measures the position of the imaging device 2, which captures an image of the object to be captured, with respect to the three axial directions thereof or the quantity related to the position, and the control device 6 controls the operation of the imaging device 2 using at least one of the three sensor values that are output from the sensor 5 and respectively corresponding to the three axial directions. Thus, a user is enabled to control the operation of the imaging device 2 mounted in the smartphone by moving the smartphone body.

Description

smartphone

The present invention relates to a smartphone that controls the operation of an imaging apparatus.

An imaging device is mounted on a smartphone, and an object to be imaged is imaged using it on a daily basis. At the time of imaging, the user supports and fixes the smartphone with one or both hands and presses the imaging button. Here, the imaging button is provided on the surface on the opposite side of the smartphone with respect to the imaging direction of the imaging device. The user can take an image by operating the user's own side of the smartphone while looking at the object to be imaged in front.

In the so-called “self-portrait” in which a user who operates a smartphone takes an image of himself / herself using the smartphone, the imaging button is not suitable for the smartphone user. For this reason, it is not easy to press the imaging button, and there is a high possibility that other problems will occur, such as a so-called “camera shake” state in which the smartphone moves and the imaging frame is misaligned. Note that such a problem caused by the operation of the imaging button may also occur when the self-taking is not performed.

Therefore, for example, Patent Document 1 discloses a system for carrying a controller independent of a smartphone and operating the smartphone wirelessly using the controller. However, it is not widely accepted to carry a controller with a smartphone. Rather, it is effective for the smartphone to have means other than the operation of the image capture button as means for causing the smartphone to perform imaging.

As a smartphone operation by means other than the button operation, for example, Non-Patent Document 1 discloses a means for starting an application by an operation of shaking a smartphone. However, for imaging, it is necessary to satisfy requirements such as positioning of an imaging frame, prevention of a so-called “camera shake” state, and it is not sufficient to simply perform imaging by an operation of shaking a smartphone. Moreover, although some recent smartphones have a function of taking an image with a volume button or the like, many users do not have it.

As described above, means for using the smartphone image pickup device by means other than the operation of the image pickup button has not been widely known.

Smartphones are developing technology to make the housing thinner while the size of the terminal is increasing due to the increasing trend of LCD screens. As a result, the on-board camera (smart phone camera) is a camera with a small light receiving element, a single focal length lens with a short focal length with respect to the angle of view and a fixed focal length is adopted, and a camera lens of more than 5 mm is attached to the smartphone. Is not often adopted.

JP2011-237716A JP 2016-63452 A JP 2016-72673 A JP 2016-122902 A International Publication No. 2013/084428 International Publication No. 2015/040809 International Publication No. 2016/133158

It is an object of the present invention to provide a smartphone that operates an imaging apparatus by moving a main body and realizes positioning of an imaging frame and prevention of a so-called “camera shake” state.

Generally, a lens employed in a smartphone camera has a very small aperture value (F value) (bright lens) and a short focal length due to the design of the housing shape. The focus is determined by the aperture value (F value), the focal length of the lens, and the shooting distance (distance between the subject and the camera). Therefore, the photos and videos you take have a deep depth of field, making it easier to obtain a pan-focus effect that looks like the subject is in focus from near to far, and clearly shows the area in the image clearly. It is possible. An object of the present invention is to provide a function of searching for a local image in an image using this structural advantage.

The smartphone of the present invention
It is a smartphone that controls the operation of the imaging device mounted on it by moving the smartphone body,
An imaging device for imaging an object to be imaged;
A sensor for measuring speed, acceleration, angular velocity or angular acceleration in the three-axis directions of the imaging device;
A control device that controls the operation of the imaging device using at least one of three sensor values corresponding to each of the three axial directions output from the sensor;
With
The control device includes:
The three sensor values are periodically received from the sensor, and when the change amount of at least one of the three sensor values exceeds a threshold value, the time point at which the threshold value is exceeded is determined as a time change pattern. As a start time, it is determined whether the time change patterns of the three sensor values correspond to a predetermined time change pattern,
After the predetermined time change pattern, it is determined whether or not the three sensor values are substantially constant with respect to time, and when the three sensor values are substantially constant, the image of the imaging device at the start time and the start time Compared with the image of the imaging device at the time when only T ′ has elapsed from the time, the imaging device is operated only at the time when the difference between both images is smaller than a threshold,
The value of T ′ is 1.5 to 2.5 seconds,
The predetermined time change pattern is a time change pattern generated by rotating the smartphone in one direction about any one of the three axes and then rotating the smartphone in the opposite direction to return to the position before the rotation. It is.

It becomes possible for the user to control the operation of the imaging device mounted on the smartphone by moving the smartphone main body so that the time change patterns of the three sensor values become predetermined. The predetermined time change pattern is a time change pattern generated by rotating the smartphone in one direction about any one of the three axes and then rotating the smartphone in the opposite direction to return to the position before the rotation. And The user positions the smartphone in a posture to image the object to be imaged, rotates it in one direction about any one of the three axes from that state, and then rotates in the opposite direction to return to the position before the rotation ( The imaging frame can be positioned), stopped (so-called “hand-shake” is eliminated), and the operation of the imaging apparatus can be controlled after taking the posture for imaging the imaged object again.

The smartphone of the present invention
The control device causes the user to rotate the smartphone in one direction about any one of the three axes, and then rotate the smartphone in the opposite direction to return to the position before the rotation. The three time sensor patterns are received to determine the predetermined time change pattern.

According to this feature, a predetermined time change pattern used for operating the imaging apparatus can be set uniquely for each user.

The smartphone of the present invention
The control device calculates a difference between the at least one sensor value and the predetermined time change pattern, and compares the sum of the difference with a threshold value.

When the accumulated difference is smaller than the threshold value, it can be determined that the time change pattern of at least one sensor value corresponds to a predetermined time change pattern, and a reliable determination can be made.

The smartphone of the present invention
The sensor is an acceleration sensor that measures acceleration in the three-axis directions of the imaging device,
When the two sensor values of the three sensor values change with respect to time and the remaining one sensor value is substantially constant with respect to time, the control device determines the time of the two sensor values. Determine the change pattern.

Acceleration sensors mounted on many smartphones allow the control device to control the operation of the imaging device. As for the acceleration sensor, as described later, two of the three sensor values change with time, and the remaining one sensor value is substantially constant with respect to time.

The smartphone of the present invention
The time change to a predetermined pattern, the acceleration value is a sensor value, (1) increased by -9.8m / ^{s 2} from X1, the flow returns to -9.8m / ^{s 2} within the time T changes ,
(2) increases or decreases from 0 m / ^{s 2} only X2, returns to 0 m / ^{s 2} within time T changes,
(3) increased from 9.8 m / ^{s 2} only X1, and any of a change pattern back to 9.8 m / ^{s 2} within the time T,
The value of X1 is 4.9 to 14.7 m / s ² ,
The value of X2 is 8.5 to 9.8 m / s ² ,
The value of T is 0.8 to 1.8 seconds.

It is possible to control the operation of the imaging device mounted on the smartphone by rotating the smartphone back around 60-120 degrees around one of the three axes within 0.8-1.8 seconds Become. When the smartphone is moved over a period of 0.8 to 1.8 seconds, the acceleration of the movement is not so large, and the time changes as the time changes in the direction of gravitational acceleration (9.8 m / s ² ) with respect to the three axes. Patterns can be defined. Note that 4.9 m / s ² ≈9.8 m / s ² × (1-cos 60 °), 14.7 m / s ² ≈9.8 m / s ² × (1-cos 120 °), 8.5 m / s ² ≈9.8 m / s ² × sin 60 °, and a time change pattern is realized by rotation of 60 to 120 degrees.

The smartphone of the present invention
The sensor is an angular velocity sensor that measures a rotational angular velocity related to three axes of the imaging device,
When the sensor value of one of the three sensor values changes with respect to time and the remaining two sensor values are substantially constant with respect to time, the control device determines the time of the one sensor value. Determine the change pattern.

The operation of the imaging device can be controlled by the control device using the angular velocity sensor mounted on the smartphone. As for the angular velocity sensor, as described later, one of the three sensor values changes with time, and the remaining two sensor values are substantially constant with respect to time.

The smartphone of the present invention
In the predetermined time change pattern, the value of the angular velocity that is the sensor value increases or decreases from 0 (1) and returns to 0, and the absolute value of the integral of the sensor value is Y radians, (2) A change pattern in which a decrease or increase from 0 returns to 0, and the change in which the absolute value of the integral of the sensor value is Y radians continues during time T;
The value of Y is π / 3 to 2π / 3 radians,
The value of T is 0.8 to 1.8 seconds.

It is possible to control the operation of the imaging device mounted on the smartphone by rotating the smartphone back around 60-120 degrees around one of the three axes within 0.8-1.8 seconds Become.

The smartphone of the present invention
When the change amount of at least one of the three sensor values exceeds a threshold value, the control device sets the time change pattern of the three sensor values as a start time point of the time change pattern when the threshold value is exceeded. to decide.

The user can grasp the starting point of the rotation operation.

The smartphone of the present invention
The control device operates the imaging device when T ′ has elapsed from the start time, and the value of T ′ is 1.5 to 2.5 seconds.

Operates the imaging device only when the user rotates in one direction and then rotates in the opposite direction and returns to the pre-rotation position in approximately 2 seconds (1.5 to 2.5 seconds). Can be made. The user can operate for approximately 2 seconds and prevent malfunction of the imaging apparatus when the user does not perform the operation.

The smartphone of the present invention
The control device compares the image of the imaging device at the start time with the image of the imaging device at the time when T ′ has elapsed from the start time, and only when the difference between the two images is smaller than a threshold value To work.

When the user has positioned the imaging frame at the start time, the imaging device can be operated when the smartphone is returned to that position.

The smartphone of the present invention
The control device compares the difference between the two images based on the difference in the histogram of the RGB luminance values of the two images.

* Comparison of the image of the image pickup device at the start time and the image pickup time can be performed with easy calculation.

The above-described smartphone of the present invention can be further provided with the functions described below. FIG. 9 explains the flow of each process executed by the smartphone of the present invention having the functions described below.

In the smartphone of the present invention,
The control device further includes a sensor that measures a speed, an acceleration, an angular velocity, or an angular acceleration related to the three axial directions of the imaging device, and three sensor values corresponding to each of the three axial directions output from the sensor. Electronic shutter when periodically receiving three sensor values for controlling the operation of the imaging apparatus using at least one and a change amount of at least one of the three sensor values exceeds a threshold value At the time (start time) when does not start up, a plurality of images that can be acquired continuously (for example, a plurality of images such as 30 to 60 frames) are cached in the memory array.

This is to cache a plurality of images that can be acquired continuously in the memory array when the electronic shutter does not start.

Even when the shutter is turned off, multiple images that can be acquired continuously can be cached in the memory array. It has a so-called continuous shooting function.

That is, in the smartphone of the present invention,
The control device further includes a sensor that measures a speed, an acceleration, an angular velocity, or an angular acceleration related to the three axial directions of the imaging device, and three sensor values corresponding to each of the three axial directions output from the sensor. Electronic shutter when periodically receiving three sensor values for controlling the operation of the imaging apparatus using at least one and a change amount of at least one of the three sensor values exceeds a threshold value A plurality of images that can be continuously acquired (for example, a plurality of images such as 30 to 60 frames) are cached in the memory array.

In these cases, the control device divides each component of the RGB color histogram of all cached frame images into 256 gradation values of 0-255 into four equal parts and replaces them with the central representative value to reduce the color to 64 colors. And using the color features of the 64-dimensional vector to detect and detect feature quantities such as brightness difference, luminance distribution, luminance gradient direction distribution, etc. of local image objects for each pixel, exceeding a predetermined threshold Then, the pixels whose arrays are continuously approximated are grouped as similar objects, and object recognition of local images existing in all frame images is determined.

Further, in the above case, the control device deletes the hue information and the saturation information while maintaining the luminance of the RGB value, converts it to a gray scale, binarizes it, and calculates the outline of each pixel. The extracted pixels are grouped as similar objects, and the recognition of local images existing in all the frame images is determined by grouping pixels having an approximate value that continuously exceeds a predetermined threshold.

In the above case, the control device configures an overlay layer file having an xy coordinate axis at the center point of pixels grouped as a similar object, and draws and displays a focus point representing an object area of a local image, In all the frame images, the position recognition processing of the object of the local image is performed. Further, the position where the object recognition of the local image is discriminated is tracked in all the frame images.

In order for the control device to execute the operation of the control device described above, a data set to be used for object recognition of an image is loaded into a memory in advance on an application installed in the smartphone, and calculation processing is performed to determine a focus point. Display the image as a line drawing, or perform calculation processing with a data set used for object recognition from an external application so that the control device executes an operation for drawing, displaying, and tracking the focus point in all frame images. You can also.

In the above, the control device determines how many frames ago the image before tilting is based on the initial motion speed in the tilting direction (tilting), fetches it to another memory, and simultaneously holds the angle and orientation at that time as well. Perform the action to be performed. At this time, the frame image is always cached in the memory array.

In addition, in the above, the control device performs an approximate value + speed (identifies a stationary state) of the angle and direction held at the initial motion speed that swings (tilts) in the reverse direction. The frame image at the position is taken out and the shutter is operated, and shooting of the moving image is started.

Also, in the above, the control device automatically converts the xy coordinate position information and local image information of the focus points of all the frames representing the region into various kinds such as an XML language when T ′ has elapsed from the start time. It can be described in a layer file in a data format, and can be described and stored as meta tags or binary data in an image, or called by a server on the cloud stored via a network.

Also, local images can be displayed by discarding non-local image information.

Furthermore, the user can manually describe local image information such as photographing information and GPS position information and photographing information in the image file inside the image file or an overlaid layer file.
Also, all the described information can be stored on the above-described application and used as a new image data set, or feature quantities such as an object shown in local image information and description information can be called from an external application.

In the determination of the local image, there are innumerable means other than the contour line distribution, and any procedure may be used.
The smartphone of another embodiment of the present invention is
A smartphone that controls the operation of an imaging device mounted on the smartphone body by moving the smartphone body, the imaging device imaging an object to be imaged;
A sensor for measuring speed, acceleration, angular velocity or angular acceleration in the three-axis directions of the imaging device;
A control device that controls the operation of the imaging device using at least one of three sensor values corresponding to each of the three axial directions output from the sensor;
With
The control device has a photo shooting mode and a video shooting mode,
The three sensor values are periodically received from the sensor, and when the change amount of at least one of the three sensor values exceeds a threshold value, the time point at which the threshold value is exceeded is determined as a time change pattern. Threshold value for determining the sensor value of the angle and / or direction at the same time as the start time, judging from the sensor value at the start time, how many frames before the image is tilted, and taking out the image to another memory To determine whether the time change pattern of the three sensor values corresponds to a predetermined time change pattern,
After the predetermined time change pattern, it is determined whether or not the three sensor values are substantially constant with respect to time, and when the three sensor values are substantially constant, the image of the imaging device at the start time and the start time From the sensor value at the time when the image of the imaging device is still at the time when T ′ has elapsed from that time, it is determined how many frames ago the image at the time of still is, and the image is taken out to another memory, and at that time The sensor value of the angle and / or direction is compared with a predetermined threshold value and held together as a stationary sensor value at the end, the stationary sensor value at the start and the stationary sensor value at the end are compared, and the sensor value of both images Only when the difference is smaller than a threshold value, the imaging device is operated at the time when the time has elapsed, and the shutter operation is started in the photo shooting mode, or in the video shooting mode Start shooting.

In other embodiments,
The control device has a photo shooting mode and a video shooting mode,
The three sensor values are periodically received from the sensor, and when the change amount of at least one of the three sensor values exceeds a threshold value, the time point at which the threshold value is exceeded is determined as a time change pattern. It is determined whether the time change pattern of the three sensor values corresponds to a predetermined time change pattern as a start time point,
After the predetermined time change pattern, it is determined whether or not the three sensor values are substantially constant with respect to time, and when the three sensor values are substantially constant, the image of the imaging device at the start time and the start time Compared with the image of the imaging device at the time when T ′ has elapsed from the time, the imaging device is operated only when the difference between the two images is smaller than the threshold, and the shutter operation is performed in the photography mode. Start shooting or start shooting in the moving image shooting mode.

The equipment control device takes out a frame image at the position based on the approximate value and speed of the angle and direction held at the speed of the initial motion that swings in the tilt direction by swinging in the reverse direction, and starts or ends the operation of the imaging device. It is something to be made.

Also, the equipment control device performs playback, stop, pause, rewind / fast forward, frame advance, frame rewind, and end operations for the moving image captured by the imaging device.

The value of T ′ is 1.5 to 2.5 seconds, and the predetermined time change pattern is obtained by rotating the smartphone in one direction about any one of the three axes, It is a time change pattern which arises by rotating in the reverse direction and returning to the position before rotation.

According to the smartphone of the present invention, it is possible to operate the imaging device by moving the main body, and to realize the positioning of the imaging frame and the prevention of the so-called “camera shake” state.

In addition, according to the smartphone of the present invention, a lens generally employed in a smartphone camera has a very small aperture value (F value) (bright lens) and a short focal length due to the design of the housing shape. , The photos and videos you take will have a deep depth of field, making it easier to obtain a pan-focus effect that appears to be in focus from close to far, and clearly and clearly showing the local area in the image. The ability to search for local images within an image can be provided by taking advantage of the structural advantage of

It is a block diagram which shows the structure of one Embodiment of a smart phone. It is a figure explaining an example of operation which operates an imaging device. It is a figure explaining the motion of the smart phone which concerns on the operation | movement of FIG. It is a figure explaining another example of operation which operates an imaging device. It is a figure explaining the motion of the smart phone which concerns on the operation | movement of FIG. It is a figure which shows the time change pattern of the sensor value which an acceleration sensor outputs in the operation | movement of FIG. It is a figure which shows the time change pattern of the sensor value which an acceleration sensor outputs in the operation | movement of FIG. It is a figure which shows the time change pattern of the sensor value which an angular velocity sensor outputs in the operation | movement of FIG. It is a flowchart explaining an example of the function of the smart phone of this invention. The figure explaining an example of imaging | photography operation | movement with the smart phone of this invention, and is a figure explaining the operation | movement which tilts (an enclosure: sideways / heaven: left). Even in the case of the outer camera and the inner camera, the image operation for tilting does not change regardless of how the camera is held vertically or horizontally, and FIG. 10 illustrates the case of the outer camera. It is a figure showing an example of reproduction | regeneration operation of the moving image by the smart phone of this invention. Although viewing is possible only on the liquid crystal screen side (inside), vertical and horizontal viewing is possible, and playback operations can be performed in both vertical and horizontal directions. FIG. 11 illustrates the playback operation in the case: horizontal orientation / top: left. . It is a figure which shows the relationship of the time change pattern of the sensor value which an acceleration sensor outputs, when the imaging | photography operation | movement by the smart phone of this invention (tilting (case: sideways / top: left)). It is a figure which shows the relationship of the imaging | photography operation | movement (tilting (case: length / top: top)) ： with the smart phone of this invention, and the time change pattern of the sensor value which an acceleration sensor outputs. It is a figure which shows the relationship of the time change pattern of the sensor value which an imaging | photography operation | movement with the smart phone of this invention (tilting (case: sideways / celestial: left)) outputs from an angular velocity sensor (gyro sensor). It is another figure which shows the relationship of the time change pattern of the sensor value which an image | photographing operation | movement (tilting (case: length / top: top)) and the angular velocity sensor (gyro sensor) of this invention output.

Hereinafter, an embodiment of the present invention will be described.

FIG. 1 shows a configuration of the smartphone 1 according to the present embodiment. The smartphone 1 is intended to control the operation of the imaging device mounted on the smartphone main body (simply referred to as the main body 1), and includes the imaging device 2, the display device 3, and the storage device 4. , Sensor 5 and control device 6. Note that these devices are mounted in a casing-like main body 1 (see, for example, FIG. 2) having a rectangular surface that extends in the direction of one axis. In addition, let the direction orthogonal to each other on one surface of the main body 1 provided with the display device 3 be the X-axis and Y-axis directions, and let the direction orthogonal to the one surface be the Z-axis direction. In imaging, the main body 1 is often supported with the display surface of the display device 3 facing the horizontal direction. In this state, the Z axis is the horizontal direction, one of the X axis and the Y axis is the vertical direction, and the other is the horizontal direction. Will turn to.

The image pickup device 2 is a device that picks up an image of an object to be picked up through a lens (not shown) provided in the main body 1. As an example, a CCD image sensor or a CMOS image sensor can be adopted. The imaging device 2 is incorporated in the main body 1. The operation of the imaging apparatus controlled in the present embodiment is to activate the electronic shutter of the imaging apparatus 2 and take a still image, that is, a photograph (also simply referred to as an image). It is possible to take a picture. It is also possible to record an image when the electronic shutter is not activated. An image picked up by the image pickup device 2 is expressed in a digital format such as a JPEG format, and transmitted to the display device 3 and the storage device 4 described later.

The display device 3 is a device that displays an image captured by the imaging device 2 via a lens (not shown) or a captured image, and is provided on the surface of the main body 1.

The storage device 4 is a device that stores an image picked up by the image pickup device 2, and can employ a memory such as a DRAM, for example. The storage device 4 is incorporated in the main body 1.

The sensor 5 is a device that is mounted in the main body 1 and measures the speed, acceleration, angular velocity, or angular acceleration in the three-axis directions of the imaging device 2. As the sensor 5, for example, an acceleration sensor that measures acceleration in the three-axis directions of the imaging device 2 can be employed. The acceleration in the three-axis direction of the imaging device 2 is measured by the acceleration sensor, and two of the three sensor values corresponding to each of the three-axis directions output from the acceleration sensor are used by the control device 6 described later. By controlling the operation of the imaging device 2, the user can control the operation of the imaging device 2 by moving the main body 1. As the sensor 5, an angular velocity sensor (also referred to as a gyro sensor) that measures a rotation angle with respect to the three axes of the imaging device 2 can be employed. The angular velocity sensor measures the rotation angle about the three axes of the imaging device 2, and the control device 6 described later uses one of the three sensor values corresponding to the rotation angle about the three axes output from the angular velocity sensor. By controlling the operation of the imaging device 2, the user can control the operation of the imaging device 2 by moving the main body 1.

The sensor 5 is not limited to an acceleration sensor and an angular velocity sensor (gyro sensor), and a velocity sensor and an angular acceleration sensor may be employed.

The control device 6 is a device that controls the operation of the imaging device 2. The control device 6 has a central processing unit (CPU), for example, and develops a function as a device that controls the operation of the imaging device 2 by executing an imaging application. The imaging application is stored in the storage device 4, for example, and can be read by the CPU. The control device 6 periodically receives three sensor values corresponding to each of the three axial directions from the sensor 5, and determines a time change pattern of at least one of the sensor values. Thereby, the operation of the imaging device 2 can be controlled based on the temporal change pattern of at least one of the three sensor values generated when the user moves the main body 1. The operation control of the imaging device 2 by the control device 6 will be described later.

In addition, a communication device (not shown) may be provided so that an image picked up by the image pickup device 2 can be transferred.

2 and 3 show an example of an operation for operating the imaging device 2 by the user and the movement of the main body 1 in this operation, respectively. As shown in FIG. 2, the user holds a short hand of the main body 1 with the right hand 8, directs the display device 3 toward himself (the back of the drawing), and supports the main body 1 sideways so that an object to be imaged, for example, a user Position itself in the imaging frame to take an image. From this state, the user (1) turns the wrist and swings the main body 1 about 90 degrees counterclockwise in FIG. Thereby, the main body 1 is supported vertically. From this state, the user (2) turns the wrist in the reverse direction and swings the main body 1 clockwise in FIG. As a result, the main body 1 is supported sideways and takes the posture for imaging the object to be imaged again. The user controls the operation of the imaging device 2 by (3) stopping the main body 1 in this state and (4) standing still. In this series of operations, as can be seen from FIG. 3, the main body 1 (1) rotates in one direction about the Z-axis passing near one short side, and then (2) rotates in the opposite direction to return to the position before rotation. Return to step (3). The same applies to the left hand instead of the right hand.

Here, the main body 1 is rotated with respect to the Z-axis passing near the short side, but the Z-axis may pass through another position of the main body 1, for example, the Z-axis passing near the long side. It is good also as rotating about. That is, if the rotation is performed in a direction parallel to the Z axis shown in the figure, the X and Y coordinates of the rotation axis are arbitrary.

4 and 5 show another example of the operation for operating the imaging device 2 by the user, and the movement of the main body 1 in this operation, respectively. As shown in FIG. 4, the user holds both the short sides of the main body 1 with both hands 9, directs the display device 3 toward himself (the back of the drawing), and supports the main body 1 sideways, whereby an object to be imaged, for example, a user Position itself in the imaging frame to take an image. From this state, the user (1) turns the wrist and tilts the main body 1 upward (or downward) by about 90 degrees. Thereby, the main body 1 is supported horizontally. From this state, the user (2) reverses the wrist and returns the main body 1. As a result, the main body 1 is supported sideways and takes the posture for imaging the object to be imaged again. The user (3) controls the operation of the imaging apparatus by holding the main body 1 stationary in this state. In this series of operations, as can be seen from FIG. 5, the main body 1 (1) tilts in one direction about the Y axis passing through the center of the main body (rotates with respect to the Y axis), and then (2) returns (Y axis To the original position) and (3) rest. Note that the same applies even if the operation is performed with one hand instead of both hands.

Note that, here, the main body 1 is tilted with respect to the Y axis passing through the center thereof, but may be tilted with respect to the X axis passing through the center of the main body 1. Further, the coordinates of the rotation axis are arbitrary.

The operation control of the imaging device 2 by the control device 6 will be described. Operation control includes determination of a temporal change pattern of sensor values and determination of imaging conditions such as stillness.

(Judgment of temporal change pattern of sensor value)
The control device 6 detects three sensor values g _{d, n} from the sensor 5 (d = 1, 2, 3: d = 1 is for the X axis, d = 2 is for the Y axis, and d = 3 is for the Z axis. Value) periodically at intervals Δt. A numerical sequence g _{d, i} (i = 1 to N) with respect to time t _i (= iΔt) can be obtained. The control device 6 determines whether or not the numerical sequence g _{d, i} corresponds to a predetermined time change pattern. Here, variations can be considered in the “predetermined time change pattern” and the procedure for determination. This will be described below.

Hereinafter, a case where the sensor 5 is an acceleration sensor will be described.

The control device 6 changes at least one change amount Δg _{d, n} (= | g _{d, n + 1} −g _d ) among the absolute values of the three sensor values g _{d, n} (d = 1, 2, 3) for each reception. _{, n |)} it is determined whether more than a threshold value, the time t _n and the beginning of the time variation pattern when the threshold is exceeded. Hereinafter, for the sake of explanation, it is assumed that the change amount of d = 1 exceeds the threshold, but the same applies to d = 2 and d = 3.

Thereafter, the control device 6 determines whether one of the change amounts Δg _{d, n} (d = 2, 3) other than the change amount exceeding the threshold value exceeds the threshold value. Hereinafter, for the sake of explanation, it is assumed that the change amount of d = 2 exceeds the threshold, but the same applies to d = 3. That is, the change amount of d = 1, 2 exceeds the threshold value, and the change amount of d = 3 does not exceed the threshold value.

Thereafter, the control device 6 determines whether the values of g _{1, i} and g _{2, i} (i = n to N) correspond to “predetermined time change patterns” or g _{3, i} (i = n to N). ) Is substantially constant. The value of g _{3, i} shall be “substantially constant” if the difference from the value of g _{3, n} (| g _{3, n} −g _{3, i} |) is less than a certain value regardless of i. Can do.

The following procedure can be considered as to whether or not d = 1, 2 corresponds to the “predetermined time change pattern”.

(Procedure 1) A time t _i is obtained when the value of Δg _{d, i} reverses in polarity (Δg _{d, i} and Δg _{d, i + 1} are different in sign) _{, and} the value of g _{d, i} with respect to _i is expressed as “extreme value ( p) ". Thereafter, it is confirmed that the value of Δg _{d, i} approaches the value of Δg _{d, n} (| g _{d, n} −g _{d, i} |). (The value of i at this time is U.) Here, if g _{d, n} ≈9.8 (m / s ² ) or g _{d, n} ≈−9.8 (m / s ² ), Since 4.9 ≦ | p−g _{d, n} | (X1) ≦ 14.7 (m / s ² ), it can correspond to a “predetermined time change pattern”. Further, if g _{d, n} ≈0 (m / s ² ), then 8.5 ≦ | p−g _{d, n} | (X2) ≦ 9.8 (m / s ² ) It is possible to correspond to the “time change pattern”. Here, since the gravitational acceleration exists only in one direction, one of g _{1, n} and g _{2, n} is close to 0 and the other is close to 9.8 or −9.8.

(Procedure 2) At the same time that the user performs an operation for operating the imaging device 2, the control device 6 receives three sensor values from the sensor 5, and pre-stores the reference pattern f _i (i = 1 to M). I ask for it. Integrated value S = Σ _{d = 1 to 3} (Σ _{i = 1 to M} | g _{d, n + i−1} −f _d) of absolute value of difference between sensor value g _{d, i} and reference pattern f _{d, i} for each reception _{, I} |). If the integrated value S is equal to or less than a predetermined threshold value, it can correspond to a “predetermined time change pattern”.

Having described Step 1 and Step 2, (1) increased by -9.8m / ^{s 2} from X1, the flow returns to -9.8m / ^{s 2} within time T changes, (2) 0 m / ^{s 2} Increase or decrease from time to time by X2, and return to 0 m / s ² within time T, (3) decrease from 9.8 m / s ² by X1 and return to 9.8 m / s ² within time T, It is only necessary to be able to determine whether or not the deviation from the reference pattern is small, and there are innumerable procedures for such determination. Either procedure may be used.

FIG. 6 shows a temporal change pattern of the sensor value output by the acceleration sensor in the operation of shaking the main body 1 shown in FIG. In the shaking operation, two sensor values of the three sensor values, that is, the sensor values for the X axis and the Y axis defined in FIG. 3 change with time, and the remaining one sensor value, that is, the definition in FIG. The sensor value regarding the Z axis to be performed is substantially constant with respect to time. That is, the Z axis maintains a direction substantially perpendicular to the direction of gravity acceleration, and the XY plane rotates including the direction of gravity acceleration. In this case, the control device 6 can control the operation of the imaging device 2 by determining the temporal change pattern of the sensor values regarding the X axis and the Y axis.

FIG. 7 shows a temporal change pattern of sensor values output by the acceleration sensor in the operation of tilting the main body 1 shown in FIG. In the tilting operation, two of the three sensor values, that is, the sensor values for the X axis and the Z axis defined in FIG. 5 change with time, and the remaining one sensor value, that is, the definition in FIG. The sensor value with respect to the Y axis is substantially constant with respect to time. The Y-axis maintains a direction substantially perpendicular to the direction of gravity acceleration, and the XZ plane rotates including the direction of gravity acceleration. In this case, the control device 6 can control the operation of the imaging device 2 by determining the temporal change pattern of the sensor values regarding the X axis and the Z axis.

As described above, both the shaking operation and the tilting operation change two sensor values with respect to time, and the remaining one sensor value is substantially constant with respect to time. Since it rotates around the axis (Y axis, Z axis) perpendicular to the direction of gravity acceleration (-X direction before the start of operation), gravity acceleration does not change in the direction of the rotation axis, and the other two axes Gravity acceleration changes in the direction. In addition, the long diameter of the smartphone moves about 20 cm over at least about 1 second, and the acceleration required for the movement is 1 m / s ² or less (strictly speaking, the sensor position on the smartphone and the rotation center) Depending on the coordinates of and possibly smaller), a change in gravitational acceleration is reliably detected. Note that the rotation around the gravity acceleration direction axis (X-axis) does not change the direction of the smartphone relative to the gravitational acceleration, and all three sensor values do not change. I can't.

Hereinafter, the case where the sensor 5 is an angular velocity sensor will be described.

The control device 6, at least one value is determined whether more than a threshold value, time variation patterns that point t _n when the threshold is exceeded for one of the absolute values of the three sensor values g _{d, n} for each reception The starting point of Hereinafter, for the sake of explanation, it is assumed that the value of d = 2 exceeds the threshold value, but the same applies to d = 1 and d = 3.

Thereafter, the control device 6 determines whether the value of g _{2, i} (i = n to N) corresponds to a “predetermined time change pattern” or g _{1, i} and g _{3, i} (i = n to N). ) To determine whether the values of g _{1, i} and g _{3, i} are substantially constant, the difference between the values of g _{1, n} and g _{3, n} (| g _{1, n} −g _{1, If i} |, | g _{3, n} −g _{3, i} |) are equal to or less than a certain value regardless of i, it can be made “almost constant”.

The following procedure can be considered as to whether or not d = 2 corresponds to the “predetermined time change pattern”.

(Procedure 1) The time point t _{i at} which the value of g _{2, i} exceeding the threshold value returns close to 0 (| g _{2, i} | is smaller than the predetermined threshold value) is determined, and the time point for i is set to u. t = n 'seek _{_{= Σ i = n ~ u (}} g 2, i), the integrated value S' t = _{g 2, i} integrated value S of up to u from the, π / 3 ≦ | S ' | ≦ 2π If it is / 3 radians, it corresponds to the first half of the “predetermined time change pattern”. Thereafter, the value of u is treated as the value of n, and the point in time when the value of g _{2, i} returns to close to 0 after exceeding the threshold value is set as u ′, and the integrated value S ″ = Σ _{i = u to u 'Find} (g _{2, i} ). If the integrated value S ″ is π / 3 ≦ | S ″ | ≦ 2π / 3 radians and the positive and negative of S ′ and S ″ are opposite, it corresponds to the “predetermined time change pattern”. Then.

(Procedure 2) At the same time when the user performs an operation for operating the imaging device 2, the control device 6 receives three sensor values from the sensor 5, and preliminarily receives the reference signal f _i (i = 1 to M). I ask for it. The sum of the absolute values of the difference between the sensor value g _{d, i} and the reference signal f _i for each reception S = Σ _{i = 1 to M} | g _{n + i−1} −f _i | If the integrated value S is equal to or less than a predetermined threshold value, it can correspond to a “predetermined time change pattern”.

The procedure 1 and the procedure 2 have been described above. Π / 3 ≦ | S ′ | ≦ 2π / 3 radians, π / 3 ≦ | S ″ | ≦ 2π / 3 radians, and S ′ and S ″ It is only necessary to be able to determine whether the sign is opposite or that the deviation from the reference pattern is small, and there are innumerable procedures for such determination. Either procedure may be used.

FIG. 8 shows a time change pattern of the sensor value output by the angular velocity sensor in the operation of tilting the main body 1 shown in FIG. In the tilting operation, one of the three sensor values, that is, the sensor value related to the Y axis defined in FIG. 5 changes with time, and the remaining two sensor values, that is, the X axis defined in FIG. And the sensor value regarding the Z-axis becomes substantially constant with respect to time. The tilting operation is rotation about the Y axis, and an angular velocity about the Y axis is generated, and an angular velocity about the other axis is not generated. In this case, the control device 6 can control the operation of the imaging device 2 by determining the temporal change pattern of the sensor value with respect to the Y axis.

In the case of a shaking motion, the rotation is about the Z axis, the sensor value related to the Z axis changes with time, and the sensor values related to the X axis and the Y axis become substantially constant with respect to time. Similarly, with respect to the rotation operation about the X axis, the sensor value related to the X axis changes with time, and the sensor value related to the Y axis and the Z axis becomes substantially constant with respect to time. The angular velocity sensor can detect a rotation operation around the X axis that cannot be detected by the acceleration sensor.

Regardless of whether the sensor 5 is an acceleration sensor or an angular velocity sensor, the operation starts from the start time (t _n ) to the end time (t _U when the sensor is an acceleration sensor, and t when the sensor is an angular velocity sensor. Confirm that the time to _{u ′} ) is 0.8 to 1.8 seconds. When the time from the start point to the end point of the operation is not 0.8 to 1.8 seconds, it does not correspond to the “predetermined time change pattern”.

When comparing with the reference pattern (procedure 2), it is not necessary to confirm the time from the start point to the end point of the operation. If the time length of the reference pattern is 0.8 to 1.8 seconds, the integrated value S inevitably increases if the time from the start time to the end time of the operation is not 0.8 to 1.8 seconds. This is because it is considered.

(Judgment of imaging conditions such as still)
The control device 6 determines the stillness after the operation is finished. The determination of stillness may be made by continuing that the value of Δg _{d, i} is not more than a constant value for all of d = 1, 2, 3 over a predetermined time (for example, 0.2 seconds).

The control device 6 operates the imaging device 2 when the stationary state is determined only when the time from when the operation starts (t _n ) until the stationary state is determined is 1.5 to 2.5 seconds. And However, the imaging device 2 may be operated when the stillness is determined regardless of the time from the start of the operation until the stillness is determined.

The control device 6 may compare the image at the start of the operation with the image when the stillness is determined, and may not operate the imaging device 2 if the difference between the two images is large. It is considered that the user positions the imaging frame (determines the position and direction of the imaging device) before starting the operation. If it is different from the positioned imaging frame, imaging is not performed and positioning displacement is prevented.

For this reason, the control device 6 operates the imaging device 2 to acquire an image at the time when the operation is started and when the operation is determined to be stationary.

Various methods are known for comparing whether the two images are similar, and any of them may be used. As an example, the following method enables high-speed processing.

Each pixel of the image data has a color value (for example, RGB value). Color values (often 24 bits) are clustered (considering close colors as the same color), eg 64 colors. For example, if only the upper 2 bits of each RGB value are viewed, there are 64 colors. Find a histogram of which of the 64 colors each pixel is throughout the image. That is, determine the total pixel appearance ratio p _c of the c-th color. Σc _{= 1 to 64} _pc = 1.

The similarity R between the two images is calculated as R = Σc _{= 1 to 64} (min (p _{1, c} , p _{2, c} )). min is the smaller of the two values. For the same image, R = 1. (Σ _{c = 1 to 64} (min (p _{1, c} , p _{2, c} )) = 1−Σ _{c = 1 to 64} (| p _{1, c} −p _{2, c} |), and the images are different. It can be seen that the value of R becomes smaller.)

When such high-speed calculation is performed, a plurality of images are processed by combining the time point when stillness is determined and the time points before and after that, and the image with the largest R is captured (stored in the smartphone). It becomes possible to do so.

As described in detail above, the smartphone 1 according to the present embodiment includes the imaging device 2 that captures an object to be captured, the position of the imaging device 2 in the three axial directions, or the sensor 5 that measures the amount related to the position, and the sensor. 5 includes a control device 6 that controls the operation of the imaging device 2 by using at least one of the three sensor values corresponding to each of the three axial directions output from 5. The sensor 5 measures the position in the three-axis direction of the imaging device 2 that images the object to be imaged or the amount related to the position, and the control device 6 3 corresponding to each of the three-axis directions output from the sensor 5. By controlling the operation of the imaging device 2 using at least one of the two sensor values, the user can control the operation of the imaging device 2 mounted on the smartphone by moving the smartphone body.

It is a smartphone that operates the imaging device by moving the main body and realizes positioning of the imaging frame and prevention of so-called “camera shake” state. The use by many individual users is conceivable, and the implementation of the present invention by a smartphone manufacturer is expected.

The control operation in the smartphone of the present invention is not limited to taking a still image, that is, a photograph (also simply referred to as an image) by activating an electronic shutter of the imaging apparatus, and also taking a moving image. The present invention is also applied to recording an image when the electronic shutter is not activated.

1 Smartphone (main unit)
2 Imaging device 3 Display device 4 Storage device 5 Sensor 6 Control device

Claims

It is a smartphone that controls the operation of the imaging device mounted on it by moving the smartphone body,
An imaging device for imaging an object to be imaged;
A sensor for measuring speed, acceleration, angular velocity or angular acceleration in the three-axis directions of the imaging device;
A control device that controls the operation of the imaging device using at least one of three sensor values corresponding to each of the three axial directions output from the sensor;
With
The control device includes:
The three sensor values are periodically received from the sensor, and when the change amount of at least one of the three sensor values exceeds a threshold value, the time point at which the threshold value is exceeded is determined as a time change pattern. As a start time, it is determined whether the time change patterns of the three sensor values correspond to a predetermined time change pattern,
After the predetermined time change pattern, it is determined whether or not the three sensor values are substantially constant with respect to time, and when the three sensor values are substantially constant, the image of the imaging device at the start time and the start time Compared with the image of the imaging device at the time when only T ′ has elapsed from the time, the imaging device is operated only at the time when the difference between both images is smaller than a threshold,
The value of T ′ is 1.5 to 2.5 seconds,
The predetermined time change pattern is a time change pattern generated by rotating the smartphone in one direction about any one of the three axes and then rotating the smartphone in the opposite direction to return to the position before the rotation. A smartphone characterized by being.
The control device causes the user to rotate the smartphone in one direction about any one of the three axes, and then rotate the smartphone in the opposite direction to return to the position before the rotation. The smartphone according to claim 1, wherein the predetermined time change pattern is determined by receiving the three sensor values from.
The smartphone according to claim 2, wherein the control device calculates a difference between the at least one sensor value and the predetermined time change pattern, and compares the sum of the difference with a threshold value.
The sensor is an acceleration sensor that measures acceleration in the three-axis directions of the imaging device,
When the two sensor values of the three sensor values change with respect to time and the remaining one sensor value is substantially constant with respect to time, the control device determines the time of the two sensor values. The smart phone according to claim 1, wherein a change pattern is determined.
The time change to a predetermined pattern, the acceleration value is a sensor value, (1) increased by -9.8m / s 2 from X1, the flow returns to -9.8m / s 2 within the time T changes ,
(2) increases or decreases from 0 m / s 2 only X2, returns to 0 m / s 2 within time T changes,
(3) decreased from 9.8 m / s 2 only X1, and any of a change pattern back to 9.8 m / s 2 within the time T,
The value of X1 is 4.9 to 14.7 m / s 2 ,
The value of X2 is 8.5 to 9.8 m / s 2 ,
The smartphone according to claim 4, wherein the value of T is 0.8 to 1.8 seconds.
The sensor is an angular velocity sensor that measures a rotation angle with respect to three axes of the imaging device,
When the sensor value of one of the three sensor values changes with respect to time and the remaining two sensor values are substantially constant with respect to time, the control device determines the time of the one sensor value. The smart phone according to claim 1, wherein a change pattern is determined.
In the predetermined time change pattern, the value of the angular velocity that is the sensor value increases or decreases from 0 (1) and returns to 0, and the absolute value of the integral of the sensor value is Y radians,
(2) A change pattern in which a decrease or increase from 0 returns to 0, and the change in which the absolute value of the integral of the sensor value is Y radians continues during time T;
The value of Y is π / 3 to 2π / 3 radians,
The smartphone according to claim 6, wherein the value of T is 0.8 to 1.8 seconds.
The smartphone according to any one of claims 1 to 7, wherein the control device compares the difference between the two images based on a difference in a histogram of RGB luminance values of the two images.
The control device further includes a sensor that measures a speed, an acceleration, an angular velocity, or an angular acceleration related to the three axial directions of the imaging device, and three sensor values corresponding to each of the three axial directions output from the sensor. Electronic shutter when periodically receiving three sensor values for controlling the operation of the imaging apparatus using at least one and a change amount of at least one of the three sensor values exceeds a threshold value The smartphone according to any one of claims 1 to 8, wherein a plurality of images that can be continuously acquired are cached in a memory array at a time when does not start.
The control device further includes a sensor that measures a speed, an acceleration, an angular velocity, or an angular acceleration related to the three axial directions of the imaging device, and three sensor values corresponding to each of the three axial directions output from the sensor. Electronic shutter when periodically receiving three sensor values for controlling the operation of the imaging apparatus using at least one and a change amount of at least one of the three sensor values exceeds a threshold value The smartphone according to any one of claims 1 to 8, wherein a plurality of images that can be continuously acquired are cached in a memory array at the time of turning off.
The control device divides each component of the RGB color histogram of all cached frame images into 256 gradation values of 0-255 into four equal parts and replaces them with the central representative value to reduce the color to 64 colors. Dimension vector color features are used to detect and detect feature quantities such as brightness differences, luminance distribution, and luminance gradient direction distribution of local image objects for each pixel, and the array continues beyond a predetermined threshold. The smartphone according to claim 9 or 10, wherein pixels having approximate values are grouped as similar objects to discriminate object recognition of local images existing in all frame images.
The control device deletes the hue information and the saturation information while maintaining the luminance of the RGB value, converts it to grayscale, binarizes it, extracts the contour line of each pixel by arithmetic processing, and sets a predetermined threshold value. The smartphone according to claim 11, wherein pixels having arrangement values that exceed the threshold value are grouped as similar objects, and object recognition of local images existing in all frame images is determined.
The control device constitutes a layer file to be overlaid with an xy coordinate axis at the center point of pixels grouped as a similar object, displays a focus point representing an object region of a local image, and displays it in all frame images. The smartphone according to claim 12, wherein position recognition processing of an object of a local image is performed.
The smartphone according to claim 13, wherein the control device tracks the position where the object recognition of the local image is determined with all frame images.
The control device has a photo shooting mode and a video shooting mode,
The three sensor values are periodically received from the sensor, and when the change amount of at least one of the three sensor values exceeds a threshold value, the time point at which the threshold value is exceeded is determined as a time change pattern. Threshold value for determining the sensor value of the angle and / or direction at the same time as the start time, judging from the sensor value at the start time, how many frames before the image is tilted, and taking out the image to another memory To determine whether the time change pattern of the three sensor values corresponds to a predetermined time change pattern,
After the predetermined time change pattern, it is determined whether or not the three sensor values are substantially constant with respect to time, and when the three sensor values are substantially constant, the image of the imaging device at the start time and the start time From the sensor value at the time when the image of the imaging device is still at the time when T ′ has elapsed from that time, it is determined how many frames ago the image at the time of still is, and the image is taken out to another memory, and at that time The sensor value of the angle and / or direction is compared with a predetermined threshold value and held together as a stationary sensor value at the end, the stationary sensor value at the start and the stationary sensor value at the end are compared, and the sensor value of both images Only when the difference is smaller than a threshold value, the imaging device is operated at the time when the time has elapsed, and the shutter operation is started in the photo shooting mode, or in the video shooting mode The smartphone according to any one of claims 9 to 14, wherein shooting is started.
The control device has a photo shooting mode and a video shooting mode,
The three sensor values are periodically received from the sensor, and when the change amount of at least one of the three sensor values exceeds a threshold value, the time point at which the threshold value is exceeded is determined as a time change pattern. It is determined whether the time change pattern of the three sensor values corresponds to a predetermined time change pattern as a start time point,
After the predetermined time change pattern, it is determined whether or not the three sensor values are substantially constant with respect to time, and when the three sensor values are substantially constant, the image of the imaging device at the start time and the start time Compared with the image of the imaging device at the time when T ′ has elapsed from the time, the imaging device is operated only when the difference between the two images is smaller than the threshold, and the shutter operation is performed in the photography mode. The smartphone according to any one of claims 9 to 14, which starts or starts shooting in the moving image shooting mode.
The smartphone according to claim 15 or 16, wherein the device control device performs a playback, stop, pause, rewind / fast forward, frame advance, frame rewind, and end operations on a moving image captured by the imaging device.
The value of T ′ is 1.5 to 2.5 seconds, and the predetermined time change pattern is obtained by rotating the smartphone in one direction about any one of the three axes, The smartphone according to any one of claims 15 to 17, which is a time change pattern generated by rotating in a reverse direction and returning to a position before the rotation.