JP2017091455A - Image processing device, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of reliably selecting a desired character string, and an image processing method and an image processing program.SOLUTION: An image processing device includes first and second acquisition sections, and a processing section. The first acquisition section acquires data related to a plurality of images including character strings in time series. The second acquisition section receives an input. The processing section executes detection operation and selection operation. The detection operation includes detection of a plurality of image regions from the plurality of images. Each of the plurality of image regions individually includes each of the character strings included in each of the plurality of images. Each of the plurality of image regions is detected in detection time, and each of the plurality of image regions has an images region position. The selection operation includes selection of at least one of the plurality of image regions. The selection is performed on the basis of a first difference between each detection time of the plurality of image regions and reception time when the input is received and a second difference between each image region position of the plurality of image regions and a position of the input.SELECTED DRAWING: Figure 1

Description

  Embodiments described herein relate generally to an image processing apparatus, an image processing method, and an image processing program.

  There is an image processing apparatus that captures a character string existing in real space with a camera and detects the character string from the captured image. In an image processing apparatus, there are cases where a user selects a desired character string in an image while displaying an image taken in real time, and displays various information related to the character string. In this case, there is a time lag until the user recognizes and selects a desired character string in the image. The position of the character string in the image changes during the time lag. For this reason, the user may not be able to select a desired character string. In such an image processing apparatus, it is desired that a desired character string can be reliably selected.

Japanese Patent Laying-Open No. 2015-88046

  Embodiments of the present invention provide an image processing apparatus, an image processing method, and an image processing program capable of reliably selecting a desired character string.

  According to the embodiment of the present invention, an image processing apparatus including a first acquisition unit, a second acquisition unit, and a processing unit is provided. The first acquisition unit acquires data related to a plurality of images including a character string in time series. The second acquisition unit receives an input. The processing unit performs a detection operation and a selection operation. The detection operation includes detecting a plurality of image regions from the plurality of images. Each of the plurality of image regions includes each of the character strings included in each of the plurality of images. Each of the plurality of image areas is detected at a detection time, and each of the plurality of image areas has an image area position. The selection operation includes selecting at least one of the plurality of image regions. The selection includes a first difference between the detection time of each of the plurality of image regions and a reception time at which the input is received by the second acquisition unit, and the image region of each of the plurality of image regions. Based on the second difference between the position and the input position.

1 is a block diagram illustrating an image processing apparatus according to a first embodiment. It is a figure which illustrates a plurality of images. It is a schematic diagram which illustrates the operation example of the detection part which concerns on 1st Embodiment. FIG. 4A to FIG. 4C are schematic views illustrating the method for acquiring the designated position. It is a schematic diagram which illustrates the operation example of the selection part which concerns on 1st Embodiment. FIG. 6A and FIG. 6B are schematic views illustrating images of the image processing apparatus according to the reference example. It is a block diagram which illustrates the image processing device concerning a 2nd embodiment. FIG. 8A and FIG. 8B are schematic views illustrating an operation example of the grouping unit according to the second embodiment. It is a block diagram which illustrates the image processing device concerning a 3rd embodiment. It is a schematic diagram which illustrates the inclination correction method which concerns on 3rd Embodiment. It is a block diagram which illustrates the image processing device concerning a 4th embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between the parts, and the like are not necessarily the same as actual ones. Further, even when the same part is represented, the dimensions and ratios may be represented differently depending on the drawings.
Note that, in the present specification and each drawing, the same elements as those described above with reference to the previous drawings are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.

(First embodiment)
FIG. 1 is a block diagram illustrating an image processing apparatus according to the first embodiment.
The image processing apparatus 110 according to the embodiment includes a first acquisition unit 11, a second acquisition object 12, a processing unit 20, and a display unit 30. For example, an input / output terminal is used for the first acquisition unit 11. The acquisition unit 10 includes an input / output interface that communicates with the outside via a wired or wireless connection. For the processing unit 20, for example, an arithmetic device including a CPU (Central Processing Unit) and a memory is used. An integrated circuit such as LSI (Large Scale Integration) or an IC (Integrated Circuit) chip set can be used for some or all of the blocks of the processing unit 20. An individual circuit may be used for each block, or a circuit in which part or all of the blocks are integrated may be used. Each block may be provided integrally, or a part of the blocks may be provided separately. In addition, a part of each block may be provided separately. The integration is not limited to LSI, and a dedicated circuit or a general-purpose processor may be used.

  The processing unit 20 is provided with a detection unit 21 and a selection unit 22. Each of these units is realized as an image processing program, for example. That is, the image processing apparatus 110 can also be realized by using a general-purpose computer apparatus as basic hardware. The functions of the units included in the image processing apparatus 110 can be realized by causing a processor mounted on the computer apparatus to execute an image processing program. At this time, the image processing apparatus 110 may be realized by installing the above-described image processing program in a computer device in advance, or may be stored in a storage medium such as a CD-ROM or via the network. It may be realized by distributing a processing program and installing the image processing program in a computer apparatus as appropriate. The processing unit 20 is realized by appropriately using a memory, a hard disk or a storage medium such as a CD-R, a CD-RW, a DVD-RAM, a DVD-R, or the like that is built in or externally attached to the computer device. Can do.

  The display unit 30 is configured by a liquid crystal display, for example. In this example, the display unit 30 includes the second acquisition unit 12. The second acquisition unit 12 includes, for example, a touch panel provided integrally with the display unit 30.

In the embodiment, the first acquisition unit 11 acquires data related to a plurality of images including a character string in time series.
The second acquisition unit 12 receives input from the user.
The detection unit 21 performs a detection operation. The detection operation includes detecting a plurality of image regions from the plurality of images. Each of the plurality of image regions includes each of character strings included in each of the plurality of images. Each of the plurality of image areas is detected at the detection time. Each of the plurality of image areas has an image area position.
The selection unit 22 performs a selection operation. The selection operation includes selecting at least one of the plurality of image regions. The selection includes the first difference between the detection time of each of the plurality of image areas and the reception time at which the input is received by the second acquisition unit 12, the position of each image area of the plurality of image areas, and the input position. And based on the second difference.

  In the image processing apparatus, it is assumed that the user selects a desired character string in the image and displays various information regarding the character string while displaying the image taken in real time. In this case, a time lag occurs until the user recognizes a desired character string in the image and selects the character string. The position of the character string in the image may change during the time lag. For this reason, the user may not be able to select a desired character string.

  On the other hand, the image processing apparatus 110 according to the embodiment acquires a plurality of images including a character string in time series, and detects a plurality of image regions from the plurality of images. Each of the plurality of image areas includes each of character strings. The image processing apparatus 110 receives an input from the user, and selects an image area close to the input position from among a plurality of image areas with reference to the input reception time. Thereby, the user can select a desired character string reliably.

FIG. 2 is a diagram illustrating a plurality of images.
As shown in FIG. 2, data 40 _j , 40 _{j + 1} ,..., 40 _k (hereinafter referred to as images 40 _j , 40 _{j + 1} ,..., 40 _k ) related to a plurality of images are acquired in time series. Each of the plurality of images 40 _j , 40 _{j + 1} ,..., 40 _k includes a character string. The images 40 _j , 40 _{j + 1} ,..., 40 _k may be images including character strings existing in real space such as signs, signs, and restaurant menus. The first acquisition unit 11 acquires the image 40 _j at time t _j , acquires the image 40 _{j + 1} at time t _{j + 1} , and acquires the image 40 _k at time t _k . The first acquisition unit 11 may acquire the images 40 _j , 40 _{j + 1} ,..., 40 _k from an imaging device such as a digital still camera. In this case, the imaging device may be provided integrally with the image processing apparatus 110 or may be provided separately. Display unit 30, a plurality of images _{40 j, 40 j + 1,} ..., are displayed in the order that has acquired the 40 _k.

The detection unit 21 detects a plurality of image regions from the plurality of images 40 _j , 40 _{j + 1} ,..., 40 _k . In this example, image regions R1 _{j to} R4 _j are detected from the image 40 _j . The image area R1 _j includes a character string c1 _j . The image area R2 _j includes a character string c2 _j . The image area R3 _j includes a character string c3 _j . The image region R4 _j includes a character string c4 _j . Similarly, image regions R1 _{j + 1 to} R4 _{j + 1} are detected from the image 40 _{j + 1} . The image area R1 _{j + 1} includes a character string c1 _{j + 1} . The image area R2 _{j + 1} includes a character string c2 _{j + 1} . The image area R3 _{j + 1} includes a character string c3 _{j + 1} . The image area R4 _{j + 1} includes a character string c4 _{j + 1} . Image regions R1 _{k to} R3 _k are detected from the image 40 _k . The image area R1 _k includes a character string c1 _k . The image area R2 _k includes a character string c2 _k . The image region R3 _k includes a character string c3 _k . One of these images 40 _j , 40 _{j + 1} ,..., 40 _k is also simply referred to as an image 40.

Each of the image areas R1 _{j to} R4 _j , R1 _{j + 1 to} R4 _{j + 1} ,..., R1 _{k to} R3 _k is represented by a rectangular area, for example. The detection unit 21 acquires the detection times and image region positions of the image regions R1 _{j to} R4 _j , R1 _{j + 1 to} R4 _{j + 1} ,..., R1 _{k to} R3 _k . Note that the processing unit 20 has a clock function for acquiring time. In this example, the detection times of the image regions R1 _{j to} R4 _j are t _j (detection time t _j ). The detection times of the image areas R1 _{j + 1 to} R4 _{j + 1} are t _{j + 1} (detection time t _{j + 1} ). The detection times of the image areas R1 _{k to} R3 _k are t _k (detection time t _k ). One of these image regions R1 _{j to} R4 _j , R1 _{j + 1 to} R4 _{j + 1} ,..., R1 _{k to} R3 _k is also simply referred to as an image region R. The display unit 30 may display a plurality of image regions R with a frame surrounding a character string so that the user can identify the image regions R.

FIG. 3 is a schematic view illustrating an operation example of the detection unit 21 according to the first embodiment.
In the figure, the X direction is along the horizontal direction of the image. The Y direction is along the vertical direction of the image. The T direction represents the time direction.

As illustrated in FIG. 3, the image 40 and the image 40 _k are acquired in time series. Each of the image 40 and the image 40 _k includes a character string (not shown). The image 40 is acquired at an arbitrary time t (t _j ≦ t <t _k ). The image 40 is one of the images 40 _j , 40 _{j + 1,.} Image 40 _k is acquired at time t _k . Although two images are illustrated here, three or more images may be used.

For example, the image 40, the image region _R ^{i t} including the character string is detected. The image area R _i ^t means the i-th detection result at time t. If the character string in the image region _R ^{i t} is linear, the image area position of the image region _R ^{i t} _{p t} is two-dimensional coordinates of 4 points ^{_{(x i t (1),}} y i t (1 ^{_{)), (x i t (}} 2), y i t (2)), (x i t (3), y i t (3)), (x i t (4), y i t (4)) Can be used.

In this example, the image area position p _t of the i-th image area R _i ^t at time t is (x _i ^t (1), y _i ^t (1), x _i ^t (2), y _i ^t ( 2), x _i ^t (3), y _i ^t (3), x _i ^t (4), y _i ^t (4), t).

That is, each image area position p _t of the plurality of image regions R, as in the image region R _i ^t, can be expressed using the two-dimensional coordinates of 4 points. In this way, it is possible to obtain the respective detection time t and the image area position p _t of the plurality of image areas R.

  Here, a specific example of a character string detection method will be described. For example, among pixels in an image, pixels having similar characteristics such as pixel color are connected between adjacent pixels to generate one or more connected components. Specifically, when pixels in the image are binarized with white and black, and two or more black pixels adjacent to each other are consecutively generated, a set of continuous black pixels is generated as a connected component. To do.

  In accordance with the positional relationship of the connected components and the similarity of the connected components, the connected components arranged on substantially the same straight line are combined and detected as a character string. Specifically, for example, a feature vector is generated for each connected component, and the positional relationship between the two connected components and the feature similarity are defined by the distance between the feature vectors. If the distance of the feature vector is less than the threshold value, the two connected components are similar and are considered to be aligned on the same straight line. In this case, two connected components are connected. Examples of each element of the feature vector include an x coordinate and ay coordinate representing the center point of the connected component, an average color of each connected component, and a size (height, width, perimeter length, etc.) of the connected component. The center point may be, for example, a rectangular center point that circumscribes the connected component. A rectangular area circumscribing the character string thus detected is detected as an image area. The method for detecting the character string is not limited to this.

The second acquisition unit 12 receives an input of the designated position p designated by the user. The second acquisition unit 12 includes, for example, a touch panel. The processing unit 20 acquires the reception time t _p (> detection time t _k ) when the second acquisition unit 12 receives the input of the designated position p using the clock function. The designated position p is, for example, a position on the display screen that is designated when the user performs a tap operation or the like on the touch panel. Indication position p, for example, can be represented using a two-dimensional coordinates on the display screen in the reception time t _p. The designated position p is represented, for example, by (x _k , y _k , t _p ). In addition, the acquisition method of the designated position p is not limited to a touch panel. The pointing position p may be acquired using a pointing device such as a mouse, for example.

FIG. 4A to FIG. 4C are schematic views illustrating a method for obtaining the designated position p.
The display unit 30 according to the embodiment integrally includes a touch panel as the second acquisition unit 12. The display unit 30 displays the image _40k . The designated position p is input as one point on the display screen of the display unit 30 as shown in FIG. The designated position p is input, for example, when the user performs a tap operation. In this case, the designated position p is represented by one coordinate. In this way, the coordinate point of the indication position p _{(x k, y} k) and a receiving time _{t p,} it is obtained.

The designated position p is input as a line segment on the display screen of the display unit 30 as shown in FIG. The designated position p is input, for example, when the user draws a line segment by a drag operation. The shape of the line segment is not limited. The line segment may be bent. In this case, the designated position p is represented by a coordinate group. The coordinate group is composed of a plurality of coordinates that are successively specified on the display screen. In this way, each coordinate point p _k (x _k (k), y _k (k)) (k = 1, 2,..., K) constituting the coordinate group of the designated position p, and the reception time t _p , Is acquired.

The designated position p is input as an area on the display screen of the display unit 30 as shown in FIG. The designated position p is input, for example, when the user draws a surrounding line by a drag operation. The shape of the surrounding line is not limited. The encircling line may be bent. In this case, the designated position p is represented by a coordinate area. The coordinate area is surrounded by a coordinate group composed of a plurality of coordinates that are successively designated on the display screen. In this way, each coordinate point p _k (x _k (k), y _k (k)) (k = 1, 2,..., K) constituting the coordinate area of the designated position p, and the reception time t _p , Is acquired.

FIG. 5 is a schematic view illustrating an operation example of the selection unit 22 according to the first embodiment.
In the figure, the X direction is along the horizontal direction of the image. The Y direction is along the vertical direction of the image. The T direction represents the time direction.

The selection unit 22 performs a selection operation for selecting at least one of the plurality of image regions R. The selection is made based on the first difference and the second difference. The first difference, each of the detection time t of the plurality of image areas R, which is the difference between the reception time t _p,. The second difference, each of the image area position p _t of the plurality of image areas R, which is the difference between the indicated position p,. Specifically, at least one first difference selected from among the plurality of image regions R is smaller than another one first difference that has not been selected from among the plurality of image regions R. At least one second difference selected from among the plurality of image regions R is smaller than another second difference that has not been selected from among the plurality of image regions R.

For example, the selection unit 22 from among the plurality of image regions R, and extracts an image region R that has been detected from the receiving time t _p in a certain time period [delta] _t. The fixed time δ _t is a value determined in advance based on experiments or the like.

  As shown in FIG. 5, image areas R1a to R1c, image areas R2a to R2c, image areas R3a and R3b, and image areas R4a and R4b are detected. The detection time t is acquired for each of the image areas R1a to R1c, image areas R2a to R2c, image areas R3a and R3b, and image areas R4a and R4b.

Selection unit 22 based on the respective detection time t of the image region R, an image region R that has been detected from the receiving time t _p of the indicated position p within a predetermined time [delta] _t, the image area R1a～R1c, the image area Extracted from R2a to R2c, image regions R3a and R3b, and image regions R4a and R4b. That is, the image area R1a～R1c, the image area R2a～R2c, image region R3a, R3b, and the image area R4a, respectively and detection time t of R4b, a receiving time _{t p,} the difference (first difference), image region R are extracted within a certain time period [delta] _t. Hereinafter, the extracted image region R is referred to as an extraction region RA.

In this example, the image areas R1a to R1c, the image areas R2a to R2c, the image areas R3a and R3b, and the image area R4a are the extraction area RA. The extraction areas RA1a to RA1c correspond to the image areas R1a to R1c. The extraction areas RA2a to RA2c correspond to the image areas R2a to R2c. The extraction areas RA3a and RA3b correspond to the image areas R3a and R3b. The extraction area RA4a corresponds to the image area R4a. Image area R4b is not detected within a predetermined time [delta] _t. For this reason, the image region R4b is not extracted.

The average reaction time of human vision is, for example, 180 ms or more and 200 ms or less. This reaction time is a time lag from when the user recognizes a desired character string in the image until the user touches the image with a finger or the like. For this reason, the fixed time δ _t is preferably set to, for example, 100 milliseconds (ms) or more and 500 ms or less in consideration of a time lag. Thus, in the embodiment, the image region R that has been detected only in the past _t certain time δ from the receiving time t _p, which has received the user operation a selection target.

Selecting unit 22 (here, the extraction region RA1a～RA1c, extraction region RA2a～RA2c, extraction region RA3a, RA3b, and extraction region RA4a) a plurality of extraction regions RA and each image area position _{p t} of the designation position A second difference (distance) from p is calculated, and at least one of the plurality of extraction regions RA is selected based on the calculated distance. Specifically, for example, as shown in FIG. 3, a distance scale dist (p, R _i ^t ) is calculated. x and y represent the horizontal and vertical coordinates of the image, and t represents the detection time. The distance measure dist (p, R _i ^t ) represents the distance between the image region R _i ^t and the designated position p. The distance measure dist (p, R _i ^t ) can be calculated by the following equation. In this example, as shown in FIG. 4A, a case where the designated position p is a coordinate point is shown.

The point p ′ (x ′, y ′, t ′) represents an arbitrary coordinate point included in the image region R _i ^t . In other words, the distance D (p, p ′) is obtained from Equation 2. The distance D (p, p ′) is determined by the coordinate point (x _k , y _k , t _p ) of the designated position p and a plurality of points p ′ (x ′, y ′, t ′) included in the image region R _i ^t. ) Between each of them. Then, using Equation 1, the minimum value of the distance D (p, p ′) is obtained as a distance scale dist (p, R _i ^t ). In this example, the coordinate system of (x, y, t) is used in consideration of the detection time t, but the distance scale is calculated using only (x, y) without considering the detection time t. You may ask for it.

Here, when the designated position p is a line segment as shown in FIG. 4B or when the designated position p is a region as shown in FIG. each coordinate point _p k of _{p (x k (k),} y k (k)) (k = 1,2, ..., K) for a plurality of points included in the image region ^{_{R i t p '(x'}} , A distance D (p _k , p ′) between each of y ′ and t ′) is _obtained . According to Equation 1, the minimum value of the distance D (p _k , p ′) is obtained as dist (p _k , R _i ^t ). ^{_{dist (p k, R i t}} ) is determined for each coordinate point _{p k} of the indication position p. Then, the total value of dist (p _k , R _i ^t ) may be calculated by the following formula and used as a distance scale.

Here, the distance scale is the sum of dist (p _k , R _i ^t ), but the distance scale may be the maximum value or the minimum value of dist (p _k , R _i ^t ).

For example, the selection unit 22 selects the extraction region RA having the smallest distance scale dist (p, R _i ^t ) according to the following equation. Here, first differential of the selected extraction area RA (difference between the detection time t and receive time t _p) is within the predetermined time [delta] _t.

That is, the distance measure dist _{(p, R} ^{i _t),} and a range of _{t p -δ t ≦ t ≦ t} p, determining the smallest _{(t min, i} min). As a result, the i _min th extraction region RA at time t _min is selected. Here, the display unit 30 may display the selected extraction region RA in a state where the user can identify it in the image. For example, a method of changing the frame color of the extraction area RA or the color of the character string in the extraction area RA is conceivable. Further, the selected extraction area RA may provide only the position information (coordinates and time) to the user. The image is stored in, for example, an external server device. In this case, it is preferable to associate the position information and the image so that the image can be read from the position information.

The selection unit 22 may select a plurality of extraction regions RA in ascending order of the distance scale dist (p, R _i ^t ). For example, it may be set in advance so that the five extraction regions RA are selected in ascending order of the distance scale dist (p, R _i ^t ). In this case, images of the five selected extraction areas RA are presented to the user. The user can select a desired image of the extraction area RA from the images of the five extraction areas RA.

FIG. 6A and FIG. 6B are schematic views illustrating images of the image processing apparatus according to the reference example.
As described above, an image processing apparatus is assumed in which a user selects a desired character string in an image and displays various information related to the character string while displaying an image captured in real time.

In the reference example of FIG. 6A, the plurality of images 40 _j , 40 _{j + 1} ,..., 40 _k include character strings and are acquired in time series. Image 40 _j is acquired at time t _j , image 40 _{j + 1} is acquired at time t _{j + 1} , and image 40 _k is acquired at time t _k . A plurality of image regions R are detected from the plurality of images 40 _j , 40 _{j + 1} ,..., 40 _k .

In this example, image regions R1 _{j to} R4 _j are detected from the image 40 _j . The image area R1 _j includes a character string c1 _j . The image area R2 _j includes a character string c2 _j . The image area R3 _j includes a character string c3 _j . The image region R4 _j includes a character string c4 _j . Similarly, image regions R1 _{j + 1 to} R4 _{j + 1} are detected from the image 40 _{j + 1} . The image area R1 _{j + 1} includes a character string c1 _{j + 1} . The image area R2 _{j + 1} includes a character string c2 _{j + 1} . The image area R3 _{j + 1} includes a character string c3 _{j + 1} . The image area R4 _{j + 1} includes a character string c4 _{j + 1} . Image regions R1 _{k to} R3 _k are detected from the image 40 _k . The image area R1 _k includes a character string c1 _k . The image area R2 _k includes a character string c2 _k . The image region R3 _k includes a character string c3 _k . Here, in the image 40 _k , the image region R4 _k is not detected.

At time t _j , the user recognizes the character string c4 _j in the image 40 _j and moves the finger f to select it. At time t _k , the user touches character string c4 _k with finger f. However, since the image region R4 _k including the character string c4 _k has not been detected, the image region R4 _k cannot be selected.

In the reference example of FIG. 6B, an image region R4 _k is detected from the image 40 _k . However, the camera turns upward at time t _k , and the position of the character string c4 _k has moved downward in the image 40 _k . That is, at time _{t j,} for the user to recognize the character string c4 _j in the image 40 _j, selected, when you move the finger f, for movement of the string c4 _k, time _{t k} in, the user would touch the top of the string c4 _k in the finger f contrary to the intention. Again, it is not possible to select an image area R4 _k.

  That is, there is a time lag until the user recognizes and selects a desired character string in the image. It is possible that the character string in the image is not detected during the time lag, or the position of the character string in the image changes during the time lag. For this reason, the user may not be able to select a desired character string.

In contrast, the image processing apparatus 110 according to the embodiment, to obtain a plurality of images ₄₀ j to 40 _k in a time series, detecting a plurality of image regions R1 _j ~ from a plurality of images ₄₀ j to 40 _k. Each of the plurality of image regions R1 _j to includes a character string. The image processing apparatus 110 receives an input from the user, based on the reception time t _p of the input, selects the image region R near the indicated position p among the _~ plurality of image regions R1 j. Thereby, the user can select a desired character string reliably.

(Second Embodiment)
FIG. 7 is a block diagram illustrating an image processing apparatus according to the second embodiment.
The image processing apparatus 111 according to the embodiment includes a processing unit 20. The processing unit 20 includes a detection unit 21 and a selection unit 22, and further includes a grouping unit 23 and an in-group selection unit 24.

The detection unit 21 performs a detection operation. Detecting operation comprises detecting a plurality of image regions R from a plurality of images 40 _j to 40 _k, to obtain the respective detection time t and the image area position p _t of the plurality of image areas R. The detection unit 21 further calculates an evaluation value S (R) that represents the character string likelihood for each of the plurality of character strings. The evaluation value S (R) is, for example, the connected components i = 1 to N (provided that the connected component degree Ci feature vector of each connected component of the black pixels constituting the character string is Vi). , 1 to N are assumed to be sorted and arranged from one starting end of the character string to the other starting end), and can be calculated using the following equation. D (A, B) represents the Euclidean distance between the vector A and the vector B.

The second acquisition unit 12 receives an input of the designated position p from the user. At this time, the processing unit 20 uses the clock function, and acquires the reception time t _p the second obtaining section 12 receives the input of the indication position p.

FIG. 8A and FIG. 8B are schematic views illustrating an operation example of the grouping unit 23 according to the second embodiment.
FIG. 8B is an enlarged view of the W portion of FIG.
In the figure, the X direction is along the horizontal direction of the image. The Y direction is along the vertical direction of the image. The T direction represents the time direction.

The grouping unit 23 performs a grouping operation. The grouping operation includes grouping image regions R including the same character string among the plurality of image regions R to generate a plurality of groups G. That is, the grouping unit 23 performs a process of grouping a plurality of adjacent image regions R into one group. More specifically, the grouping unit 23 detects a plurality of character strings from a plurality of images ₄₀ j to 40 _k, associating the character string with each other is determined identical between the image ₄₀ j to 40 _k. As a result, the character string is tracked, and a process of collecting a plurality of image regions R including the same character string into one group is performed based on the tracking result. For the grouping process, for example, a method generally used for tracking a moving object (person, car, etc.) can be used.

When a plurality of image regions R are grouped into one group G, they are represented as a three-dimensional shape on a three-dimensional space-time. In this example, as shown in FIG. 8A, four groups G1 to G4 are generated. Each of the four groups G1 to G4 is represented as a rectangular parallelepiped shape, for example. For example, the set R ^g of the image region R included in the group G1, as the following equation, and a N (g) pieces of the image region R.

The selection unit 22 performs a selection operation for selecting at least one of the plurality of groups G1 to G4. The selection is made based on the third difference and the fourth difference. The third difference is each a group detection time t of the plurality of groups G1 to G4, which is the difference between the reception time _{t p,.} The fourth difference is each a group position _{p t} of the plurality of groups G1 to G4, which is the difference between the indicated position p,. The group detection time t is based on the detection time t of each image region R. Group position _{p t} is based on the image area position _{p t} of each image region R.

  At least one third difference selected from the plurality of groups G1 to G4 is smaller than another third difference that has not been selected from the plurality of groups G1 to G4. At least one fourth difference selected from the plurality of groups G1 to G4 is smaller than another one fourth difference that has not been selected from the plurality of groups G1 to G4.

For example, the selection unit 22 extracts a group G including a plurality of extraction regions RA from the plurality of groups G1 to G4. Incidentally, the extraction area RA, this means that the detected image area R for a predetermined time δ in _t from the receiving time t _p of the indication position p. And each group detection time t of the plurality of groups G1 to G4, a receiving time t _p, the difference (third differential), the group G is extracted within a certain time period [delta] _t. Hereinafter, the extracted group G is referred to as an extracted group GA. In this example, all the groups G1 to G4 are extracted. For this reason, the groups G1 to G4 coincide with the extraction groups GA1 to GA4.

The selection unit 22 calculates a fourth difference (distance) between each of the plurality of extraction groups GA1 to GA4 and the designated position p, and based on the calculated distance, at least one of the plurality of extraction groups GA1 to GA4. Select. This distance is represented by, for example, a distance scale dist (p, R ^g ). The selection unit 22 calculates the distance measure dist (p, R ^g ) according to the following equation, and selects the extraction group GA having the smallest distance measure dist (p, R ^g ). For example, the extraction group GA1 is selected.

In Expression 7, the point p ′ (x ′, y ′, t ′) is an arbitrary coordinate included in each of the image regions R ₁ ^g , R ₂ ^g ,... _{RN (g)} ^g constituting the extraction group GA. Represents a point. That is, the distance D (p, p ′) is obtained by the above-described equation 2. The distance D (p, p ′) is a plurality of coordinate points (x _k , y _k , t _p ) of the designated position p and a plurality of image regions R ₁ ^g , R ₂ ^g ,... _{RN (g)} ^g The distance between each of the points p ′ (x ′, y ′, t ′). Then, the minimum value of the distance D (p, p ′) is obtained as a distance scale dist (p, R ^g ) using Equation 1.

The intra-group selection unit 24 performs an intra-group selection operation. The intra-group selection operation includes selecting at least one from among a plurality of extraction regions RA to RAN included in the extraction group GA1 selected by the selection unit 22. For example, the extraction region RA including the character string having the highest evaluation value S (R) is selected from the plurality of extraction regions RA1 to RAN. From among N (g) extraction regions RA belonging to a certain extraction group GA, the extraction region R _max ^g having the highest evaluation value S (R _j ^g ) can be selected by the following equation. Note that R _j ^g means the j-th extraction region of a certain extraction group GA.

The intra-group selection unit 24 may perform another intra-group selection operation. In this case, the selection is made based on the fifth difference. Fifth difference, the respective image area position _{p t} of the plurality of extraction regions RA1-RAn, which is the difference between the indicated position p,. That is, the intra-group selection unit 24 calculates the fifth difference (distance) between each of the plurality of extraction regions RA1 to RAN and the designated position p, and selects the extraction region RA with the shortest calculated distance. May be. This distance is represented by a distance scale d (p, R _j ^g ), for example. For example, an extraction region R _min ^g having the smallest distance scale d (p, R _j ^g ) among N (g) extraction regions RA belonging to a certain detection group GA can be calculated by the following equation. Note that R _j ^g means the j-th extraction region of a certain extraction group GA.

The intra-group selection unit 24 may select a plurality of extraction regions RA in ascending order of the distance scale d (p, R _j ^g ). For example, it may be set in advance so that the five extraction regions RA are selected in ascending order of the distance scale d (p, R _j ^g ). In this case, images of the five selected extraction areas RA are presented to the user. The user can select a desired image of the extraction area RA from the images of the five extraction areas RA.

  As described above, according to the embodiment, it is possible to further improve the accuracy of selecting a desired character string by adding a character string grouping process.

(Third embodiment)
FIG. 9 is a block diagram illustrating an image processing apparatus according to the third embodiment.
The image processing apparatus 112 according to the embodiment includes a processing unit 20. The processing unit 20 includes a detection unit 21 and a selection unit 22, and further includes an inclination detection unit 25 and a position correction unit 26.

The inclination detection unit 25 performs an inclination detection operation. The tilt detection operation includes detecting the tilt of the image processing apparatus 112 when the input of the designated position p is received. For detecting the tilt, for example, various sensors such as a gyro sensor are used.
The position correction unit 26 performs a position correction operation. Position correcting operation includes a respective image area position p _t of the plurality of image regions R, so as to correct according to the inclination.

FIG. 10 is a schematic view illustrating a tilt correction method according to the third embodiment.
The image processing apparatus 112 includes a camera 50. This example will be described using the world coordinate system Wc and the camera coordinate system Cc.

The position of the camera 50 is expressed as (x _o , _yo , z _o ) in the world coordinate system Wc. The direction of the camera 50 is represented by a rolling angle ω, a pitching angle φ, and a yawing angle κ. The rolling angle ω, the pitching angle φ, and the yawing angle κ represent rotation angles on the X axis, the Y axis, and the Z axis of the world coordinate system Wc, respectively.

Consider a vector from the projection center of the camera 50 to the image region R taken. The world coordinates of a certain vertex on the image area R are (x _p , y _p , z _p ), the position of the camera 50 at the current time is (x _o , _yo , z _o ), and the direction of the camera 50 is (ω _o , phi _o, when the kappa _o), components of the vector of the camera coordinate system _{Cc (u p, v p,} w p) and the component of the vector in the world coordinate system _{Wc (x p -x o, y} p -y _o , z _p -z _o ) and the following relationship holds:

In order to simplify Expression 10, if the rotation matrix for each of the three axes is represented by one matrix having elements a _{11 to} a ₃₃ as elements, the following expression is obtained.

  Further, the position of the image region R on the camera coordinate system Cc (u, v) has the following relationship. Note that c represents the focal length of the lens.

When the position and direction of the camera 50 change by (Δx _o , Δy _o , Δz _o ) and (Δω _o , Δφ _o , Δκ _o ), the position change (Δu) of the image region R on the camera coordinate system Cc. , Δv) can be approximately obtained from Equations 12 and 13.

The right side of Expression 12 and the right side of Expression 13 are respectively expressed as Fu (x _o , _yo , _zo, ω _o , φ _o , κ _o ), Fv (x _o , _yo , z _o, ω _o , φ _o). , Κ _o ), the following equation is obtained.

Fu (x _o , _yo , _zo, ω _o , φ _o , κ _o ), Fv (x _o , _yo , _zo, ω _o , φ _o , κ _o ) are respectively (x _o , _yo , If Taylor expansion is performed around z _o, ω _o , φ _o , κ _o ), (Δu, Δv) can be calculated by the following equation.

  The derivative by Taylor expansion is calculated as follows.

As shown in FIG. 10, the world coordinate system Wc is set so that the Z-axis direction of the world coordinate system Wc and the directions of the camera 50 (ω _o , φ _o , κ _o ) are the same. The position (x _o , y _o , z _o ) of the camera 50 at the current time is at a distance H on the Z axis of the world coordinate system Wc. At this time, the camera position (0, 0, H) and the camera direction (0, 0, 0) are obtained, and (Δu, Δv) can be calculated. Actually, since the distance H is unknown, a predetermined value may be used instead. As a result, the approximate position can be corrected. Then, (Δu, Δv) is output as a position correction value in the camera coordinate system Cc.

  As described above, according to the embodiment, it is possible to detect the inclination of the image processing apparatus 112 when a touch operation or the like from the user is received, and to correct the detection position of the image area according to the inclination. For this reason, it becomes possible to select a desired character string with higher accuracy.

(Fourth embodiment)
FIG. 11 is a block diagram illustrating an image processing apparatus according to the fourth embodiment.
The image processing apparatus 200 according to the embodiment includes a desktop or laptop general-purpose computer, a portable general-purpose computer, other portable information devices, an information device having an imaging device, a smartphone, and other information processing devices. It can be realized by various devices.

  As illustrated in FIG. 11, the image processing apparatus 200 according to the embodiment includes a CPU 201, an input unit 202, an output unit 203, a RAM 204, a ROM 205, an external memory interface 206, and a communication interface as hardware configuration examples. 207.

  The instructions shown in the processing procedure shown in the above-described embodiment can be executed based on a program that is software. The general-purpose computer system stores this program in advance and reads this program, so that the same effect as that obtained by the image processing apparatus according to the above-described embodiment can be obtained. The instructions described in the above-described embodiments are, as programs that can be executed by a computer, magnetic disks (flexible disks, hard disks, etc.), optical disks (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD). ± R, DVD ± RW, etc.), semiconductor memory, or a similar recording medium. As long as the recording medium is readable by the computer or the embedded system, the storage format may be any form. If the computer reads the program from the recording medium and causes the CPU to execute instructions described in the program based on the program, the same operation as the image processing apparatus of the above-described embodiment can be realized. Of course, when the computer acquires or reads the program, it may be acquired or read through a network.

  Further, an OS (operating system) operating on a computer based on instructions from a program installed in a computer or an embedded system from a recording medium, database management software, MW (middleware) operating on a network, etc. You may perform a part of each process for implement | achieving.

  Furthermore, the recording medium in the embodiment is not limited to a recording medium independent of a computer or an embedded system, but also includes a recording medium in which a program transmitted via a LAN or the Internet is downloaded and stored or temporarily stored. Further, the number of recording media is not limited to one, and the case where the processing in the embodiment is executed from a plurality of recording media is also included in the recording medium in the embodiment. The configuration of the recording medium may be any configuration.

  The computer or the embedded system in the embodiment is for executing each process in the embodiment based on a program stored in a recording medium, and is a device composed of one of a personal computer, a microcomputer, or the like, or Any configuration such as a system in which a plurality of devices are network-connected may be used.

  In addition, the computer in the embodiment is not limited to a personal computer, and includes an arithmetic processing device, a microcomputer, and the like included in an information processing device, and is a generic term for devices and devices that can realize the functions in the embodiment by a program. ing.

  According to the embodiment, it is possible to provide an image processing apparatus, an image processing method, and an image processing program capable of reliably selecting a desired character string.

  The embodiments of the present invention have been described above with reference to specific examples. However, the present invention is not limited to these specific examples. For example, regarding the specific configuration of each element such as the acquisition unit and the processing unit, the present invention can be implemented in the same manner by appropriately selecting from a well-known range by those skilled in the art, as long as the same effect can be obtained. It is included in the scope of the invention.

  Moreover, what combined any two or more elements of each specific example in the technically possible range is also included in the scope of the present invention as long as the gist of the present invention is included.

  In addition, all image processing apparatuses, image processing methods, and image processing that can be implemented by those skilled in the art based on the image processing apparatus, the image processing method, and the image processing program described above as the embodiments of the present invention. A program also belongs to the scope of the present invention as long as it includes the gist of the present invention.

  In addition, in the category of the idea of the present invention, those skilled in the art can conceive of various changes and modifications, and it is understood that these changes and modifications also belong to the scope of the present invention. .

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 11 ... 1st acquisition part, 12 ... 2nd acquisition part, 20 ... Processing part, 21 ... Detection part, 22 ... Selection part, 23 ... Grouping part, 24 ... In-group selection part, 25 ... Inclination detection part, 26 ... position correction unit, 30 ... display _{unit, 40,40 j, 40 j + 1} , 40 k ... image (data), 50 ... camera, [delta] _t ... fixed time, kappa ... yawing angle, phi ... pitching angle, omega ... rolling angle, 110-112, 200 ... image processing apparatus, 201 ... CPU, 202 ... input unit, 203 ... output unit, 204 ... RAM, 205 ... ROM, 206 ... external memory interface, 207 ... communication interface, Cc ... camera coordinate system, G , G1 to G4 ... group, GA, GA1~GA4 ... detector groups, H ... _{distance, R, R1 j ~R4 j,} R1 j + 1 ~R4 j + 1, R1 k ~R4 k R1a~R1c, R2a~R2c, R3a, R3b, R4a, R4b ... image area, RA, RA1a~RA1c, RA2a~RA2c, RA3a , RA3b, RA4a ... extraction region, Wc ... world coordinate _{system, c1} j ~c4 j, c1 _{j + 1 to} c4 _{j + 1} , c1 _{k to} c4 _k ... character string, f ... finger, p ... designated position, p _t ... image area position, t, t _j , t _{j + 1} , t _k ... detection time, t _p ... reception time

Claims (15)

A first acquisition unit that acquires data related to a plurality of images including character strings in time series;
A second acquisition unit for receiving input;
A processing unit;
With
The processor is
A detection operation for detecting a plurality of image areas from the plurality of images, wherein each of the plurality of image areas includes each of the character strings included in each of the plurality of images; Each of the plurality of image areas is detected at a detection time, and each of the plurality of image areas has an image area position;
A selection operation for selecting at least one of the plurality of image areas, wherein the selection includes the detection time of each of the plurality of image areas and the reception time at which the input is received by the second acquisition unit. And the selection operation performed based on the first difference between the first image area and the second difference between the image area position of each of the plurality of image areas and the input position;
An image processing apparatus that implements   2. The at least one first difference selected among the plurality of image regions is smaller than another one of the first differences that was not selected among the plurality of image regions. Image processing device.   The at least one second difference selected among the plurality of image regions is smaller than another second difference that is not selected among the plurality of image regions. An image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the at least one first difference selected from the plurality of image regions is not less than 100 milliseconds and not more than 500 milliseconds. A first acquisition unit that acquires data related to a plurality of images including character strings in time series;
A second acquisition unit for receiving input;
A processing unit;
With
The processor is
A detection operation for detecting a plurality of image areas from the plurality of images, wherein each of the plurality of image areas includes each of the character strings included in each of the plurality of images; Each of the plurality of image areas is detected at a detection time, and each of the plurality of image areas has an image area position;
A grouping operation for grouping image regions including the same character string among the plurality of image regions and generating a plurality of groups, each of the plurality of groups having a group detection time based on the detection time, And the grouping operation having a group position based on the image area position;
A selection operation for selecting at least one of the plurality of groups, wherein the selection includes a third difference between the group detection time and the reception time of each of the plurality of groups; The selection operation performed based on a fourth difference between the group position of each of the groups and the position of the input;
An image processing apparatus that implements   The image processing according to claim 5, wherein the at least one third difference selected from the plurality of groups is smaller than another third difference that has not been selected from the plurality of groups. apparatus.   The at least one fourth difference selected from the plurality of groups is smaller than another one fourth difference that was not selected from the plurality of groups. Image processing apparatus. The detection operation further includes calculating an evaluation value representing character-likeness for each of the character strings,
The processing unit performs an intra-group selection operation of selecting an image region including a character string having the highest evaluation value from among a plurality of image regions included in the at least one selected from the plurality of groups. The image processing apparatus according to claim 5, further implemented.   The processing unit is an intra-group selection operation that selects at least one of a plurality of image regions included in the at least one selected from the plurality of groups, and the selection includes the plurality of the plurality of image regions. The image according to any one of claims 5 to 7, further performing the intra-group selection operation performed based on a fifth difference between each of the image region positions of the image regions and the input position. Processing equipment. The processor is
An inclination detection operation for detecting an inclination of the image processing apparatus when the input is received;
A position correction operation for correcting the image area position of each of the plurality of image areas according to the inclination;
The image processing apparatus according to claim 1, wherein the image processing apparatus is further implemented.   The image according to any one of claims 1 to 10, wherein the image region position includes horizontal and vertical coordinates in each of the plurality of images and the detection times of the plurality of image regions. Processing equipment. A display unit for displaying the image;
The image processing apparatus according to claim 1, wherein the display unit displays the at least one selected from the plurality of image areas so as to be identifiable.   The image processing apparatus according to claim 12, wherein the second acquisition unit includes a touch panel provided on the display unit. Data on multiple images including character strings is acquired in time series,
Takes input,
A plurality of image regions are detected from the plurality of images, each of the plurality of image regions includes each of the character strings included in each of the plurality of images, and each of the plurality of image regions is detected at a detection time. Each of the plurality of image areas has an image area position;
Selecting at least one of the plurality of image regions, wherein the selection includes a first difference between the detection time of each of the plurality of image regions and the reception time at which the input is received; An image processing method performed based on a second difference between each of the image region positions of the image region and the input position. Acquiring data related to a plurality of images including character strings in time series;
Receiving input; and
A plurality of image regions are detected from the plurality of images, each of the plurality of image regions includes each of the character strings included in each of the plurality of images, and each of the plurality of image regions is detected at a detection time. Detected and each of the plurality of image regions has an image region position;
Selecting at least one of the plurality of image regions, wherein the selection includes a first difference between the detection time of each of the plurality of image regions and the reception time at which the input is received; A step of performing based on a second difference between each of the image region positions of the image region and the input position;
An image processing program for causing a computer to execute.