WO2015169202A1 - 对实景图像中的字符进行识别的方法和设备 - Google Patents

对实景图像中的字符进行识别的方法和设备 Download PDF

Info

Publication number
WO2015169202A1
WO2015169202A1 PCT/CN2015/078254 CN2015078254W WO2015169202A1 WO 2015169202 A1 WO2015169202 A1 WO 2015169202A1 CN 2015078254 W CN2015078254 W CN 2015078254W WO 2015169202 A1 WO2015169202 A1 WO 2015169202A1
Authority
WO
WIPO (PCT)
Prior art keywords
plane
image
detected
connected body
real scene
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2015/078254
Other languages
English (en)
French (fr)
Inventor
张庆久
乐宁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sharp Corp
Original Assignee
Sharp Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sharp Corp filed Critical Sharp Corp
Publication of WO2015169202A1 publication Critical patent/WO2015169202A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images

Definitions

  • the present application relates generally to optical character recognition OCR techniques, and more particularly to methods and apparatus for identifying characters in a live view image.
  • the accuracy of the recognition of characters in the image by the traditional OCR technology depends on the complexity of the image to be processed, the parallelism between the plane of the text and the image presentation plane, and the degree of flatness of the text in the image.
  • the image to be processed contains only one page of text, the page text is substantially parallel to the image presentation plane, and the text in the image is substantially flat (ie, non-skewed).
  • the user may wish to optically recognize text in a live-action image that is taken daily, such as text on a sign of a perspective projection in a street scene, and may wish to word-process it.
  • text in a live-action image that is taken daily
  • the number of solid objects is large
  • the angle between the real scene plane and the image presentation plane is large (due to perspective projection)
  • the text inclination is large
  • the recognition accuracy is difficult to achieve satisfactory. Degree.
  • U.S. Patent Application No. US 20130242054 A1 entitled “Generating Hi-Res Dewarped Book Images” proposes a method of removing distortion from a book stereo image.
  • the method utilizes a three-dimensional geometric surface (eg, a cylindrical surface) to generate 3D information without the use of text lines or other content analysis.
  • a three-dimensional geometric plane is first generated according to the stereoscopic photograph. Then, calculate the corresponding model. Then, the stereo image is removed by the model.
  • a method of recognizing a character in a live view image comprising: acquiring a real scene image and a depth map, each pixel representation in the depth map corresponding to a pixel in a real scene image The distance from the real sight to the camera; according to the depth map, the real scene in the real image is detected; the detected plane is corrected so that the characters in the plane are easily recognized; and the characters in the corrected plane are recognized.
  • the detecting the real scene plane in the real scene image according to the depth map may include: detecting the connected body in the real scene image based on the depth difference of the adjacent pixels in the depth map; using the detected coordinates of the pixels of the connected body and the corresponding depth value, Plane fitting; filtering out pixels of the connected body that are outside the fitting plane; and determining convex polygons surrounding the remaining pixels of the connected body. If the depth difference of the adjacent pixels is smaller than the threshold, the adjacent pixels may be determined to belong to the same connected body. Planar fitting can be done by least squares.
  • the method may further include filtering out the noise communication body based on the detected area of the connected body, the average depth, and/or the relative position in the live image.
  • Correcting the detected plane may include: adjusting a tilted posture of the detected plane in a three-dimensional space to be in a state parallel to the display plane; and rotating the adjusted plane in a two-dimensional space corresponding to the display plane, Make the text in it flat.
  • Adjusting the tilting posture of the detected plane in the three-dimensional space may include: calculating a perspective transformation matrix that rotates the detected plane into a plane parallel to the display plane based on the plane parameters obtained by the plane fitting; and detecting and detecting the real image
  • the perspective transformation matrix is applied to the pixels of the corresponding portion of the plane.
  • the method may further include: before confirming the detected plane, the user confirms or selects the real scene plane detected in the live image.
  • the method may further include the user confirming or selecting the corrected plane before recognizing the characters in the corrected plane.
  • an apparatus for recognizing a character in a live view image comprising: image acquisition means for acquiring a real scene image and a depth map, each pixel representation and real scene in the depth map The distance between the real point of the image corresponding to the pixel in the image; the plane detecting device for detecting the real scene plane in the real scene image according to the depth map; and the plane correcting device for correcting the detected plane so that the plane is in the plane
  • the characters are easy to recognize; and a character recognition device for recognizing characters in the corrected plane.
  • the plane detecting device may include: a connected body detecting unit configured to be based on depth a depth difference between adjacent pixels in the figure, detecting a connected body in the real image; the plane fitting unit is configured to perform plane fitting by using the detected coordinates of each pixel of the connected body and the corresponding depth value; the noise filtering unit, a pixel configured to filter out the connected body outside the fitting plane; and a determining unit configured to determine a convex polygon surrounding the remaining pixels of the connected body. If the depth difference of the adjacent pixels is smaller than the threshold, the adjacent pixels may be determined to belong to the same connected body. Planar fitting can be done by least squares.
  • the noise filtering unit may be further configured to filter out the noise communication body based on the detected area of the connected body, the average depth, and/or the relative position in the live image.
  • the plane correcting device may include: a three-dimensional tilt correcting unit configured to adjust a tilt posture of the detected plane in a three-dimensional space to be in a state parallel to the display plane; and a two-dimensional skew correction unit configured to The adjusted plane is rotated in the two-dimensional space corresponding to the display plane, so that the text therein is displayed correctly.
  • the three-dimensional tilt correction unit may include: a perspective transformation matrix calculation subunit configured to calculate a perspective transformation matrix that rotates the detected plane into a plane parallel to the display plane based on the plane parameters obtained by the plane fitting; A matrix application sub-unit configured to apply the perspective transformation matrix to pixels of a portion of the live image corresponding to the detected plane.
  • the apparatus may further include: a real-area plane confirmation/selection means for the user to confirm or select the real-life plane detected in the real-life image.
  • the apparatus may further comprise: a correction plane confirmation/selection means for the user to confirm or select the corrected plane.
  • the live image may be a color image or a grayscale image.
  • FIG. 1 is a flow chart showing a method of recognizing characters in a live view image in accordance with the present invention
  • 2 and 3 respectively show examples of the obtained live image and depth map
  • FIG. 4 is a flow chart showing an example implementation of a plane detection operation in accordance with the present invention.
  • Figure 5 shows a schematic representation of a connected body obtained from the depth map shown in Figure 3;
  • Figure 6 shows a schematic effect after removing the noise connected body
  • Figure 7 shows a schematic representation of the detected planar area
  • FIG. 8 is a flow chart showing an example implementation of a screen correction operation in accordance with the present invention.
  • Figure 9 shows a schematic effect of three-dimensional tilt correction
  • 12 and 13 are schematic diagrams showing binarization processing and rotation processing of the image after three-dimensional tilt correction, respectively;
  • 14 to 16 are block diagrams showing an example structure of an apparatus for recognizing characters in a live view image according to the present invention.
  • FIG. 17 shows an example process of photographing a live view using a smart phone, applying a recognition method according to the present invention to a live view image, and translating the recognized Chinese text into English text.
  • the method 100 begins at step s110 by acquiring a live view image and a depth map, wherein each pixel in the depth map represents a real attraction to camera distance corresponding to a pixel in the live view image.
  • step s120 is performed to detect a real scene plane in the real scene image according to the depth map.
  • step s130 is performed to correct the detected plane so that the characters in the plane are easily recognized.
  • step s140 is performed to identify the characters in the corrected plane.
  • the user confirms or selects the real scene plane detected in the real scene image.
  • the user can confirm the real scene plane detected in all or part of the real scene image.
  • the user selects the real scene plane detected in all or part of the real scene image.
  • the user confirms or selects the corrected plane before performing step s140.
  • the user can confirm all or part of the corrected plane.
  • the user selects all or part of the corrected plane.
  • the depth map may be directly captured by a device such as Kinect, or may be generated by a stereo system by calculating a plurality of images taken by a plurality of cameras having fixed relative spatial positions.
  • a schematic representation of a live image and a depth map are given, respectively.
  • step s120 An example implementation of step s120 is further illustrated in FIG. As shown in the figure, the process starts in step s1201, and the connected body in the live image is detected as a candidate plane based on the depth difference of adjacent pixels in the depth map.
  • the entire depth map may be scanned, and adjacent pixels whose depth difference is smaller than the threshold are determined to belong to the same connected body, and adjacent pixels whose depth difference is greater than the threshold are determined as boundary pixels of different connected bodies.
  • Fig. 5 a schematic representation of a connected body obtained by the depth map shown in Fig. 3 is given.
  • the detected connected body may be filtered based on a plurality of features of the connected body (eg, based on the detected area of the connected body, the average depth, and/or the relative position in the real-life image), and the removal does not include Noise interconnects in the word plane, thereby avoiding unnecessary subsequent calculations.
  • a schematic effect after removing the noise communication body is given. In fact, removing the noise connected body not only saves a lot of computational overhead, but also removes a large amount of noise present in the image that may result in a lower text recognition rate as shown.
  • step s1202 may be performed to perform plane fitting by using the detected coordinates of each pixel of the connected body and the corresponding depth value, and calculating the characterization parameters of the plane.
  • various existing planar fitting methods can be employed including, but not limited to, least squares.
  • step s1203 is performed to determine pixels of the connected body that are outside the fitting plane as noise points to be filtered out.
  • step s1204 is performed to determine convex polygons surrounding the remaining pixels of the connected body to form a regular planar area.
  • Fig. 7 a schematic representation of a planar area obtained by performing the above operation on the connected body shown in Fig. 6 is given.
  • Figure 8 shows an example implementation of this operation. As shown, the operation can be subdivided into two steps of three-dimensional tilt correction s1301 and two-dimensional skew correction s1302.
  • the three-dimensional tilt correction s1301 is responsible for adjusting the tilt posture of the detected plane in a three-dimensional space so as to be in a state parallel to the display plane.
  • a point in the detected plane may be selected as a reference point to determine the Z value of the target plane.
  • a perspective transformation matrix that rotates the detected plane into a plane parallel to the display plane is calculated. Then, the perspective transformation matrix is applied to a portion of the pixels of the live image corresponding to the detected plane. 10 and 11 respectively show schematic views of the portion of the real image corresponding to the detected plane before and after the application of the three-dimensional tilt correction.
  • the text in the three-dimensional tilt-corrected image may be skewed, and it is still not possible to obtain a satisfactory recognition by directly identifying it.
  • the two-dimensional skew correction s1302 is applied before the recognition to rotate the three-dimensional tilt-corrected plane in the two-dimensional space corresponding to the display plane, so that the text therein is displayed flat.
  • the three-dimensional tilt-corrected image may be first binarized, and the features of the binarized image may be used to calculate a rotation angle required to correct the text therein. Then, the three-dimensional tilt-corrected image is rotated by a corresponding angle.
  • 12 and 13 are schematic views showing the binarization processing and the rotation processing of the three-dimensional tilt corrected image, respectively. As can be seen from Fig. 13, the corrected image obtained after the plane detection and the plane correction processing is substantially similar to the scanned image obtained by directly placing the paper into the scanner, and character recognition of the image ensures satisfactory recognition accuracy.
  • the present invention also proposes related devices. 14 to 16 show a schematic block diagram of such a device.
  • an apparatus 1400 for recognizing characters in a live view image includes an image acquisition device 1410, a plane detecting device 1420, a plane correcting device 1430, and a character recognizing device 1440.
  • the image acquisition device 1410 is configured to acquire a live image and a depth map, each pixel in the depth map representing a real attraction to camera distance corresponding to a pixel in the live image.
  • the plane detecting device 1420 is configured to detect a real scene plane in the real scene image according to the depth map.
  • the plane correcting means 1430 is for correcting the detected plane so that characters in the plane are easily recognized.
  • Character recognition means 1440 is for identifying characters in the corrected plane.
  • a real-area plane confirmation/selection device is provided for the user to confirm or select the real-life plane detected in the real-life image.
  • the live view confirming device can be used by the user to confirm the real scene plane detected in all or part of the real scene image.
  • the real plane selection device may be used by the user to select a real scene plane detected in all or part of the real scene image.
  • a correction plane confirmation/selection device may be included for the user to confirm or select the corrected plane.
  • the calibration plane confirmation device can be used by the user to confirm all or part of the corrected plane.
  • the correction plane selection means can be used by the user to select all or part of the corrected plane.
  • the plane detecting device 1420 may include a connected body detecting unit 1421, a plane fitting unit 1422, a noise filtering unit 1423, and a determining unit 1424.
  • the connected body detecting unit 1421 is configured to detect the connected body in the live view image based on the depth difference of the adjacent pixels in the depth map.
  • the plane fitting unit 1422 is configured to perform plane fitting using the coordinates of the detected pixels of the connected body and the corresponding depth values.
  • the noise filtering unit 1423 is configured to filter out pixels of the connected body that are outside the fitting plane.
  • the determining unit 1424 is configured to determine a convex polygon that surrounds the remaining pixels of the connected body.
  • the adjacent pixels may be determined to belong to the same connected body.
  • Planar fitting can be done by least squares.
  • the noise filtering unit may be further configured to filter out the noise communication body based on the detected area of the connected body, the average depth, and/or the relative position in the live image.
  • the plane correcting device 1430 may include: a three-dimensional tilt correcting unit 1431 configured to adjust a tilt posture of the detected plane in a three-dimensional space to be in a state parallel to the display plane; and two-dimensional The skew correcting unit 1432 is configured to rotate the adjusted plane in a two-dimensional space corresponding to the display plane to display the text therein.
  • the three-dimensional correction unit 1431 may include: a perspective transformation matrix calculation sub-unit configured to calculate a perspective transformation matrix that rotates the detected plane into a plane parallel to the display plane based on the plane parameters obtained by the plane fitting; and A transform matrix application sub-unit configured to apply the perspective transformation matrix to pixels of a portion of the live image corresponding to the detected plane.
  • the method and apparatus for recognizing characters in a live view image according to the present invention can be advantageously applied to a handheld device (such as a smart phone, a tablet computer, etc.) that is equipped with more than two cameras and is thus capable of reconstructing a 3D scene.
  • the user who purchased the handheld device can conveniently use the handheld device to take a real-life image during a foreign tour, and the real recognition map with a higher recognition degree.
  • Foreign characters in the picture for example, road signs, bulletin boards, billboards, etc. in a street scene
  • are identified and translated into Mandarin. 17 shows a process of photographing a live view image using a smart phone, applying a recognition method according to the present invention to a live view image, and translating the recognized Chinese text into English text.
  • the first step is to obtain a real scene image and a depth map by using a smart phone;
  • the second step is to detect a real scene plane in the real scene image according to the depth map, where the "dock" is taken as an example to detect the strip.
  • the third and fourth steps the corrected plane is corrected, wherein the third step is perspective transformation, the fourth step is tilt correction;
  • the fifth step is the corrected plane
  • the characters in the character are recognized to obtain the Chinese text of the "dock”; in the sixth step, the recognized Chinese text "dock” is translated into the English text "Pier".

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)
  • Character Input (AREA)

Abstract

公开了一种对实景图像中的字符进行识别的方法以及相应的设备。所述方法包括：获取实景图像和深度图，所述深度图中的每个像素表示与实景图像中的像素相对应的实景点到摄像机的距离；根据深度图，检测出实景图像中的实景平面；对检测到的平面进行校正，使得平面中的字符易于识别；以及对校正后的平面中的字符进行识别。采用所公开的方法和设备可以提高对实景图像中的字符进行识别的准确度。

Description

对实景图像中的字符进行识别的方法和设备 技术领域
本申请总体涉及光学字符识别OCR技术，具体涉及对实景图像中的字符进行识别的方法和设备。
背景技术
除了识别算法本身的性能以外，传统OCR技术对图像中字符的识别准确度还取决于待处理图像的复杂度、文字所在平面与图像呈现平面的平行度、以及图像中文字的平正程度。为了使识别准确度达到可接受的程度，通常要求待处理图像中仅包含一页文字，该页文字与图像呈现平面基本平行，且图像中的文字基本平正(即，非歪斜)摆放。
在实际应用中，用户可能希望对日常拍摄的实景图像中的文字(例如街拍场景中某一透视投影的标牌上的文字)进行光学识别，并可能希望对其进行文字处理。然而，由于日常拍摄的实景图像的构图通常较为复杂、实体对象较多、实景平面与图像呈现平面夹角较大(由于透视投影导致)、文字倾斜度较大，识别准确度难以达到令人满意的程度。
题为“Generating Hi-Res Dewarped Book Images”的美国专利申请US 20130242054 A1提出了一种对书本立体图像去除扭曲的方法。该方法利用三维几何表面(例如圆柱表面)产生3D信息，而无需使用文本行或其他内容分析。具体地，在捕捉到立体图像后，首先根据立体照片产生三维几何平面。然后，计算相应的模型。接着，由模型对立体图像去除扭曲。
虽然上述方案能够提高对书页中文字的识别准确度，还是难于处理具有复杂场景的实体图像。
发明内容
鉴于现有技术的上述问题和缺陷，本发明的目的在于提出一种能够以较高的识别准确度对实景图像中的字符进行识别的方案。
根据本发明的第一方面，提供了一种对实景图像中的字符进行识别的方法，包括：获取实景图像和深度图，所述深度图中的每个像素表示与实景图像中的像素相对应的实景点到摄像机的距离；根据深度图，检测出实景图像中的实景平面；对检测到的平面进行校正，使得平面中的字符易于识别；以及对校正后的平面中的字符进行识别。
根据深度图检测出实景图像中的实景平面可以包括：基于深度图中相邻像素的深度差，检测实景图像中的连通体；利用检测出的连通体各像素的坐标及对应的深度值，进行平面拟合；滤除连通体的位于拟合平面以外的像素；以及确定包围连通体的剩余像素的凸多边形。如果相邻像素的深度差小于阈值，则可以将相邻像素确定为属于同一连通体。平面拟合可以采用最小二乘法。
所述方法还可以包括：基于检测出的连通体的面积、平均深度和/或在实景图像中的相对位置，滤除噪声连通体。
对检测到的平面进行校正可以包括：在三维空间中调整检测到的平面的倾斜姿态，使其处于与显示平面平行的状态；以及在显示平面所对应的二维空间中旋转经调整的平面，使其中的文字平正显示。
在三维空间中调整检测到的平面的倾斜姿态可以包括：基于平面拟合得到的平面参数，计算出使检测到的平面旋转为与显示平面平行的平面的透视变换矩阵；对实景图像的与检测到的平面相对应的部分的像素应用所述透视变换矩阵。
所述方法还可以包括：对检测到的平面进行校正前，使用者确认或选择实景图像中检测出的实景平面。
所述方法还可以包括：对校正后的平面中的字符进行识别前，使用者确认或选择校正后的平面。
根据本发明的第二方面，提供了一种对实景图像中的字符进行识别的设备，包括：图像获取装置，用于获取实景图像和深度图，所述深度图中的每个像素表示与实景图像中的像素相对应的实景点到摄像机的距离；平面检测装置，用于根据深度图，检测出实景图像中的实景平面；平面校正装置，用于对检测到的平面进行校正，使得平面中的字符易于识别；以及字符识别装置，用于对校正后的平面中的字符进行识别。
所述平面检测装置可以包括：连通体检测单元，被配置为基于深度 图中相邻像素的深度差，检测实景图像中的连通体；平面拟合单元，被配置为利用检测出的连通体各像素的坐标及对应的深度值，进行平面拟合；噪声过滤单元，被配置为滤除连通体的位于拟合平面以外的像素；以及确定单元，被配置为确定包围连通体的剩余像素的凸多边形。如果相邻像素的深度差小于阈值，则可以将相邻像素确定为属于同一连通体。平面拟合可以采用最小二乘法。
所述噪声过滤单元还可以被配置为：基于检测出的连通体的面积、平均深度和/或在实景图像中的相对位置，滤除噪声连通体。
所述平面校正装置可以包括：三维倾斜校正单元，被配置为在三维空间中调整检测到的平面的倾斜姿态，使其处于与显示平面平行的状态；以及二维歪斜校正单元，被配置为在显示平面所对应的二维空间中旋转经调整的平面，使其中的文字平正显示。
所述三维倾斜校正单元可以包括：透视变换矩阵计算子单元，被配置为基于平面拟合得到的平面参数，计算出使检测到的平面旋转为与显示平面平行的平面的透视变换矩阵；透视变换矩阵应用子单元，被配置为对实景图像中与检测到的平面相对应的部分的像素应用所述透视变换矩阵。
所述的设备还可以包括：实景平面确认/选择装置，用于使用者确认或选择实景图像中检测出的实景平面。
所述设备还可以包括：校正平面确认/选择装置，用于使用者确认或选择校正后的平面。
所述实景图像可以是彩色图像或灰度图像。
附图说明
通过下面结合附图说明本发明的优选实施例，将使本发明的上述及其它目的、特征和优点更加清楚，其中：
图1是示出了根据本发明的对实景图像中的字符进行识别的方法的流程图；
图2和3分别示出了所获得的实景图像和深度图的示例；
图4是示出了根据本发明的平面检测操作的示例实现的流程图；
图5示出了由图3所示的深度图获得的连通体的示意表示；
图6示出了移除噪声连通体后的示意效果；
图7示出了检测出的平面区域的示意表示；
图8是示出了根据本发明的屏幕校正操作的示例实现的流程图；
图9示出了三维倾斜校正的示意效果；
图10和11分别给出了对实景图像的与检测到的平面相对应的部分应用三维倾斜校正前后的示意图；
图12和13分别给出了对三维倾斜校正后的图像进行二值化处理和旋转处理后的示意图；
图14～16示出了根据本发明的对实景图像中的字符进行识别的设备的示例结构的框图；以及
图17示出了使用智能电话拍摄实景图像、对实景图像应用根据本发明的识别方法、并将识别出的中文文本翻译成英文文本的示例过程。
具体实施方式
下面参照附图对本发明的优选实施例进行详细说明，在描述过程中省略了对于本发明来说是不必要的细节和功能，以防止对本发明的理解造成混淆。
首先，参照图1，描述根据本发明的对实景图像中的字符进行识别的方法100。如图1所示，方法100起始于步骤s110，获取实景图像和深度图，其中，深度图中的每个像素表示与实景图像中的像素相对应的实景点到摄像机的距离。接着，执行步骤s120，根据深度图，检测出实景图像中的实景平面。然后，执行步骤s130，对检测到的平面进行校正，使得平面中的字符易于识别。最后，执行步骤s140，对校正后的平面中的字符进行识别。
优选地，在执行步骤s130前，使用者确认或选择实景图像中检测出的实景平面。使用者可以确认全部或部分实景图像中检测出的实景平面。或者使用者选择全部或部分实景图像中检测出的实景平面。
优选地，在执行步骤s140前，使用者确认或选择校正后的平面。使用者可以确认全部或部分校正后的平面。或者使用者选择全部或部分校正后的平面。
通过在对实景图像中的字符进行识别之前先借助深度图检测出实景 图像中的实景平面并对其校正，可以极大地提高对实景图像中的字符进行识别的准确度。
在具体实现中，深度图可由Kinect等设备直接捕捉，也可由立体系统通过对具有固定的相对空间位置的多个摄像机所拍摄的多个图像进行计算来生成。在图2和3中，分别给出了实景图像和深度图的示意表示。
图4进一步给出了步骤s120的一种示例实现。如图所示，过程起始于步骤s1201，基于深度图中相邻像素的深度差，检测实景图像中的连通体，作为候选平面。在具体实现中，可以对整个深度图进行扫描，将深度差小于阈值的相邻像素确定为属于同一连通体，将深度差大于阈值的相邻像素确定为不同连通体的边界像素。在图5中，给出了由图3所示的深度图获得的连通体的示意表示。
优选地，可以基于连通体的多项特征(如基于检测出的连通体的面积、平均深度和/或在实景图像中的相对位置)，对检测出的连通体进行过滤，移除不包含有字平面的噪声连通体，进而避免非必要的后续计算。在图6中，给出了移除噪声连通体后的示意效果。事实上，移除噪声连通体不仅能够节省大量的计算开销，还能够如图所示地移除图像中存在的可能导致较低文字识别率的大量噪声。
由图6可见，即便对噪声连通体进行了滤除，剩余连通体仍具有与平面无关的噪声点(图6中的噪声点对应于与纸张平面相连的人手的一部分)。为消除这些噪点，可以执行步骤s1202，利用检测出的连通体各像素的坐标及对应的深度值，进行平面拟合，计算出平面的表征参数。为此，可以采用现有的各种平面拟合方法，包括但不限于最小二乘法。此后，执行步骤s1203，将连通体的位于拟合平面以外的像素确定为噪声点加以滤除。最后，执行步骤s1204，确定包围连通体的剩余像素的凸多边形，形成规则的平面区域。在图7中，给出了对图6所示的连通体执行上述操作后得到的平面区域的示意表示。
此后，可以进一步对检测到的平面进行校正，使得平面中的字符易于识别。图8给出了此操作的一种示例实现。如图所示，该操作可以细分为三维倾斜校正s1301以及二维歪斜校正s1302两个步骤。
三维倾斜校正s1301负责在三维空间中调整检测到的平面的倾斜姿态，使其处于与显示平面平行的状态。图9示出了三维倾斜校正的效果 示意。如图所示，通过校正，检测到的平面在三维空间中被旋转为与XOY(Z＝0)平行的目标平面，其中，(X，Y)表示像素在显示平面中的坐标，Z表示其深度。
在实际实现中，可以选择检测到的平面中的一点作为参考点，以此确定目标平面的Z值。基于平面拟合得到的平面参数，计算出使检测到的平面旋转为与显示平面平行的平面的透视变换矩阵。然后，对实景图像的与检测到的平面相对应的部分的像素应用所述透视变换矩阵。图10和11分别给出了对实景图像的与检测到的平面相对应的部分应用三维倾斜校正前后的示意图。
三维倾斜校正后的图像中的文本可能是歪斜的，直接对其进行识别仍无法获得令人满意的识别度。为对此进行改善，在识别前应用二维歪斜校正s1302，以在显示平面所对应的二维空间中旋转经三维倾斜校正的平面，使其中的文字平正显示。
在实际实现中，可以首先对三维倾斜校正后的图像进行二值化，并利用二值化图像的特征，计算使其中的文字平正显示所需的旋转角。然后，将三维倾斜校正后的图像旋转相应的角度。图12和13分别给出了对三维倾斜校正后的图像进行二值化处理和旋转处理后的示意图。由图13可见，经过平面检测和平面校正处理后得到的校正图像与将纸张直接放入扫描仪得到的扫描图像基本相近，对该图像进行文字识别能够确保令人满意的识别准确度。
与上述对实景图像中的字符进行识别的方法相对应地，本发明还提出了相关的设备。图14～16示出了这样的设备的示意结构方框图。
如图14所示，根据本发明的对实景图像中的字符进行识别的设备1400包括图像获取装置1410、平面检测装置1420、平面校正装置1430以及字符识别装置1440。图像获取装置1410用于获取实景图像和深度图，所述深度图中的每个像素表示与实景图像中的像素相对应的实景点到摄像机的距离。平面检测装置1420用于根据深度图，检测出实景图像中的实景平面。平面校正装置1430用于对检测到的平面进行校正，使得平面中的字符易于识别。字符识别装置1440用于对校正后的平面中的字符进行识别。
优选地，在平面检测装置1420和平面校正装置1430之间，可以包 括实景平面确认/选择装置，用于使用者确认或选择实景图像中检测出的实景平面。所述实景平面确认装置可以用于使用者确认全部或部分实景图像中检测出的实景平面。或者所述实景平面选择装置可以用于使用者选择全部或部分实景图像中检测出的实景平面。
优选地，在平面校正装置1430和字符识别装置1440之间，可以包括校正平面确认/选择装置，用于使用者确认或选择校正后的平面。所述校正平面确认装置可以用于使用者确认全部或部分校正后的平面。或者校正平面选择装置可以用于使用者选择全部或部分校正后的平面。
如图15所示，所述平面检测装置1420可以包括连通体检测单元1421、平面拟合单元1422、噪声过滤单元1423和确定单元1424。连通体检测单元1421被配置为基于深度图中相邻像素的深度差，检测实景图像中的连通体。平面拟合单元1422被配置为利用检测出的连通体各像素的坐标及对应的深度值，进行平面拟合。噪声过滤单元1423被配置为滤除连通体的位于拟合平面以外的像素。确定单元1424被配置为确定包围连通体的剩余像素的凸多边形。如果相邻像素的深度差小于阈值，则可以将相邻像素确定为属于同一连通体。平面拟合可以采用最小二乘法。所述噪声过滤单元还可以被配置为：基于检测出的连通体的面积、平均深度和/或在实景图像中的相对位置，滤除噪声连通体。
如图16所示，所述平面校正装置1430可以包括：三维倾斜校正单元1431，被配置为在三维空间中调整检测到的平面的倾斜姿态，使其处于与显示平面平行的状态；以及二维歪斜校正单元1432，被配置为在显示平面所对应的二维空间中旋转经调整的平面，使其中的文字平正显示。
所述三维校正单元1431可以包括：透视变换矩阵计算子单元，被配置为基于平面拟合得到的平面参数，计算出使检测到的平面旋转为与显示平面平行的平面的透视变换矩阵；以及透视变换矩阵应用子单元，被配置为对实景图像中与检测到的平面相对应的部分的像素应用所述透视变换矩阵。
根据本发明的对实景图像中的字符进行识别的方法和设备可以有利地应用于自身配备了两个以上的摄像机因而能够重建3D场景的手持设备(如智能电话、平板计算机等)。购买了该手持设备的用户在国外旅游期间可以方便地用手持设备拍摄实景图像，以较高的识别度对实景图 像中的外国文字(例如，街拍场景中透视投影的路标、布告栏、广告牌等上的文字)进行识别，并将其翻译成本国语。图17示出了使用智能电话拍摄实景图像、对实景图像应用根据本发明的识别方法、并将识别出的中文文本翻译成英文文本的过程。在图17中，第一步，使用智能电话拍摄获取实景图像和深度图；第二步，根据深度图，检测出实景图像中的实景平面，此处，以“码头”为例，检测出带有“码头”字符的实景平面；第三步和第四步，对检测到的平面进行校正，其中，第三步为透视变换，第四步为倾斜校正；第五步，对校正后的平面中的字符进行识别，得到“码头”的中文文本；第六步，将识别出的中文文本“码头”翻译成英文文本“Pier”。
应当注意的是，在以上的描述中，仅以示例的方式，示出了本发明的技术方案，但并不意味着本发明局限于上述步骤和单元结构。在可能的情形下，可以根据需要对步骤和单元结构进行调整和取舍。因此，某些步骤和单元并非实施本发明的总体发明思想所必需的元素。因此，本发明所必需的技术特征仅受限于能够实现本发明的总体发明思想的最低要求，而不受以上具体实例的限制。
至此已经结合优选实施例对本发明进行了描述。应该理解，本领域技术人员在不脱离本发明的精神和范围的情况下，可以进行各种其它的改变、替换和添加。因此，本发明的范围不局限于上述特定实施例，而应由所附权利要求所限定。

Claims (20)

  1. 一种对实景图像中的字符进行识别的方法，包括：
    获取实景图像和深度图，所述深度图中的每个像素表示与实景图像中的像素相对应的实景点到摄像机的距离；
    根据深度图，检测出实景图像中的实景平面；
    对检测到的平面进行校正，使得平面中的字符易于识别；以及
    对校正后的平面中的字符进行识别。
  2. 根据权利要求1所述的方法，其中，根据深度图检测出实景图像中的实景平面包括：
    基于深度图中相邻像素的深度差，检测实景图像中的连通体；
    利用检测出的连通体各像素的坐标及对应的深度值，进行平面拟合；
    滤除连通体的位于拟合平面以外的像素；以及
    确定包围连通体的剩余像素的凸多边形。
  3. 根据权利要求2所述的方法，其中，如果相邻像素的深度差小于阈值，则将相邻像素确定为属于同一连通体。
  4. 根据权利要求2所述的方法，其中，平面拟合采用最小二乘法。
  5. 根据权利要求2所述的方法，还包括：基于检测出的连通体的面积、平均深度和/或在实景图像中的相对位置，滤除噪声连通体。
  6. 根据权利要求1或2所述的方法，其中，对检测到的平面进行校正包括：
    在三维空间中调整检测到的平面的倾斜姿态，使其处于与显示平面平行的状态；以及
    在显示平面所对应的二维空间中旋转经调整的平面，使其中的文字平正显示。
  7. 根据权利要求6所述的方法，其中，在三维空间中调整检测到的平面的倾斜姿态包括：
    基于平面拟合得到的平面参数，计算出使检测到的平面旋转为与显示平面平行的平面的透视变换矩阵；
    对实景图像的与检测到的平面相对应的部分的像素应用所述透视变换矩阵。
  8. 根据权利要求1所述的方法，还包括：对检测到的平面进行校正前，使用者确认或选择实景图像中检测出的实景平面。
  9. 根据权利要求1所述的方法，还包括：对校正后的平面中的字符进行识别前，使用者确认或选择校正后的平面。
  10. 根据权利要求1或2所述的方法，其中，所述实景图像是彩色图像或灰度图像。
  11. 一种对实景图像中的字符进行识别的设备，包括：
    图像获取装置，用于获取实景图像和深度图，所述深度图中的每个像素表示与实景图像中的像素相对应的实景点到摄像机的距离；
    平面检测装置，用于根据深度图，检测出实景图像中的实景平面；
    平面校正装置，用于对检测到的平面进行校正，使得平面中的字符易于识别；以及
    字符识别装置，用于对校正后的平面中的字符进行识别。
  12. 根据权利要求11所述的设备，其中，所述平面检测装置包括：
    连通体检测单元，被配置为基于深度图中相邻像素的深度差，检测实景图像中的连通体；
    平面拟合单元，被配置为利用检测出的连通体各像素的坐标及对应的深度值，进行平面拟合；
    噪声过滤单元，被配置为滤除连通体的位于拟合平面以外的像素；以及
    确定单元，被配置为确定包围连通体的剩余像素的凸多边形。
  13. 根据权利要求12所述的设备，其中，如果相邻像素的深度差小于阈值，则将相邻像素确定为属于同一连通体。
  14. 根据权利要求12所述的设备，其中，平面拟合采用最小二乘法。
  15. 根据权利要求12所述的设备，其中，所述噪声过滤单元还被配置为：基于检测出的连通体的面积、平均深度和/或在实景图像中的相对位置，滤除噪声连通体。
  16. 根据权利要求11或12所述的设备，其中，所述平面校正装置包括：
    三维倾斜校正单元，被配置为在三维空间中调整检测到的平面的倾斜姿态，使其处于与显示平面平行的状态；以及
    二维歪斜校正单元，被配置为在显示平面所对应的二维空间中旋转经调整的平面，使其中的文字平正显示。
  17. 根据权利要求16所述的设备，其中，所述三维倾斜校正单元包括：
    透视变换矩阵计算子单元，被配置为基于平面拟合得到的平面参数，计算出使检测到的平面旋转为与显示平面平行的平面的透视变换矩阵；
    透视变换矩阵应用子单元，被配置为对实景图像中与检测到的平面相对应的部分的像素应用所述透视变换矩阵。
  18. 根据权利要求11所述的设备，还包括：实景平面确认/选择装置，用于使用者确认或选择实景图像中检测出的实景平面。
  19. 根据权利要求11所述的设备，还包括：校正平面确认/选择装置，用于使用者确认或选择校正后的平面。
  20. 根据权利要求11或12所述的设备，其中，所述实景图像是彩色图像或灰度图像。
PCT/CN2015/078254 2014-05-06 2015-05-05 对实景图像中的字符进行识别的方法和设备 Ceased WO2015169202A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201410187592.2A CN105095832A (zh) 2014-05-06 2014-05-06 对实景图像中的字符进行识别的方法和设备
CN201410187592.2 2014-05-06

Publications (1)

Publication Number Publication Date
WO2015169202A1 true WO2015169202A1 (zh) 2015-11-12

Family

ID=54392143

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/078254 Ceased WO2015169202A1 (zh) 2014-05-06 2015-05-05 对实景图像中的字符进行识别的方法和设备

Country Status (2)

Country Link
CN (1) CN105095832A (zh)
WO (1) WO2015169202A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003051005A (ja) * 2001-08-06 2003-02-21 Fuji Photo Film Co Ltd 画像処理装置、画像処理方法、取付部材、画像処理システム、及びプログラム
CN101625762A (zh) * 2009-06-19 2010-01-13 深圳市中瀛鑫科技发展有限公司 目标分割方法及装置
CN101887521A (zh) * 2010-06-22 2010-11-17 中兴通讯股份有限公司 一种实现文件纠偏的方法及终端
CN102609934A (zh) * 2011-12-22 2012-07-25 中国科学院自动化研究所 一种基于深度图像的多目标分割和跟踪方法
CN103065149A (zh) * 2012-12-21 2013-04-24 上海交通大学 网纹甜瓜果实表型提取与量化方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003051005A (ja) * 2001-08-06 2003-02-21 Fuji Photo Film Co Ltd 画像処理装置、画像処理方法、取付部材、画像処理システム、及びプログラム
CN101625762A (zh) * 2009-06-19 2010-01-13 深圳市中瀛鑫科技发展有限公司 目标分割方法及装置
CN101887521A (zh) * 2010-06-22 2010-11-17 中兴通讯股份有限公司 一种实现文件纠偏的方法及终端
CN102609934A (zh) * 2011-12-22 2012-07-25 中国科学院自动化研究所 一种基于深度图像的多目标分割和跟踪方法
CN103065149A (zh) * 2012-12-21 2013-04-24 上海交通大学 网纹甜瓜果实表型提取与量化方法

Also Published As

Publication number Publication date
CN105095832A (zh) 2015-11-25

Similar Documents

Publication Publication Date Title
CN105279372B (zh) 一种确定建筑物高度的方法和装置
US10798359B2 (en) Generating hi-res dewarped book images
JP6587421B2 (ja) 情報処理装置、情報処理方法、及びプログラム
CN107155341B (zh) 三维扫描系统和框架
CN107545592B (zh) 动态摄像机校准
KR101333871B1 (ko) 멀티-카메라 교정을 위한 방법 및 장치
TW201118791A (en) System and method for obtaining camera parameters from a plurality of images, and computer program products thereof
CN109429060B (zh) 瞳孔距离测量方法、可穿戴眼部设备及存储介质
CN107610041A (zh) 基于3d体感摄像头的视频人像抠图方法及系统
CN105005964B (zh) 基于视频序列影像的地理场景全景图快速生成方法
US20210295467A1 (en) Method for merging multiple images and post-processing of panorama
WO2017067262A1 (zh) 一种图像处理方法和装置
KR102200866B1 (ko) 2차원 이미지를 이용한 3차원 모델링 방법
US10303969B2 (en) Pose detection using depth camera
WO2016188010A1 (zh) 运动图像补偿方法及装置、显示装置
CN108961182B (zh) 针对视频图像的竖直方向灭点检测方法及视频扭正方法
CN110059537B (zh) 一种基于Kinect传感器的三维人脸数据获取方法及装置
CN107610078A (zh) 图像处理方法和装置
CN104636743A (zh) 文字图像校正的方法和装置
Cutter et al. Capture and dewarping of page spreads with a handheld compact 3D camera
CN106023170B (zh) 一种基于gpu处理器的双目3d畸变矫正方法
CN108830804B (zh) 基于线扩展函数标准差的虚实融合模糊一致性处理方法
CN112997217A (zh) 从视频图像进行文档检测
CN104933430B (zh) 一种用于移动终端的交互式图像处理方法及系统
WO2015169202A1 (zh) 对实景图像中的字符进行识别的方法和设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15789276

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15789276

Country of ref document: EP

Kind code of ref document: A1