KR102962877B1 - 오프셋 매핑을 사용한 워드브레이크 알고리즘 - Google Patents

오프셋 매핑을 사용한 워드브레이크 알고리즘

Info

Publication number
KR102962877B1
KR102962877B1 KR1020237040866A KR20237040866A KR102962877B1 KR 102962877 B1 KR102962877 B1 KR 102962877B1 KR 1020237040866 A KR1020237040866 A KR 1020237040866A KR 20237040866 A KR20237040866 A KR 20237040866A KR 102962877 B1 KR102962877 B1 KR 102962877B1
Authority
KR
South Korea
Prior art keywords
string
character
target
index value
offset index
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
KR1020237040866A
Other languages
English (en)
Korean (ko)
Other versions
KR20240011718A (ko
Inventor
마노즈 굽타
카빈 모트라니
Original Assignee
마이크로소프트 테크놀로지 라이센싱, 엘엘씨
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/444,347 external-priority patent/US11899698B2/en
Application filed by 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 filed Critical 마이크로소프트 테크놀로지 라이센싱, 엘엘씨
Publication of KR20240011718A publication Critical patent/KR20240011718A/ko
Application granted granted Critical
Publication of KR102962877B1 publication Critical patent/KR102962877B1/ko
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/284—Lexical analysis, e.g. tokenisation or collocates
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/103—Formatting, i.e. changing of presentation of documents
    • G06F40/109—Font handling; Temporal or kinetic typography
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/12—Use of codes for handling textual entities
    • G06F40/163—Handling of whitespace
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/166—Editing, e.g. inserting or deleting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G06F40/53—Processing of non-Latin text

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Artificial Intelligence (AREA)
  • Bioethics (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Machine Translation (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
KR1020237040866A 2021-05-28 2022-05-05 오프셋 매핑을 사용한 워드브레이크 알고리즘 Active KR102962877B1 (ko)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
IN202141023933 2021-05-28
IN202141023933 2021-05-28
US17/444,347 2021-08-03
US17/444,347 US11899698B2 (en) 2021-05-28 2021-08-03 Wordbreak algorithm with offset mapping
PCT/IB2022/000257 WO2022248933A1 (en) 2021-05-28 2022-05-05 Wordbreak algorithm with offset mapping

Publications (2)

Publication Number Publication Date
KR20240011718A KR20240011718A (ko) 2024-01-26
KR102962877B1 true KR102962877B1 (ko) 2026-05-12

Family

ID=82846495

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020237040866A Active KR102962877B1 (ko) 2021-05-28 2022-05-05 오프셋 매핑을 사용한 워드브레이크 알고리즘

Country Status (4)

Country Link
EP (1) EP4348490A1 (enExample)
JP (1) JP2024521833A (enExample)
KR (1) KR102962877B1 (enExample)
WO (1) WO2022248933A1 (enExample)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12669985B2 (en) * 2023-04-06 2026-06-30 Oracle International Corporation Compile-time checking for exhaustive switch statements and expressions
KR102994229B1 (ko) 2023-08-29 2026-07-22 고려대학교 산학협력단 문맥에 최적화된 한국어 토큰화 장치, 방법 및 이를 구현하기 위한 컴퓨터로 판독 가능한 저장 매체

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190286702A1 (en) 2018-03-13 2019-09-19 Fujitsu Limited Display control apparatus, display control method, and computer-readable recording medium
US20200334381A1 (en) 2019-04-16 2020-10-22 3M Innovative Properties Company Systems and methods for natural pseudonymization of text

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5963893A (en) * 1996-06-28 1999-10-05 Microsoft Corporation Identification of words in Japanese text by a computer system
US7747629B2 (en) * 2006-08-23 2010-06-29 International Business Machines Corporation System and method for positional representation of content for efficient indexing, search, retrieval, and compression
JP5833087B2 (ja) * 2013-11-13 2015-12-16 日本電信電話株式会社 文字列処理装置、方法、及びプログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190286702A1 (en) 2018-03-13 2019-09-19 Fujitsu Limited Display control apparatus, display control method, and computer-readable recording medium
US20200334381A1 (en) 2019-04-16 2020-10-22 3M Innovative Properties Company Systems and methods for natural pseudonymization of text

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"Fine-tuning with custom datasets - transformers 4.0.0 documentation", 3 December 2020

Also Published As

Publication number Publication date
KR20240011718A (ko) 2024-01-26
JP2024521833A (ja) 2024-06-04
EP4348490A1 (en) 2024-04-10
WO2022248933A1 (en) 2022-12-01

Similar Documents

Publication Publication Date Title
US10552462B1 (en) Systems and methods for tokenizing user-annotated names
US10303689B2 (en) Answering natural language table queries through semantic table representation
US9904673B2 (en) Conversation advisor
US10169471B2 (en) Generating and executing query language statements from natural language
TWI616762B (zh) 動態資料遮罩方法以及資料庫系統
US11409775B2 (en) Recommending documents sets based on a similar set of correlated features
US8875302B2 (en) Classification of an electronic document
US9971809B1 (en) Systems and methods for searching unstructured documents for structured data
CA2957674C (en) Testing insecure computing environments using random data sets generated from characterizations of real data sets
US10354078B2 (en) Multi-focused fine-grained security framework
US10169427B2 (en) Personalized highlighter for textual media
US11062129B2 (en) Systems and methods for enabling search services to highlight documents
WO2025239972A1 (en) Automated workflow creation using large language models
US20250165590A1 (en) Preventing attacks on generative models
US20130159346A1 (en) Combinatorial document matching
US20190171774A1 (en) Data filtering based on historical data analysis
CN121569287A (zh) 识别大型语言模型输出中的幻觉
KR20240011718A (ko) 오프셋 매핑을 사용한 워드브레이크 알고리즘
US11899698B2 (en) Wordbreak algorithm with offset mapping
US20160179954A1 (en) Systems and methods for culling search results in electronic discovery
US9852288B2 (en) Securing data on a computing system
US9483535B1 (en) Systems and methods for expanding search results
US9286348B2 (en) Dynamic search system
US11132400B2 (en) Data classification using probabilistic data structures
CN117396878A (zh) 带有偏移映射的分词算法

Legal Events

Date Code Title Description
PA0105 International application

St.27 status event code: A-0-1-A10-A15-nap-PA0105

PG1501 Laying open of application

St.27 status event code: A-1-1-Q10-Q12-nap-PG1501

A201 Request for examination
P11-X000 Amendment of application requested

St.27 status event code: A-2-2-P10-P11-nap-X000

D22 Grant of ip right intended

Free format text: ST27 STATUS EVENT CODE: A-1-2-D10-D22-EXM-PE0701 (AS PROVIDED BY THE NATIONAL OFFICE)

PE0701 Decision of registration

St.27 status event code: A-1-2-D10-D22-exm-PE0701

F11 Ip right granted following substantive examination

Free format text: ST27 STATUS EVENT CODE: A-2-4-F10-F11-EXM-PR0701 (AS PROVIDED BY THE NATIONAL OFFICE)

PR0701 Registration of establishment

St.27 status event code: A-2-4-F10-F11-exm-PR0701

PR1002 Payment of registration fee

St.27 status event code: A-2-2-U10-U12-oth-PR1002

Fee payment year number: 1

U12 Designation fee paid

Free format text: ST27 STATUS EVENT CODE: A-2-2-U10-U12-OTH-PR1002 (AS PROVIDED BY THE NATIONAL OFFICE)

Year of fee payment: 1

PG1601 Publication of registration

St.27 status event code: A-4-4-Q10-Q13-nap-PG1601

Q13 Ip right document published

Free format text: ST27 STATUS EVENT CODE: A-4-4-Q10-Q13-NAP-PG1601 (AS PROVIDED BY THE NATIONAL OFFICE)