CN106326243A - Data processing method and apparatus - Google Patents
Data processing method and apparatus Download PDFInfo
- Publication number
- CN106326243A CN106326243A CN201510345408.7A CN201510345408A CN106326243A CN 106326243 A CN106326243 A CN 106326243A CN 201510345408 A CN201510345408 A CN 201510345408A CN 106326243 A CN106326243 A CN 106326243A
- Authority
- CN
- China
- Prior art keywords
- data
- tables
- commodity
- major key
- lucene
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Embodiments of the invention disclose a data processing method and apparatus, and relate to the technical field of e-commerce. Through the method and the apparatus, the time consumption of a database in an index establishment process can be reduced, so that the computing resource consumption is reduced. The method comprises the steps of obtaining data tables which represent commodity information, and generating lucene files corresponding to the data tables which represent the commodity information according to data in the data tables; obtaining primary keys of the data tables, and reading data corresponding to the obtained primary keys from the generated lucene files according to the primary keys; and establishing a correlation among the generated lucene files according to the data corresponding to the primary keys, and submitting to a specified storage region. The method and the apparatus are suitable for quick retrieval creation.
Description
Technical field
The present invention relates to technical field of electronic commerce, particularly relate to a kind of data processing method and device.
Background technology
In recent years, e-commerce industry development is swift and violent, and the most of commodity on market the most can be by online flat
Platform is traded.In order to expand the trading range at line platform, strengthen transaction business ability, need relatively
High performance Database Systems manage merchandise news.In order to ensure the stable operation at line platform, major part
Electricity business is partial to the data base using framework ripe, such as relational database.
Owing to commodity usually contain a lot of attribute, such as: the base attribute such as brand name, category name,
Attribute is sold in stock, price, distribution state, sales region, attraction etc..In relational database, commodity
Each of attribute can store with the form of tables of data, for the different attribute data of same commodity, need
The incidence relation of different pieces of information table to be recorded, and set up index according to incidence relation, in order to client and electricity business
Can quick search merchandise news.
For the merchandise news of each commodity, data base is required for first associating and records each of item property data
Item data table, then inquired about the attribute data of these commodity by the incidence relation between several tables of data, the most again
Set up index.But in actual applications, owing to commodity amount is huge, data base must process the business of magnanimity
Product information, index preparation time-consuming the longest and to take calculating resource a lot.
Summary of the invention
Embodiments of the invention provide a kind of data processing method and device, it is possible to reduce data base and setting up rope
Spent time during drawing, thus reduce calculating resource consumption.
For reaching above-mentioned purpose, embodiments of the invention adopt the following technical scheme that
First aspect, a kind of data processing method of embodiments of the invention offer, data processing method, including:
Obtain the tables of data representing merchandise news, and generate respectively corresponding to described according to the data in tables of data
Represent the lucene file of each tables of data of merchandise news；
Obtain the major key of tables of data, and read corresponding to being obtained from the lucene file generated according to described major key
The data of the major key taken；
According to the data corresponding to described major key, set up the association between the lucene file generated, and to
Designated storage area is submitted to.
In conjunction with first aspect, in the first possible implementation of first aspect, described expression commodity are believed
The tables of data of breath includes: commodity master meter and the tables of data of expression item property, the number of described expression item property
At least include according to table: commodity base attribute table, commodity price information table and commodity stocks information table.
In conjunction with the first possible implementation of first aspect, in the implementation that the second is possible, institute
State the major key obtaining tables of data, and read corresponding to acquired from the lucene file generated according to described major key
The data of major key, including:
Obtain the commodity described in described commodity master meter uniquely to identify；
Inquiry obtains uniquely identifying described commodity the lucene file of the tables of data as major key, and from being looked into
Reading item property data in the lucene file ask, described item property data at least include: commodity base
This attribute data, commodity price data and commodity stocks data.
In conjunction with the implementation that the second of first aspect is possible, in the implementation that the third is possible, root
According to the data corresponding to described major key, set up the association between the lucene file generated, and to specifying storage
Region is submitted to, including:
Uniquely identify every item property data mart modeling according to described commodity and obtain association results；
Described association results is committed to solrcloud.
In conjunction with first aspect, in the 4th kind of possible implementation of first aspect, also include:
According to the inquiry request received, determine the major key associated with described inquiry request；
According to the major key associated with described inquiry request, read data from described designated storage area.
In conjunction with first to fourth kind of possible implementation of first aspect or first aspect, possible at the 5th kind
In implementation, also include:
When incremental data occurs in the tables of data described expression merchandise news being detected, obtain described incremental data
The major key of place tables of data；
The data of tables of data determined by reading, and be updated to described determined by the lucene file of tables of data.
In conjunction with first to fourth kind of possible implementation of first aspect or first aspect, possible at the 6th kind
In implementation, also include:
When buffer queue occurs incremental data, determine the tables of data associated with described incremental data；
And described incremental data is updated to the lucene file of the tables of data associated with described incremental data.
Second aspect, embodiments of the invention provide a kind of data processing equipment, including: pretreatment module,
For obtaining the tables of data representing merchandise news, and generate respectively corresponding to described according to the data in tables of data
Represent the lucene file of each tables of data of merchandise news；
Extraction module, for obtaining the major key of tables of data, and according to described major key from the lucene file generated
Read the data corresponding to acquired major key；
Analyze module, for obtaining according to the data corresponding to described major key, set up the lucene literary composition generated
Association between part, and submit to designated storage area.
In conjunction with second aspect, in the first possible implementation of second aspect, described expression commodity are believed
The tables of data of breath includes: commodity master meter and the tables of data of expression item property, the number of described expression item property
At least include according to table: commodity base attribute table, commodity price information table and commodity stocks information table.
In conjunction with the first possible implementation of second aspect, in the implementation that the second is possible, institute
State extraction module, uniquely identify specifically for obtaining the commodity described in described commodity master meter；
Inquiry obtains uniquely identifying described commodity the lucene file of the tables of data as major key, and from being looked into
Reading item property data in the lucene file ask, described item property data at least include: commodity base
This attribute data, commodity price data and commodity stocks data.
In conjunction with the implementation that the second of second aspect is possible, in the implementation that the third is possible, institute
State analysis module, specifically for uniquely identifying every item property data mart modeling according to described commodity and obtaining
Association results；And described association results is committed to solrcloud.
In conjunction with second aspect, in the 4th kind of possible implementation of second aspect, also include: inquiry mould
Block, for according to the inquiry request received, determining the major key associated with described inquiry request；And according to
The major key of described inquiry request association, reads data from described designated storage area.
In conjunction with first to fourth kind of possible implementation of second aspect or second aspect, possible at the 5th kind
In implementation, also include the first more new module, for when the tables of data described expression merchandise news being detected
When incremental data occurs, obtain the major key of described incremental data place tables of data；Tables of data determined by reading
Data, and be updated to described determined by the lucene file of tables of data.
In conjunction with first to fourth kind of possible implementation of second aspect or second aspect, possible at the 6th kind
In implementation, also include the second more new module, for when buffer queue occurs incremental data, determine
The tables of data associated with described incremental data；And be updated to described incremental data associate with described incremental data
The lucene file of tables of data.
The data processing method of embodiment of the present invention offer and device, it is possible to tables of data in existing database is turned
Turn to corresponding lucene file, from lucene file, obtain merchandise news according still further to the major key in commodity master meter,
For use in user search.The present invention is when processing magnanimity commodity, and major key and commodity by commodity master meter are believed
Breath generates for the lucene file of user search and association results, it is to avoid for the association of each commodity
Relation sets up the mode of index, reduces index preparation spent time, thus reduces and calculate taking of resource.
Accompanying drawing explanation
For the technical scheme being illustrated more clearly that in the embodiment of the present invention, below will be to required in embodiment
The accompanying drawing used is briefly described, it should be apparent that, the accompanying drawing in describing below is only the one of the present invention
A little embodiments, for those of ordinary skill in the art, on the premise of not paying creative work, also
Other accompanying drawing can be obtained according to these accompanying drawings.
The structural representation of the arithmetic facility that Fig. 1 provides for the embodiment of the present invention；
The flow chart of the data processing method that Fig. 2 provides for the embodiment of the present invention；
The schematic diagram of the instantiation that Fig. 3 provides for the embodiment of the present invention；
The system architecture schematic diagram of the instantiation that Fig. 4 a-4d provides for the embodiment of the present invention；
The flow chart of the another kind of data processing method that Fig. 5 provides for the embodiment of the present invention；
The flow chart of another data processing method that Fig. 6 a-6b provides for the embodiment of the present invention；
The structural representation of the data processing equipment that Fig. 7 provides for the embodiment of the present invention.
Detailed description of the invention
For making those skilled in the art be more fully understood that technical scheme, below in conjunction with the accompanying drawings and specifically
The present invention is described in further detail by embodiment.Embodiments of the present invention are described in more detail below,
The example of described embodiment is shown in the drawings, and the most same or similar label represents identical
Or similar element or there is the element of same or like function.The embodiment party described below with reference to accompanying drawing
Formula is exemplary, is only used for explaining the present invention, and is not construed as limiting the claims.
Those skilled in the art of the present technique are appreciated that unless expressly stated, singulative used herein " ",
" one ", " described " and " being somebody's turn to do " may also comprise plural form.It is to be further understood that the present invention
Description in use wording " include " referring to there is described feature, integer, step, operation, element
And/or assembly, but it is not excluded that existence or add other features one or more, integer, step, operation,
Element, assembly and/or their group.It should be understood that when we claim element to be " connected " or " coupled " to
During another element, it can be directly connected or coupled to other elements, or can also there is intermediary element.
Additionally, " connection " used herein or " coupling " can include wireless connections or couple.Used herein
Wording "and/or" includes one or more any cell listing item being associated and all combinations.
Those skilled in the art of the present technique are appreciated that unless otherwise defined, all terms used herein (bag
Include technical term and scientific terminology) have and being commonly understood by of the those of ordinary skill in art of the present invention
Identical meaning.Should also be understood that those terms defined in such as general dictionary should be understood that
There is the meaning consistent with the meaning in the context of prior art, and unless defined as here, no
Meeting idealization or the most formal implication are explained.
In the following description, by the intelligent terminal to include touch control display as embodiment, on its display
Being configured with can touch interface.In the following detailed description, many details are illustrated to provide the present invention
Understand in depth.But, the present invention may be carried out for ability in the case of not having these details
The those of ordinary skill in territory will be apparent from.In other cases, it is thus well known that method, code,
Parts, circuit and network are not described in detail in order to avoid unnecessarily obscuring the various aspects of embodiment.
Disclosed data processing method can be performed as executive agent by a single equipment,
Or be integrated in various different arithmetic facility, such as server, work station, supercomputer etc..Such as Fig. 1
It show the arithmetic facility according to one detailed description of the invention of the present invention.Described arithmetic facility include input block,
The assemblies such as processor unit, communication unit, memory element, peripheral unit.These assemblies pass through one or many
Bar bus communicates.It will be understood by those skilled in the art that the structure of the arithmetic facility shown in figure not
Constituting limitation of the invention, it both can be busbar network, it is also possible to be hub-and-spoke configuration, it is also possible to bag
Include ratio and illustrate more or less of parts, or combine some parts, or different parts are arranged.
Input block is input in arithmetic facility with the mutual of arithmetic facility and/or information for realizing user.Example
As, input block can receive numeral or the character information of user's input, to produce and user setup or function
Control relevant signal input.In the specific embodiment of the invention, input block can be contact panel,
Can also be that other people machine interactive interface, such as entity input key, mike etc., can also be other outside letters
Breath capture device, such as, image first-class.
Processor unit is the control centre of arithmetic facility, utilizes various interface and the whole computing of connection to set
Standby various piece, by running or perform software program and/or the module being stored in memory element, and
Call the data being stored in memory element, to perform the various functions of arithmetic facility and/or to process data.Institute
Stating processor unit can be the most permissible by integrated circuit (Integrated Circuit is called for short IC) composition
It is made up of the IC of single encapsulation, it is also possible to the group by many identical functions of connection or the encapsulation IC of difference in functionality
Become.For example, processor unit can only include central processing unit (Central Processing Unit,
It is called for short CPU), it is also possible to be GPU, (Digital Signal Processor is called for short digital signal processor
The combination of the control chip (such as baseband chip) DSP) and in communication unit.In embodiment of the present invention
In, CPU can be single arithmetic core, it is also possible to include multioperation core.
Described communication unit is used for setting up communication channel, makes arithmetic facility by described communication channel to be connected to
The data base of store commodity information, and media data from the remote server of data base.Owing to needs process
The merchandise news of magnanimity, should use the most in a preferred approach the communication mode possessing higher transmission bandwidth with
And with the communication unit of this communication mode respective type.Such as: the optical fiber network interface card of the interfaces such as LC, SC, SFP.
Described communication unit can also include that (Wireless Local Area Network is called for short WLAN
Wireless LAN) communication module such as module, bluetooth module, base band (Base Band) module, Yi Jisuo
State radio frequency corresponding to communication module (Radio Frequency is called for short RF) circuit, be used for carrying out wireless local
Network service, Bluetooth communication, infrared communication and/or cellular communications system communication, such as wideband code division are many
Re-access (Wideband Code Division Multiple Access is called for short W-CDMA) and/or high speed
Downlink Packet Access (High Speed Downlink Packet Access is called for short HSDPA).Described communication
Module is for controlling the communication of each assembly in arithmetic facility, and can support direct memory access (Direct
Memory Access)。
Memory element can be used for storing software program and module, and it is single that processing unit is stored in storage by operation
The software program of unit and module, thus perform the various functions application of arithmetic facility and realize data process.
Memory element mainly includes program storage area and data storage area, and wherein, program storage area can store operation system
System, application program needed at least one function, such as data submit to download, DAP etc.
Deng；Data storage area can store from the merchandise news of server download of data base, tables of data etc..At this
In bright detailed description of the invention, memory element can include volatile memory, the most non-volatile dynamic random
Access memory (Nonvolatile Random Access Memory is called for short NVRAM), phase change are deposited at random
Take internal memory (Phase Change RAM is called for short PRAM), magnetic-resistance random access memory (Magetoresistive
RAM, is called for short MRAM) etc., it is also possible to include nonvolatile memory, for example, at least one disk memory,
Electrical Erasable programmable read only memory (Electrically Erasable Programmable
Read-Only Memory, is called for short EEPROM), flush memory device, such as anti-or flash memory (NOR flash memory)
Or anti-and flash memory (NAND flash memory).Nonvolatile storage stores the operation performed by processing unit
System and application program.Described processing unit from described nonvolatile storage load operating program with data to interior
Deposit and digital content is stored in mass storage.Described operating system includes for controlling and often managing
Rule system task, such as memory management, storage device control, power management etc., and contribute to various soft
The various assemblies communicated between hardware and/or driver.
It should be noted that can also by External memory equipment as arithmetic facility memory element or with deposit
Storage unit combines the data for storing arithmetic facility, such as: by communication unit and distributed storage device
Carrying out data interaction, pending merchandise news download obtained is stored in distributed storage device, and
Waiting in line processor unit uses the data processing method of embodiment of the present invention offer process and obtain
Association results between lucene file and each lucene file.Or, it is also possible to by communication unit by number
It is stored in distributed storage device according to the intermediate data in processing procedure or ephemeral data.Described in the present embodiment
Lucene file obtain based on lucene, Lucene refer to apache software foundation issue one open
Put the full-text search engine tool kit of source code, this tool kit framework based on a full-text search engine, carry
Complete query engine and index engine, and part text analyzing engine are supplied.
Power supply is for being powered maintaining it to run to the different parts of arithmetic facility.Understand as generality,
Described power supply can be built-in battery, the most common lithium ion battery, Ni-MH battery etc., also includes straight
Connect the external power supply powered to arithmetic facility, such as AC adapter etc..In certain embodiments of the present invention,
Described power supply can also be made to define the most widely, such as can also include power-supply management system, charging system,
Power failure detection circuit, power supply changeover device or inverter, power supply status indicator (such as light emitting diode),
And generate with the electric energy of arithmetic facility, manage and be distributed other any assemblies being associated.
The embodiment of the present invention provides a kind of data processing method as shown in Figure 2, including:
101, obtain represent merchandise news tables of data, and according to the data in tables of data generate respectively corresponding to
The lucene file of each tables of data of described expression merchandise news.
Wherein, arithmetic facility can read the tables of data for representing merchandise news from data base, such as:
Directly can read tables of data from the data base of online transaction platform；Can also be from the data backup of data base
System reads tables of data, to avoid increasing the transmission pressure of data base.Wherein, if arithmetic facility is one
Individually equipment, then can be by the communication unit of arithmetic facility, by wired or wireless network from data base
Or data backup system reads tables of data.If arithmetic facility is integrated in the service of data base or data backup system
In device, then can read from the storage device of the storage device of data base or data backup system.
In the present embodiment, represent that the tables of data of merchandise news may include that commodity master meter and represents that commodity belong to
The tables of data of property.Wherein, commodity master meter at least includes commodity uniquely identify, the data such as affiliated businessman.Number
According to the identification code automatically distributed for registered commodity in storehouse, the identification code of each commodity and other business arbitrary
Product are different, and this kind of identification code is properly termed as commodity and uniquely identifies；Or can also by trade name, model,
The combination of the data such as type information uniquely identifies as commodity.Represent that the tables of data of item property at least includes:
Commodity base attribute table, commodity price information table and commodity stocks information table.It should be noted that in this reality
Execute in example, the data of data that will have recorded for describing the aspects such as the character of commodity, form, sales situation
Table is referred to as the tables of data representing item property, such as: above-mentioned commodity base attribute table, commodity price information
Table and commodity stocks information table, it is not limited to this.Commodity base attribute table specifically can include commodity
The data such as the place of production, date of manufacture, Time To Market (time that this type product puts goods on the market).
For representing each tables of data of merchandise news, arithmetic facility generates and corresponds respectively to each tables of data
Lucene file, such as: corresponding to commodity base attribute table commodity base attribute lucene file, correspond to
The commodity price information lucene file of commodity price information table and the commodity storehouse corresponding to commodity stocks information table
Deposit information lucene file etc..As a example by commodity price information table: arithmetic facility can first read commodity price letter
Data in breath table, and directly obtain or by data being converted the text message obtaining representing commodity price.
And utilize software kit based on Lucene be indexed according to text message and create index file, as commodity
The lucene file of pricing information table.The lucene literary composition of the obtained each tables of data corresponding to representing merchandise news
Part can be stored in the memorizer of arithmetic facility, or is committed to the dedicated memory space opened up in data base.
102, obtain the major key of tables of data, and from the lucene file generated, read correspondence according to described major key
Data in acquired major key.
The major key of tables of data is the field in tables of data with uniqueness, i.e. in tables of data any two record not
There is identical major key field.It should be understood that arithmetic facility should be from obtaining the major key obtained tables of data
The set being made up of a plurality of major key field.In the present embodiment, commodity uniquely can be identified as major key,
Then what arithmetic facility obtained is the set of unique identification field of all commodity in commodity master meter.
In the present embodiment, the data in each tables of data are separately added into each tables of data with the form of text message
Lucene file in, uniquely identify as a example by major key using commodity: the major key of tables of data is commodity 1 to commodity
The set of unique identification field of 10, then the data corresponding to major key in commodity base attribute lucene file are:
The text message of the commodity base attributes such as the place of production of commodity 1-10, date of manufacture, Time To Market；Commodity price is believed
In breath lucene file, the data corresponding to major key are: represent the text message of commodity 1-10 price；Commodity stocks
In information lucene file, the data corresponding to major key are: represent commodity 1-10 stockpile number, stock locations, storehouse
Deposit the text message of the aspects such as time.
103, according to the data corresponding to described major key, set up the association between the lucene file generated,
And submit to designated storage area.
Wherein, the data that arithmetic facility is submitted to designated storage area include: by the data representing merchandise news
Table processes the lucene file obtained, and the incidence relation between lucene file.
Incidence relation between lucene file can be expressed as lucene file respective mark character (such as:
The ID of master meter can be processed, as this, the mark character of lucene file, the lucene file generated obtained by table
Time numbering etc.) with the set of the character of major key.The character of lucene file respective mark character and major key
Set can be used as the association results in the present embodiment.It should be noted that association results can also include
Other merchandise newss such as commodity base attribute, price and stock, in order to can obtain according to association results is quick
To some merchandise news.
In the present embodiment, uniquely identify as a example by major key using commodity: arithmetic facility is by commodity 1-commodity 10
The set of unique identification field, and represent commodity base attribute, price and the stock of commodity 1-10 respectively
Text message is incorporated as association results, using the ID of commodity master meter as file identification, in order to retrieval phase is led to
Cross ID and retrieve commodity shown in master meter.
The data processing method that the embodiment of the present invention provides, it is possible to tables of data in existing database is converted into phase
The lucene file answered, obtains merchandise news according still further to the major key in commodity master meter from lucene file, in order to
For user search.The present invention is when processing magnanimity commodity, and major key and merchandise news by commodity master meter are raw
Become for the lucene file of user search and association results, it is to avoid for the incidence relation of each commodity
Set up the mode of index, reduce index preparation spent time, thus reduce and calculate taking of resource.
As a kind of implementation, specifically can be implemented as of step 102:
1011, obtain the commodity described in described commodity master meter and uniquely identify.
Wherein, commodity uniquely identify can be data base when merchandise news is write tables of data, the business of distribution
Product are numbered.As it is shown on figure 3, recorded goods number and affiliated Merchant name in commodity master meter, wherein commodity are compiled
Number there is in master meter uniqueness, can be as major key.
1012, inquire about and obtain uniquely identifying described commodity the lucene file of the tables of data as major key, and from
The lucene file inquired reads item property data.
Wherein, item property data at least include: commodity basic attribute data, commodity price data and commodity
Inventory data.As a example by as shown in Figure 3: commodity base attribute lucene file, commodity price information lucene
File and commodity stocks information lucene file be in step 101 arithmetic facility respectively according to commodity base attribute table
, commodity price information table and commodity stocks information table obtains, and at commodity base attribute table, business
Product pricing information table and commodity stocks information table in, goods number is major key, the most above-mentioned 3 tables of data and business
Product master meter has identical major key, arithmetic facility read respectively from above-mentioned 3 lucene files commodity base attribute,
Price and inventory data.
The major key of commodity master meter is present in data base, and arithmetic facility is without resetting the knowledge of lucene file
Other information, and in actual applications, lucene file has the advantages that reading speed is fast, utilizes commodity master meter
Major key determine lucene file and therefrom read item property data, further reduce processing and obtain lucene
File and association results required time.
As a kind of implementation, specifically can be implemented as of step 103: according to described commodity uniquely identify by
Every item property data mart modeling also obtains association results；Described association results is committed to solrcloud.
In the preferred version of the present embodiment, after generating lucene file and association results, it is also possible to will
Association results submit to solrCloud, wherein, after SolrCloud refers to Solr4.0 version, Solr based on
Zookeeper deployment way, Solr is that a kind of Java5 of employing develops, and full text based on Lucene is searched
Rope server, ZooKeeper is a kind of distributed, the distributed application program coordination service skill of open source code
Art.As shown in fig. 4 a, the data base of scene be integrated in to(for) arithmetic facility, arithmetic facility can will obtain
Lucene file and association results transmit the dedicated memory space opened up to other storage devices, in order to
After data base receives the retrieval request of commodity, arithmetic facility or data base read institute from other storage devices
Look into lucene file and the association results of commodity, and to user return the commodity base attribute about these commodity,
The information such as price and stock.Wherein, other storage devices may include that cloud disk, shared storage device,
Distributed memory system and other can support arithmetic facility or data base carry out reading and writing data storage device or
Storage system.
Optionally, as shown in Figure 4 b, lucene file and association results can also store to arithmetic facility
In memorizer, retrieval request is being transmitted to arithmetic facility by data base, and arithmetic facility returns lucene to data base
File and association results have the merchandise news of underlying commodity, thus reduces the transmission pressure of data base, need
Illustrate, the scene shown in Fig. 4 b can be passed through between arithmetic facility and data base data transmission interface
Being connected, arithmetic facility is sent a request for for being processed into lucene file and pass to data base by interface
The original list item (i.e. representing the tables of data of merchandise news) of connection relation, data base querying obtains arithmetic facility is asked
Seek the original list item of acquisition and be sent to arithmetic facility by this interface；As illustrated in fig. 4 c, arithmetic facility is simultaneously
Also the retrieval request of user terminal transmission be can directly receive and lucene file and association returned to user terminal
Result has the merchandise news of underlying commodity, thus reduces the transmission pressure of data base further, need explanation
It is, it should be noted that can directly carry out between arithmetic facility and data base in the scene shown in Fig. 4 c
The storage address of each original list item in storage data base in data read-write operation, i.e. arithmetic facility, thus computing
Equipment directly reads original list item according to storage address from data base, and is processed into lucene file and association
Relation；As shown in figure 4d, association results can also be committed to solrCloud by arithmetic facility, in order to user is eventually
Hold and retrieve merchandise news by solrCloud, thus utilize existing solrCloud, reduce data base and arithmetic facility
Transmission pressure.Association results is committed to solrCloud by arithmetic facility, can be deposited by lucene file simultaneously
Storage is in arithmetic facility, or submission as shown in figure 4d being stored in other storage devices.Generating
After lucene file and association results, it is also possible to include the searching step of merchandise news, as it is shown in figure 5,
Can also include:
104, according to the inquiry request received, determine the major key associated with described inquiry request.
In the present embodiment, major key is that the commodity in commodity master meter uniquely indicate (such as: goods number), uses
The inquiry request that family terminal is sent is for inquiring about the attribute information of a certain commodity, and inquiry request includes business
The information such as the name of an article claims, affiliated businessman.Perform the equipment (such as: arithmetic facility) of searching step according to inquiry
Request determines unique sign of required inquiry commodity, as the major key associated with inquiry request.
105, according to the major key associated with described inquiry request, read data from described designated storage area.
In the present embodiment, lucene file and association results include major key and the data corresponding to major key.Perform
The equipment of searching step obtains the lucene file corresponding with the major key of described inquiry request association and closes connection
Really, specifically can be implemented as: detect in the major key of lucene file and the association results stored, if deposit
Determining that by inquiry request the commodity obtained uniquely indicate, if existing, then this lucene file and association results
For the lucene file corresponding with the major key that described inquiry request associates and association results.
It should be noted that the equipment performing searching step in the present embodiment should be storage lucene file and
The equipment of association results, in conjunction with scheme as shown in figures 4a-4d, the equipment performing searching step in Fig. 4 a is number
According to storehouse, the equipment performing searching step in Fig. 4 b, Fig. 4 c is arithmetic facility, performs searching step in Fig. 4 d
Equipment is solrCloud.Lucene file and association results can be stored in the equipment beyond data base, subtract
Lack data base's transmission pressure during merchandise query.
After generating lucene file and association results, it is also possible to include the incremental update mistake of merchandise news
Journey, as shown in Figure 6 a, it is also possible to including:
106a, when incremental data occurs in the tables of data described expression merchandise news being detected, obtains described increasing
The major key of amount data place tables of data.
As a example by goods number is as major key: incremental data occurs in commodity price information table, arithmetic facility obtains business
The major key of product pricing information table, and inquire about there is the commodity base attribute table of identical major key, commodity stocks information
Table and commodity master meter.It should be noted that major key acquired here can include all of business in tables of data
Product are numbered, it is also possible to only include the goods number corresponding to incremental data.
107a, the data of tables of data determined by reading, and be updated to described determined by the lucene of tables of data
File.
Arithmetic facility regenerate determined by the lucene file of tables of data, in order to replace former lucene file；
Or according to incremental data, generate the code information meeting lucene file, and tables of data determined by being updated to
Lucene file.If lucene file and association results are stored in solrCloud, then arithmetic facility updates
Lucene file, and according to incremental data atomic update to solrCloud.
According to the difference of concrete applied environment, the incremental update process in the embodiment of the present invention, it is also possible to realize
Flow process for as shown in Figure 6 b:
106b, when occurring incremental data in buffer queue, determines the tables of data associated with described incremental data.
107b, is updated to the lucene file of the tables of data associated with described incremental data by described incremental data.
Arithmetic facility also can detect in buffer queue whether incremental data occur in real time, if incremental data occurs,
Then atomic update extremely stores the equipment of lucene file, and the solrCloud of association results.
After generating lucene file and association results by above-mentioned data processing method, if increment number occurs
According to, it is possible to synchronized update to lucene file and lucene file and association results, it is to avoid weight in prior art
The data refresh mode that newly-built lithol draws, reduces and updates index spent time, thus reduce data updating process
Taking of middle calculating resource.
The embodiment of the present invention also provides for a kind of data processing equipment 70 as shown in Figure 7, including:
Pretreatment module 71, for obtaining the tables of data representing merchandise news, and according to the data in tables of data
Generate the lucene file of each tables of data corresponding to described expression merchandise news respectively.
Extraction module 72, for obtaining the major key of tables of data, and according to described major key from the lucene file generated
Middle reading is corresponding to the data of acquired major key.
Analyze module 73, for obtaining according to the data corresponding to described major key, set up the lucene literary composition generated
Association between part, and submit to designated storage area.
Wherein, the tables of data of described expression merchandise news includes: commodity master meter and the data of expression item property
Table, the tables of data of described expression item property at least includes: commodity base attribute table, commodity price information table
With commodity stocks information table.
Further, described extraction module 72, specifically for obtaining the commodity described in described commodity master meter only
One mark.Inquiry obtains uniquely identifying described commodity the lucene file of the tables of data as major key, and from institute
Reading item property data in the lucene file inquired, described item property data at least include: commodity base
This attribute data, commodity price data and commodity stocks data.
Described analysis module 73, uniquely identifies by every item property data mart modeling also with specific reference to described commodity
Obtain association results；And described association results is committed to solrcloud.
Further, device 70 also includes: enquiry module 75, the inquiry request received for basis, really
The fixed major key associated with described inquiry request；And according to the major key associated with described inquiry request, from described finger
Determine memory area and read data.
Further, device 70 also includes the first more new module 76, for when detecting that described expression commodity are believed
When incremental data occurs in the tables of data of breath, obtain the major key of described incremental data place tables of data；Read institute really
The data of fixed tables of data, and be updated to described determined by the lucene file of tables of data.
Optionally, device 70 also includes the second more new module 77, for when occurring incremental data in buffer queue
Time, determine the tables of data associated with described incremental data；And described incremental data is updated to and described increment
The lucene file of the tables of data of data association.
The data processing equipment that the embodiment of the present invention provides, it is possible to tables of data in existing database is converted into phase
The lucene file answered, obtains merchandise news according still further to the major key in commodity master meter from lucene file, in order to
For user search.The present invention is when processing magnanimity commodity, and major key and merchandise news by commodity master meter are raw
Become for the lucene file of user search and association results, it is to avoid for the incidence relation of each commodity
Set up the mode of index, reduce index preparation spent time, thus reduce and calculate taking of resource.
Those skilled in the art of the present technique are appreciated that the present invention can relate to perform behaviour described herein
The equipment of the one or more operation in work.Described equipment can be required purpose and specialized designs and system
Make, or can also include that the known device in general purpose computer, described general purpose computer have storage within it
Procedure Selection ground activate or reconstruct.Such computer program can be stored in equipment (such as, meter
Calculation machine) in computer-readable recording medium or be stored in and be suitable to store e-command and be coupled to any type of bus respectively
Medium in, described computer-readable medium include but not limited to any kind of dish (include floppy disk, hard disk,
CD, CD-ROM and magneto-optic disk), memorizer (RAM), read only memory (ROM), electrically programmable immediately
ROM, electrically erasable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, magnetic card or
Light card.Computer-readable recording medium includes for be stored by the form that equipment (such as, computer) is readable or to be passed
Any mechanism of transmission information.Such as, computer-readable recording medium include immediately memorizer (RAM), read only memory (ROM),
Magnetic disk storage medium, optical storage medium, flash memory device, with electricity, light, sound or other form
The signal (such as carrier wave, infrared signal, digital signal) etc. propagated.
Those skilled in the art of the present technique are appreciated that and can realize these structure charts with computer program instructions
And/or the frame in each frame in block diagram and/or flow graph and these structure charts and/or block diagram and/or flow graph
Combination.These computer program instructions can be supplied to general purpose computer, special purpose computer or other can compile
The processor of journey data processing method generates machine, thus is processed by computer or other programmable datas
The instruction that the processor of method performs create the frame for realizing structure chart and/or block diagram and/or flow graph or
The method specified in multiple frames.
Those skilled in the art of the present technique be appreciated that the present invention had discussed various operations, method,
Step in flow process, measure, scheme can be replaced, changed, combined or deleted.Further, have
Other steps in the various operations discussed in the present invention, method, flow process, measure, scheme also may be used
To be replaced, changed, reset, decomposed, combined or to be deleted.Further, of the prior art have with
Step in various operations disclosed in the present invention, method, flow process, measure, scheme can also be replaced,
Change, reset, decompose, combine or delete.
The above is only the some embodiments of the present invention, it is noted that common for the art
For technical staff, under the premise without departing from the principles of the invention, it is also possible to make some improvements and modifications,
These improvements and modifications also should be regarded as protection scope of the present invention.
Claims (14)
1. a data processing method, it is characterised in that including:
Obtain the tables of data representing merchandise news, and generate respectively corresponding to described according to the data in tables of data
Represent the lucene file of each tables of data of merchandise news；
Obtain the major key of tables of data, and read corresponding to being obtained from the lucene file generated according to described major key
The data of the major key taken；
According to the data corresponding to described major key, set up the association between the lucene file generated, and to finger
Determine memory area to submit to.
Data processing method the most according to claim 1, it is characterised in that described expression merchandise news
Tables of data include: commodity master meter and represent item property tables of data, the data of described expression item property
Table at least includes: commodity base attribute table, commodity price information table and commodity stocks information table.
Data processing method the most according to claim 2, it is characterised in that described acquisition tables of data
Major key, and from the lucene file generated, read the data corresponding to acquired major key according to described major key,
Including:
Obtain the commodity described in described commodity master meter uniquely to identify；
Inquiry obtains uniquely identifying described commodity the lucene file of the tables of data as major key, and from being inquired about
To lucene file in read item property data, described item property data at least include: commodity belong to substantially
Property data, commodity price data and commodity stocks data.
Data processing method the most according to claim 3, it is characterised in that described basis corresponds to institute
State the data of major key, set up the association between the lucene file generated, and submit to designated storage area,
Including:
Uniquely identify every item property data mart modeling according to described commodity and obtain association results；
Described association results is committed to solrcloud.
Data processing method the most according to claim 1, it is characterised in that also include:
According to the inquiry request received, determine the major key associated with described inquiry request；
According to the major key associated with described inquiry request, read data from described designated storage area.
6. according to the data processing method described in any one in claim 1-5, it is characterised in that also include:
When incremental data occurs in the tables of data described expression merchandise news being detected, obtain described incremental data
The major key of place tables of data；
The data of tables of data determined by reading, and be updated to described determined by the lucene file of tables of data.
7. according to the data processing method described in any one in claim 1-5, it is characterised in that also include:
When buffer queue occurs incremental data, determine the tables of data associated with described incremental data；
Described incremental data is updated to the lucene file of the tables of data associated with described incremental data.
8. a data processing equipment, it is characterised in that including:
Pretreatment module, for obtaining the tables of data representing merchandise news, and divides according to the data in tables of data
Do not generate the lucene file of each tables of data corresponding to described expression merchandise news；
Extraction module, for obtaining the major key of tables of data, and according to described major key from the lucene file generated
Read the data corresponding to acquired major key；
Analyze module, for obtaining according to the data corresponding to described major key, set up the lucene file generated
Between association, and to designated storage area submit to.
Data processing equipment the most according to claim 8, it is characterised in that described expression merchandise news
Tables of data include: commodity master meter and represent item property tables of data, the data of described expression item property
Table at least includes: commodity base attribute table, commodity price information table and commodity stocks information table.
Data processing equipment the most according to claim 9, it is characterised in that described extraction module, tool
Body uniquely identifies for obtaining the commodity described in described commodity master meter；
Inquiry obtains uniquely identifying described commodity the lucene file of the tables of data as major key, and from being inquired about
To lucene file in read item property data, described item property data at least include: commodity belong to substantially
Property data, commodity price data and commodity stocks data.
11. data processing equipments according to claim 10, it is characterised in that described analysis module,
Specifically for uniquely identifying every item property data mart modeling according to described commodity and obtaining association results；And
Described association results is committed to solrcloud.
12. data processing equipments according to claim 8, it is characterised in that also include: enquiry module,
For according to the inquiry request received, determining the major key associated with described inquiry request；And according to described
The major key of inquiry request association, reads data from described designated storage area.
13. data processing equipments described in any one in-12 according to Claim 8, it is characterised in that also wrap
Include the first more new module, be used for when incremental data occurs in the tables of data described expression merchandise news being detected,
Obtain the major key of described incremental data place tables of data；The data of tables of data determined by reading, and be updated to
The lucene file of tables of data determined by described.
14. data processing equipments described in any one in-12 according to Claim 8, it is characterised in that also wrap
Include the second more new module, for when buffer queue occurs incremental data, determine and close with described incremental data
The tables of data of connection；And described incremental data is updated to the lucene literary composition of the tables of data associated with described incremental data
Part.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510345408.7A CN106326243B (en) | 2015-06-19 | 2015-06-19 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510345408.7A CN106326243B (en) | 2015-06-19 | 2015-06-19 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106326243A true CN106326243A (en) | 2017-01-11 |
| CN106326243B CN106326243B (en) | 2020-02-21 |
Family
ID=57728004
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510345408.7A Active CN106326243B (en) | 2015-06-19 | 2015-06-19 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106326243B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107871020A (en) * | 2017-12-07 | 2018-04-03 | 郑州云海信息技术有限公司 | A kind of method based on SolrCloud completion data object attributes |
| CN108280226A (en) * | 2018-02-12 | 2018-07-13 | 中国银行股份有限公司 | Data processing method and relevant device |
| CN108509448A (en) * | 2017-02-27 | 2018-09-07 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| CN110019292A (en) * | 2017-09-06 | 2019-07-16 | 华为技术有限公司 | A kind of querying method and device of data |
| CN111382990A (en) * | 2018-12-29 | 2020-07-07 | 达丰(上海)电脑有限公司 | Information association management method and device, equipment and storage medium thereof |
| CN111585897A (en) * | 2020-04-02 | 2020-08-25 | 深圳壹账通智能科技有限公司 | Request route management method, system, computer system and readable storage medium |
| CN113763097A (en) * | 2020-12-14 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for updating article information |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102779185A (en) * | 2012-06-29 | 2012-11-14 | 浙江大学 | High-availability distribution type full-text index method |
| CN102915365A (en) * | 2012-10-24 | 2013-02-06 | 苏州两江科技有限公司 | Hadoop-based construction method for distributed search engine |
| CN102968495A (en) * | 2012-11-29 | 2013-03-13 | 河海大学 | Vertical search engine and method for searching contrast association shopping information |
| CN103729463A (en) * | 2014-01-14 | 2014-04-16 | 赛特斯信息科技股份有限公司 | Method for implementing full-text retrieval based on Lucene and Solr |
| CN104484398A (en) * | 2014-12-12 | 2015-04-01 | 北京国双科技有限公司 | Method and device for aggregation of data in datasheet |
| CN104516895A (en) * | 2013-09-27 | 2015-04-15 | 阿里巴巴集团控股有限公司 | Stock information processing method and system for commodity object |
| CN104537047A (en) * | 2014-12-25 | 2015-04-22 | 江南大学 | Garment basic sample plate retrieval system based on Lucene |
-
2015
- 2015-06-19 CN CN201510345408.7A patent/CN106326243B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102779185A (en) * | 2012-06-29 | 2012-11-14 | 浙江大学 | High-availability distribution type full-text index method |
| CN102915365A (en) * | 2012-10-24 | 2013-02-06 | 苏州两江科技有限公司 | Hadoop-based construction method for distributed search engine |
| CN102968495A (en) * | 2012-11-29 | 2013-03-13 | 河海大学 | Vertical search engine and method for searching contrast association shopping information |
| CN104516895A (en) * | 2013-09-27 | 2015-04-15 | 阿里巴巴集团控股有限公司 | Stock information processing method and system for commodity object |
| CN103729463A (en) * | 2014-01-14 | 2014-04-16 | 赛特斯信息科技股份有限公司 | Method for implementing full-text retrieval based on Lucene and Solr |
| CN104484398A (en) * | 2014-12-12 | 2015-04-01 | 北京国双科技有限公司 | Method and device for aggregation of data in datasheet |
| CN104537047A (en) * | 2014-12-25 | 2015-04-22 | 江南大学 | Garment basic sample plate retrieval system based on Lucene |
Non-Patent Citations (1)
| Title |
|---|
| 王富强 等: "基于Lucene的数据库全文信息检索", 《测绘科学》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509448A (en) * | 2017-02-27 | 2018-09-07 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| CN110019292A (en) * | 2017-09-06 | 2019-07-16 | 华为技术有限公司 | A kind of querying method and device of data |
| CN110019292B (en) * | 2017-09-06 | 2021-05-04 | 华为技术有限公司 | Data query method and device |
| CN107871020A (en) * | 2017-12-07 | 2018-04-03 | 郑州云海信息技术有限公司 | A kind of method based on SolrCloud completion data object attributes |
| CN107871020B (en) * | 2017-12-07 | 2020-09-15 | 浪潮云信息技术股份公司 | Method for complementing data object attributes based on SolrCloud |
| CN108280226A (en) * | 2018-02-12 | 2018-07-13 | 中国银行股份有限公司 | Data processing method and relevant device |
| CN108280226B (en) * | 2018-02-12 | 2022-04-12 | 中国银行股份有限公司 | Data processing method and related equipment |
| CN111382990A (en) * | 2018-12-29 | 2020-07-07 | 达丰(上海)电脑有限公司 | Information association management method and device, equipment and storage medium thereof |
| CN111585897A (en) * | 2020-04-02 | 2020-08-25 | 深圳壹账通智能科技有限公司 | Request route management method, system, computer system and readable storage medium |
| CN111585897B (en) * | 2020-04-02 | 2022-04-15 | 深圳壹账通智能科技有限公司 | Request route management method, system, computer system and readable storage medium |
| CN113763097A (en) * | 2020-12-14 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for updating article information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106326243B (en) | 2020-02-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106326243A (en) | Data processing method and apparatus | |
| US9928537B2 (en) | Management and storage of distributed bookmarks | |
| US10127285B2 (en) | Customizable ranking of search engine results in multi-tenant architecture | |
| CN108280365B (en) | Data access authority management method, device, terminal device and storage medium | |
| CN106844372B (en) | Logistics information query method and device | |
| US20200218737A1 (en) | Method, system and program product for matching of transaction records | |
| CN102253936B (en) | Method for recording access of user to merchandise information, search method and server | |
| CN108228817A (en) | Data processing method, device and system | |
| CN105446991A (en) | Data storage method, query method and device | |
| CN105808653B (en) | A kind of data processing method and device based on user tag system | |
| CN104299094A (en) | Enterprise purchase-sales-inventory management information system | |
| CN103699549B (en) | The renewal of a kind of graphic code storehouse, querying method and relevant apparatus | |
| CN103365946A (en) | Database performance analysis | |
| KR20140058408A (en) | Method, system and computer program to provide fares detection from rules attributes | |
| CN111324786A (en) | Method and device for processing consultation problem information | |
| US11308102B2 (en) | Data catalog automatic generation system and data catalog automatic generation method | |
| CN105335386A (en) | Method and apparatus for providing navigation tag | |
| EP2884449A1 (en) | Transaction support system | |
| CN109947797B (en) | Data inspection device and method | |
| CN113282583A (en) | Data storage method, device, equipment and storage medium | |
| CN113379177A (en) | Task scheduling system and method | |
| KR102256814B1 (en) | Method and system for selecting target data | |
| CN115481026A (en) | Test case generation method and device, computer equipment and storage medium | |
| KR20170055379A (en) | Purchase price forecasting methods for new developments utilizing the ERP database | |
| CN114297317A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200115 Address after: 210042 No. 1-1 Suning Avenue, Xuzhuang Software Park, Xuanwu District, Nanjing City, Jiangsu Province Applicant after: Suning Cloud Computing Co., Ltd. Address before: 210042 Nanjing Province, Xuanwu District, Jiangsu Suning Avenue, Suning headquarters, No. 1 Applicant before: Yun Shang Group Plc of Suning |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20201117 Address after: No.1-9 Suning Avenue, Xuanwu District, Nanjing City, Jiangsu Province Patentee after: NANJING SUNING ELECTRONIC INFORMATION TECHNOLOGY Co.,Ltd. Address before: 210042 No. 1-1 Suning Avenue, Xuzhuang Software Park, Xuanwu District, Nanjing City, Jiangsu Province Patentee before: Suning Cloud Computing Co.,Ltd. |
|
| TR01 | Transfer of patent right |