From the beginning of the development of the Internet, search technology has bloomed with amazing social and economic value. With the rapid development of information society, the data is exploding. Search technology meets the needs of information sharing and rapid retrieval through data collection and processing.
ESCloud, a cloud search service, is a fully hosted online distributed search service provided by Volcano Engine, which is compatible with Elasticsearch, Kibana and other software and common open source plug-ins. It can provide multi-conditional retrieval, statistics and reports of structured and unstructured texts, help realize one-click deployment, flexible expansion and contraction, simplify operation and maintenance, and quickly build practical services such as log analysis and information retrieval analysis.
With the rise of Serverless and the general trend, Volcano Engine’s cloud search service has upgraded its cloud native new architecture.
Cloud Search Service Cloud Native Edition
K-NN, Native Vector Search and Database in the Age of Big Model
With the emergence of recommendation, audio and video applications and the demand for large model scenes, it is imperative to introduce multimodal search to meet more complex search needs. On the basis of full-text retrieval, we increase the ability of vector search to analyze and retrieve unstructured data.
In the scene of vector search, the machine learning model is used to generate vectors to represent data objects (text, image, audio and video, etc.); Vector distance to represent the similarity between objects. The commonly used vector database uses ANN algorithm to complete the retrieval of massive vectors in a very short time.
K-NN can be used as a vector database. By introducing an advanced vector algorithm library to build a vector index, the constructed vector index will be persisted to disk, making the index more stable. Combined with the inverted index of ESCloud products, the capabilities of vector retrieval and full-text retrieval can be integrated to achieve a more powerful Hybrid Search capability. Based on the cluster of ESCloud, k-NN vector database can provide large-scale distributed capability and bring scalable vector search to users.
Scenario case
There are six main business scenarios based on k-NN, which are currently used in complex business scenarios in ByteDance:
· Multi-modal search: including image search, semantic search, audio-video similarity retrieval, etc.
· Intelligent recommendation: video recommendation, advertising recommendation, relationship recommendation, product recommendation, etc.
· Intelligent question answering: FAQ based on Transformer, domain knowledge question answering based on LLM, and generative QA based on LangChain set;
· Data de-duplication: video, audio and pictures are audited and de-duplicated, and the copyright of various materials is detected;
· Safety risk control: fraud detection, black-out detection, risk assessment and anomaly detection;
· Other applications: data mining, data analysis, search reordering, text search.
Take the copy similarity recognition scheme as an example.
In the scenario where users push the copy, in order to ensure the user experience, it is necessary to ensure that there will be no duplicate content in the pushed copy, so the similarity of each pushed content will be identified and duplicated. Each copy generates Embedding through BERT model, which is retrieved once in cloud search. If the similarity is lower than the threshold, it is judged as a new copy, which will be written into the k-NN vector database and gradually improved into a copy database; If the similarity is higher than the threshold, it is judged as a duplicate copy, and the push amount is reduced.
ESCloud, a cloud search service, is compatible with Elasticsearch, Kibana and other softwares and common open source plug-ins, and provides multi-conditional retrieval, statistics and reports of structured and unstructured texts, which can realize one-click deployment, flexible expansion and contraction, simplify operation and maintenance, and quickly build business capabilities such as log analysis and information retrieval analysis.