stella-pl-retrieval-mini-8k

This is an embedding model based on stella_en_400M_v5 and further fine-tuned for retrieval tasks in Polish. It transforms texts into 1024-dimensional vectors. The model training consisted of two stages: - In the first stage, we adapted the model to support the Polish language using the multilingual knowledge distillation method, leveraging a diverse corpus of 20 million Polish-English text pairs. - The original Stella model and the output of the first stage were limited to a short context of 512 tokens. In the second stage, we extended the context to 8192 tokens and then fine-tuned the model using contrastive loss on a dataset comprising 1.5 million queries. Positive and negative passages for each query have been selected with the help of BAAI/bge-reranker-v2.5-gemma2-lightweight reranker. The model was trained for five epochs with a batch size of 1024 queries. Note: The model uses a custom implementation that requires the XFormers library. For XFormers to function correctly, you need a compatible versions of Flash-Attention and PyTorch installed. Before using the model, make sure your XFormers installation is properly configured. For retrieval, queries should be prefixed with **"Instruct: Given a web search query, retrieve relevant passages that answer the query.\nQuery: "**.