Search FoxAI

AI Models 14

• jina-embeddings-v2-base-en↗

  arkohut

  The easiest way to starting using `jina-embeddings-v2-base-en` is to use Jina AI's Embedding API. `jina-embeddings-v2-base-en` is an English, monolingual **embedding model** supporting **8192 sequence length**. It is based on a BERT architecture (JinaBERT) that supports the symmetric bidirectional variant of ALiBi to allow longer sequence length. The backbone `jina-bert-v2-base-en` is pretrained on the C4 dataset. The model is further trained on Jina AI's collection of more than 400 millions of sentence pairs and hard negatives. These pairs were obtained from various domains and were carefully selected through a thorough cleaning process. The embedding model was trained using 512 sequence length, but extrapolates to 8k sequence length (or even longer) thanks to ALiBi. This makes our model useful for a range of use cases, especially when processing long documents is needed, including long document retrieval, semantic textual similarity, text reranking, recommendation, RAG and LLM-based generative search, etc.

• Smaug-Llama-3-70B-Instruct-ExPO↗

  chujiezheng

  The extrapolated (ExPO) model based on `abacusai/Smaug-Llama-3-70B-Instruct` and `meta-llama/Meta-Llama-3-70B-Instruct`, as in the "Weak-to-Strong Extrapolation Expedites Alignment" paper. Specifically, we obtain this model by extrapolating **(alpha = 0.3)** from the weights of the SFT and DPO/RLHF checkpoints, achieving superior alignment with human preference. Evaluation results on the **AlpacaEval 2.0** benchmark (you can find the evaluation outputs on the official GitHub repo):

• internlm2-chat-7b-ExPO↗

  chujiezheng

  The extrapolated (ExPO) model based on `internlm2-chat-7b` and `internlm/internlm2-chat-7b-sft`, as in the "Weak-to-Strong Extrapolation Expedites Alignment" paper. Specifically, we obtain this model by extrapolating **(alpha = 0.5)** from the weights of the SFT and DPO/RLHF checkpoints, achieving superior alignment with human preference. Evaluation results on the **AlpacaEval 2.0** benchmark (you can find the evaluation outputs on the official GitHub repo):

• Llama3-70B-Chinese-Chat-ExPO↗

  chujiezheng

  The extrapolated (ExPO) model based on `shenzhi-wang/Llama3-70B-Chinese-Chat` and `meta-llama/Meta-Llama-3-70B-Instruct`, as in the "Weak-to-Strong Extrapolation Expedites Alignment" paper. Specifically, we obtain this model by extrapolating **(alpha = 0.3)** from the weights of the SFT and DPO/RLHF checkpoints, achieving superior alignment with human preference. **Note:** This is an experimental model, as I have not comprehensively evaluated its Chinese ability. **Unexpected issues may occur when we apply extrapolation to the DPO/RLHF alignment training for new languages (e.g., Chinese).**