AIが動く。未来が更新される。Zamba2-2.7B
Zamba2-2.7B is a hybrid model composed of state-space and transformer blocks. It broadly follows the Zamba architecture which consists of a Mamba backbone alternating with shared transformer blocks (see diagram in Model Details). Zamba2-2.7B possesses three major improvements over Zamba1: 2.) Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. 3.) We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. LoRA enables us to add depth-specialization for only a minimal increase in total parameter count.
翻訳準備中のため原文を表示しています
Zamba2-2.7B is a hybrid model composed of state-space and transformer blocks. It broadly follows the Zamba architecture which consists of a Mamba backbone alternating with shared transformer blocks (see diagram in Model Details). Zamba2-2.7B possesses three major improvements over Zamba1: 2.) Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. 3.) We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. LoRA enables us to add depth-specialization for only a minimal increase in total parameter count.