arXiv subjects

Uriya Pumerantz

Publications and source records attributed to Uriya Pumerantz.

2 recordsLinked to original sources

arXiv2024

Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.

cs.CL↗

arXiv2020

Random walks, word metric and orbits distribution on the plane

Given a countably infinite group $G$ acting on some space $X$, an increasing family of finite subsets $G_n$ and $x\in X$, a natural question to ask is what asymptotical distribution the sets $G_nx$ form. More formally, we define for a function $f$ over $X$ the sums $S_n(f,x)=\sum_{g\in G_n}f(gx)$ and ask whether exists a function $\Psi(n):\mathbb{N}\to\mathbb{R}$ such that the sequence $\Psi(n)S_n(f,x)$ converges. This is a delicate problem that was studied under various settings. We first show a full solution when elements are chosen using a carefully chosen word metric from a specific lattice in $SL(2,\mathbb{Z})$ acting on the circle. In addition, it is proven that the resulting measure is stationary with respect to a certain random walk and has a tight connection to a well studied function from the field of Diophantine approximations. We then proceed to study the asymptotic distribution problem when elements are chosen using a random walk over $SL(2,\mathbb{R})$ acting on $\mathbb{R}^2$. We offer a variant of our initial problem which yields some surprising and interesting results.

math.DS↗