arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2310.16861

General Point Model with Autoencoding and Autoregressive

Zhe Li·Zhangyang Gao·Cheng Tan·Stan Z. Li·Laurence T. Yang
View original source ↗Read source document ↗Export metadata ↓

Abstract

The pre-training architectures of large language models encompass various types, including autoencoding models, autoregressive models, and encoder-decoder models. We posit that any modality can potentially benefit from a large language model, as long as it undergoes vector quantization to become discrete tokens. Inspired by GLM, we propose a General Point Model (GPM) which seamlessly integrates autoencoding and autoregressive tasks in point cloud transformer. This model is versatile, allowing fine-tuning for downstream point cloud representation tasks, as well as unconditional and conditional generation tasks. GPM enhances masked prediction in autoencoding through various forms of mask padding tasks, leading to improved performance in point cloud understanding. Additionally, GPM demonstrates highly competitive results in unconditional point cloud generation tasks, even exhibiting the potential for conditional generation tasks by modifying the input's conditional information. Compared to models like Point-BERT, MaskPoint and PointMAE, our GPM achieves superior performance in point cloud understanding tasks. Furthermore, the integration of autoregressive and autoencoding within the same transformer underscores its versatility across different downstream tasks.

Explore related subjects

cs.LGcs.CV

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Zhe Li, Zhangyang Gao, Cheng Tan, Stan Z. Li, Laurence T. Yang. 2023-10-25. General Point Model with Autoencoding and Autoregressive. https://arxiv.org/abs/2310.16861

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Scaling Online Complex Event Detection with Synthetic Supervision and Mamba-Based Neural Algorithmic Reasoning

Modern machine learning models excel at detecting individual actions, sounds, or scene attributes from short, localized observations. However, many real-world tasks, such as in smart cities and healthcare, require reasoning over high-level complex events (CEs): spatiotemporal, rule-governed patterns of short-term atomic events (AEs). Complex event detection (CED) is challenging due to long temporal dependencies, generalization beyond the training horizon, sparse CE-level supervision without temporally aligned fine-grained AE labels, and cognitively demanding annotation, as CE labels often depend on ordering, duration, negation, and completion-time semantics. These challenges are further amplified in an online setting that requires causal, streaming inference with limited computation. We identify the primary bottleneck in online CED as learning robust CE rules, and propose a Neural Algorithmic Reasoning framework that decouples rule learning from low-level sensor semantics by (i) generating large-scale synthetic AE-level concept traces to pretrain a Mamba-based CE-rule reasoner, and (ii) introducing an adapter that learns to map raw sensor inputs into the reasoner's latent space using limited, labeled sensor data. We introduce a controlled simulator-generated online multilabel CED testbed built from real-world multimodal sensor clips and rule-generated CE labels, with stress-test settings that vary sensor noise, distribution shift, and the window size used to segment streaming sensor sequences. Experiments on this controlled benchmark show that NAROCE is competitive with the strongest baselines and often outperforms them under these stress tests and longer-horizon generalization, while using 5x fewer labeled sensor sequences and 10-20x fewer FLOPs than all non-Mamba baselines. Code and dataset available at https://github.com/nesl/naroce_dailyoce.

cs.LG↗
arXiv2026

Safe Learning Under Irreversible Dynamics via Asking for Help

Most learning algorithms with formal regret guarantees essentially rely on trying all possible behaviors, which is problematic when some errors cannot be recovered from. Instead, we allow the learning agent to ask for help from a mentor and to transfer knowledge between similar states. We show that this combination enables the agent to learn both safely and effectively. Under standard online learning assumptions, we provide an algorithm whose regret and number of mentor queries are both sublinear in the time horizon for Markov decision processes with irreversible dynamics and infinite state spaces. Our proof involves a sequence of three reductions, making our result more general than a single algorithm. Conceptually, our result may be the first formal proof that it is possible for an agent to obtain high reward while becoming self-sufficient in an unknown, unbounded, and high-stakes environment without resets.

cs.LG↗
arXiv2026

Reinforcement Learning from Human Feedback

Reinforcement learning from human feedback (RLHF) has become a crucial tool to build the latest machine learning systems at scale. The field grew around the core methods of RLHF into today's broader suite of post-training techniques. In this book, we give a comprehensive introduction to the core methods for post-training models for people with some level of quantitative background, organized around the canonical RLHF recipe. The book starts with what RLHF does and why it was created, with seminal technical milestones in its young history and a primer on reinforcement learning context needed to understand the book. The core of the book details every optimization stage in using RLHF, from starting with instruction tuning to training a reward model and finally all of rejection sampling, reinforcement learning, on-policy distillation, and direct alignment algorithms. The book also discusses broader topics, such as the origins of RLHF -- both in recent literature and in a convergence of disparate fields of science in economics, philosophy, and optimal control. The book concludes with advanced topics -- understudied or emerging research questions in synthetic data, tool-use, character training, and evaluation -- and open questions for the field. The book is released with a variety of companion resources, including a codebase, a library to compare model completions from within post-training stages, and an educational course, to be a one-stop shop for learning all foundational concepts for post-training language models.

cs.LG↗