arXiv Science⌕ Search

arXiv subjects

Tung Nguyen

Publications and source records attributed to Tung Nguyen.

At least 55 records · Page 3Linked to original sources
arXiv2024

Probing the Decision Boundaries of In-context Learning in Large Language Models

In-context learning is a key paradigm in large language models (LLMs) that enables them to generalize to new tasks and domains by simply prompting these models with a few exemplars without explicit parameter updates. Many attempts have been made to understand in-context learning in LLMs as a function of model scale, pretraining data, and other factors. In this work, we propose a new mechanism to probe and understand in-context learning from the lens of decision boundaries for in-context binary classification. Decision boundaries are straightforward to visualize and provide important information about the qualitative behavior of the inductive biases of standard classifiers. To our surprise, we find that the decision boundaries learned by current LLMs in simple binary classification tasks are often irregular and non-smooth, regardless of linear separability in the underlying task. This paper investigates the factors influencing these decision boundaries and explores methods to enhance their generalizability. We assess various approaches, including training-free and fine-tuning methods for LLMs, the impact of model architecture, and the effectiveness of active prompting techniques for smoothing decision boundaries in a data-efficient manner. Our findings provide a deeper understanding of in-context learning dynamics and offer practical improvements for enhancing robustness and generalizability of in-context learning.

cs.LG↗
arXiv2024

ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction

Accurate prediction of climate in the subseasonal-to-seasonal scale is crucial for disaster preparedness and robust decision making amidst climate change. Yet, forecasting beyond the weather timescale is challenging because it deals with problems other than initial condition, including boundary interaction, butterfly effect, and our inherent lack of physical understanding. At present, existing benchmarks tend to have shorter forecasting range of up-to 15 days, do not include a wide range of operational baselines, and lack physics-based constraints for explainability. Thus, we propose ChaosBench, a challenging benchmark to extend the predictability range of data-driven weather emulators to S2S timescale. First, ChaosBench is comprised of variables beyond the typical surface-atmospheric ERA5 to also include ocean, ice, and land reanalysis products that span over 45 years to allow for full Earth system emulation that respects boundary conditions. We also propose physics-based, in addition to deterministic and probabilistic metrics, to ensure a physically-consistent ensemble that accounts for butterfly effect. Furthermore, we evaluate on a diverse set of physics-based forecasts from four national weather agencies as baselines to our data-driven counterpart such as ViT/ClimaX, PanguWeather, GraphCast, and FourCastNetV2. Overall, we find methods originally developed for weather-scale applications fail on S2S task: their performance simply collapse to an unskilled climatology. Nonetheless, we outline and demonstrate several strategies that can extend the predictability range of existing weather emulators, including the use of ensembles, robust control of error propagation, and the use of physics-informed models. Our benchmark, datasets, and instructions are available at https://leap-stc.github.io/ChaosBench.

cs.CV↗
arXiv2024

Scaling transformer neural networks for skillful and reliable medium-range weather forecasting

Weather forecasting is a fundamental problem for anticipating and mitigating the impacts of climate change. Recently, data-driven approaches for weather forecasting based on deep learning have shown great promise, achieving accuracies that are competitive with operational systems. However, those methods often employ complex, customized architectures without sufficient ablation analysis, making it difficult to understand what truly contributes to their success. Here we introduce Stormer, a simple transformer model that achieves state-of-the-art performance on weather forecasting with minimal changes to the standard transformer backbone. We identify the key components of Stormer through careful empirical analyses, including weather-specific embedding, randomized dynamics forecast, and pressure-weighted loss. At the core of Stormer is a randomized forecasting objective that trains the model to forecast the weather dynamics over varying time intervals. During inference, this allows us to produce multiple forecasts for a target lead time and combine them to obtain better forecast accuracy. On WeatherBench 2, Stormer performs competitively at short to medium-range forecasts and outperforms current methods beyond 7 days, while requiring orders-of-magnitude less training data and compute. Additionally, we demonstrate Stormer's favorable scaling properties, showing consistent improvements in forecast accuracy with increases in model size and training tokens. Code and checkpoints are available at https://github.com/tung-nd/stormer.

physics.ao-ph↗
arXiv2024

Induced subgraph density. V. All paths approach Erdos-Hajnal

The Erdős-Hajnal conjecture says that, for every graph $H$, there exists $c>0$ such that every $H$-free graph on $n$ vertices has a clique or stable set of size at least $n^c$. In this paper we are concerned with the case when $H$ is a path. The conjecture has been proved for paths with at most five vertices, but not for longer paths. We prove that the conjecture is ``nearly'' true for all paths: for every path $H$, all $H$-free graphs with $n$ vertices have cliques or stable sets of size at least $2^{(\log n)^{1-o(1)}}$.

math.CO↗
arXiv2024

NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization

Recent advances in neural topic models have concentrated on two primary directions: the integration of the inference network (encoder) with a pre-trained language model (PLM) and the modeling of the relationship between words and topics in the generative model (decoder). However, the use of large PLMs significantly increases inference costs, making them less practical for situations requiring low inference times. Furthermore, it is crucial to simultaneously model the relationships between topics and words as well as the interrelationships among topics themselves. In this work, we propose a novel framework called NeuroMax (Neural Topic Model with Maximizing Mutual Information with Pretrained Language Model and Group Topic Regularization) to address these challenges. NeuroMax maximizes the mutual information between the topic representation obtained from the encoder in neural topic models and the representation derived from the PLM. Additionally, NeuroMax employs optimal transport to learn the relationships between topics by analyzing how information is transported among them. Experimental results indicate that NeuroMax reduces inference time, generates more coherent topics and topic groups, and produces more representative document embeddings, thereby enhancing performance on downstream tasks.

cs.CL↗
arXiv2024

Some results and problems on tournament structure

This paper is a survey of results and problems related to the following question: is it true that if G is a tournament with sufficiently large chromatic number, then G has two vertex-disjoint subtournaments A,B, both with large chromatic number, such that all edges between them are directed from A to B? We describe what we know about this question, and report some progress on several other related questions, on tournament colouring and domination.

math.CO↗
arXiv2024

Induced subgraph density. II. Sparse and dense sets in cographs

A well-known theorem of Rödl says that for every graph $H$, and every $ε>0$, there exists $δ>0$ such that if $G$ does not contain an induced copy of $H$, then there exists $X\subseteq V(G)$ with $|X|\ge δ|G|$ such that one of $G[X],\overline{G}[X]$ has edge-density at most $ε$. But how does $δ$ depend on $ε$? Fox and Sudakov conjectured that the dependence is at most polynomial: that for all $H$ there exists $c>0$ such that for all $ε$ with $0<ε\le 1/2$, Rödl's theorem holds with $δ=ε^c$. This conjecture implies the Erdős-Hajnal conjecture, and until now it had not been verified for any non-trivial graphs $H$. Our first result shows that it is true when $H=P_4$. Indeed, in that case we can take $δ=ε$, and insist that one of $G[X],\overline{G}[X]$ has maximum degree at most $ε^2|G|$). Second, we will show that every graph $H$ that can be obtained by substitution from copies of $P_4$ satisfies the Fox-Sudakov conjecture. To prove this, we need to work with a stronger property. Let us say $H$ is {\em viral} if there exists $c>0$ such that for all $ε$ with $0<ε\le 1/2$, if $G$ contains at most $ε^c|G|^{|H|}$ copies of $H$ as induced subgraphs, then there exists $X\subseteq V(G)$ with $|X|\ge ε^c|G|$ such that one of $G[X],\overline{G}[X]$ has edge-density at most $ε$. We will show that $P_4$ is viral, using a ``polynomial $P_4$-removal lemma'' of Alon and Fox. We will also show that the class of viral graphs is closed under vertex-substitution. Finally, we give a different strengthening of Rödl's theorem: we show that if $G$ does not contain an induced copy of $P_4$, then its vertices can be partitioned into at most $480ε^{-4}$ subsets $X$ such that one of $G[X],\overline{G}[X]$ has maximum degree at most $ε|X|$.

math.CO↗
arXiv2024

Distant digraph domination

A {\em $k$-kernel} in a digraph $G$ is a stable set $X$ of vertices such that every vertex of $G$ can be joined from $X$ by a directed path of length at most $k$. We prove three results about $k$-kernels. First, it was conjectured by Erdős and Székely in 1976 that every digraph $G$ with no source has a 2-kernel $|K|$ with $|K|\le |G|/2$. We prove this conjecture when $G$ is a ``split digraph'' (that is, its vertex set can be partitioned into a tournament and a stable set), improving a result of Langlois et al., who proved that every split digraph $G$ with no source has a 2-kernel of size at most $2|G|/3$. Second, the Erdős-Székely conjecture implies that in every digraph $G$ there is a 2-kernel $K$ such that the union of $K$ and its out-neighbours has size at least $|G|/2$. We prove that this is true if $V(G)$ can be partitioned into a tournament and an acyclic set. Third, in a recent paper, Spiro asked whether, for all $k\ge 3$, every strongly-connected digraph $G$ has a $k$-kernel of size at most about $|G|/(k+1)$. This remains open, but we prove that there is one of size at most about $|G|/(k-1)$.

math.CO↗
arXiv2024

Induced subgraph density. III. Cycles and subdivisions

We show that for every two cycles $C,D$, there exists $c>0$ such that if $G$ is both $C$-free and $\overline{D}$-free then $G$ has a clique or stable set of size at least $|G|^c$. ("$H$-free" means with no induced subgraph isomorphic to $H$, and $\overline{D}$ denotes the complement graph of $D$.) Since the five-vertex cycle $C_5$ is isomorphic to its complement, this extends the earlier result that $C_5$ satisfies the Erdős-Hajnal conjecture. It also unifies and strengthens several other results. The results for cycles are special cases of results for subdivisions, as follows. Let $H,J$ be obtained from smaller graphs by subdividing every edge exactly twice. We will prove that there exists $c>0$ such that if $G$ is both $H$-free and $\overline{J}$-free then $G$ has a clique or stable set of size at least $|G|^c$. And the same holds if $H$ and/or $J$ is obtained from a graph bychoosing a forest $F$ and subdividing every edge not in $F$ at least five times. Our proof uses the framework of iterative sparsification developed in other papers of this series. Along the way, we will also give a short and simple proof of a celebrated result of Fox and Sudakov, that says that for all $H$, every $H$-free graph contains either a large stable set or a large complete bipartite subgraph.

math.CO↗
arXiv2024

Rank-Preference Consistency as the Appropriate Metric for Recommender Systems

In this paper we argue that conventional unitary-invariant measures of recommender system (RS) performance based on measuring differences between predicted ratings and actual user ratings fail to assess fundamental RS properties. More specifically, posing the optimization problem as one of predicting exact user ratings provides only an indirect suboptimal approximation for what RS applications typically need, which is an ability to accurately predict user preferences. We argue that scalar measures such as RMSE and MAE with respect to differences between actual and predicted ratings are only proxies for measuring RS ability to accurately estimate user preferences. We propose what we consider to be a measure that is more fundamentally appropriate for assessing RS performance, rank-preference consistency, which simply counts the number of prediction pairs that are inconsistent with the user's expressed product preferences. For example, if an RS predicts the user will prefer product A over product B, but the user's withheld ratings indicate s/he prefers product B over A, then rank-preference consistency has been violated. Our test results conclusively demonstrate that methods tailored to optimize arbitrary measures such as RMSE are not generally effective at accurately predicting user preferences. Thus, we conclude that conventional methods used for assessing RS performance are arbitrary and misleading.

cs.IR↗
arXiv2024

Induced subgraph density. I. A loglog step towards Erdos-Hajnal

In 1977, Erdős and Hajnal made the conjecture that, for every graph $H$, there exists $c>0$ such that every $H$-free graph $G$ has a clique or stable set of size at least $|G|^c$; and they proved that this is true with $ |G|^c$ replaced by $2^{c\sqrt{\log |G|}}$. Until now, there has been no improvement on this result (for general $H$). We prove a strengthening: that for every graph $H$, there exists $c>0$ such that every $H$-free graph $G$ with $|G|\ge 2$ has a clique or stable set of size at least $$2^{c\sqrt{\log |G|\log\log|G|}}.$$ Indeed, we prove the corresponding strengthening of a theorem of Fox and Sudakov, which in turn was a common strengthening of theorems of Rödl, Nikiforov, and the theorem of Erdős and Hajnal mentioned above.

math.CO↗
arXiv2023

A Study on Social Robot Behavior in Group Conversation

Recently, research in human-robot interaction began to consider a robot's influence at the group level. Despite the recent growth in research investigating the effects of robots within groups of people, our overall understanding of what happens when robots are placed within groups or teams of people is still limited. This paper investigates several key problems for social robots that manage conversations in a group setting, where the number of participants is more than two. In a group setting, the conversation dynamics are a lot more complicated than the conventional one-to-one conversation, thus, there are more challenges need to be solved.

cs.HC↗
arXiv2023

ClimaX: A foundation model for weather and climate

Most state-of-the-art approaches for weather and climate modeling are based on physics-informed numerical models of the atmosphere. These approaches aim to model the non-linear dynamics and complex interactions between multiple variables, which are challenging to approximate. Additionally, many such numerical models are computationally intensive, especially when modeling the atmospheric phenomenon at a fine-grained spatial and temporal resolution. Recent data-driven approaches based on machine learning instead aim to directly solve a downstream forecasting or projection task by learning a data-driven functional mapping using deep neural networks. However, these networks are trained using curated and homogeneous climate datasets for specific spatiotemporal tasks, and thus lack the generality of numerical models. We develop and demonstrate ClimaX, a flexible and generalizable deep learning model for weather and climate science that can be trained using heterogeneous datasets spanning different variables, spatio-temporal coverage, and physical groundings. ClimaX extends the Transformer architecture with novel encoding and aggregation blocks that allow effective use of available compute while maintaining general utility. ClimaX is pre-trained with a self-supervised learning objective on climate datasets derived from CMIP6. The pre-trained ClimaX can then be fine-tuned to address a breadth of climate and weather tasks, including those that involve atmospheric variables and spatio-temporal scales unseen during pretraining. Compared to existing data-driven baselines, we show that this generality in ClimaX results in superior performance on benchmarks for weather forecasting and climate projections, even when pretrained at lower resolutions and compute budgets. The source code is available at https://github.com/microsoft/ClimaX.

cs.LG↗
arXiv2023

ExPT: Synthetic Pretraining for Few-Shot Experimental Design

Experimental design is a fundamental problem in many science and engineering fields. In this problem, sample efficiency is crucial due to the time, money, and safety costs of real-world design evaluations. Existing approaches either rely on active data collection or access to large, labeled datasets of past experiments, making them impractical in many real-world scenarios. In this work, we address the more challenging yet realistic setting of few-shot experimental design, where only a few labeled data points of input designs and their corresponding values are available. We approach this problem as a conditional generation task, where a model conditions on a few labeled examples and the desired output to generate an optimal input design. To this end, we introduce Experiment Pretrained Transformers (ExPT), a foundation model for few-shot experimental design that employs a novel combination of synthetic pretraining with in-context learning. In ExPT, we only assume knowledge of a finite collection of unlabelled data points from the input domain and pretrain a transformer neural network to optimize diverse synthetic functions defined over this domain. Unsupervised pretraining allows ExPT to adapt to any design task at test time in an in-context fashion by conditioning on a few labeled data points from the target task and generating the candidate optima. We evaluate ExPT on few-shot experimental design in challenging domains and demonstrate its superior generality and performance compared to existing methods. The source code is available at https://github.com/tung-nd/ExPT.git.

cs.LG↗
arXiv2023

Tensor Completion with Provable Consistency and Fairness Guarantees for Recommender Systems

We introduce a new consistency-based approach for defining and solving nonnegative/positive matrix and tensor completion problems. The novelty of the framework is that instead of artificially making the problem well-posed in the form of an application-arbitrary optimization problem, e.g., minimizing a bulk structural measure such as rank or norm, we show that a single property/constraint: preserving unit-scale consistency, guarantees the existence of both a solution and, under relatively weak support assumptions, uniqueness. The framework and solution algorithms also generalize directly to tensors of arbitrary dimensions while maintaining computational complexity that is linear in problem size for fixed dimension d. In the context of recommender system (RS) applications, we prove that two reasonable properties that should be expected to hold for any solution to the RS problem are sufficient to permit uniqueness guarantees to be established within our framework. This is remarkable because it obviates the need for heuristic-based statistical or AI methods despite what appear to be distinctly human/subjective variables at the heart of the problem. Key theoretical contributions include a general unit-consistent tensor-completion framework with proofs of its properties, e.g., consensus-order and fairness, and algorithms with optimal runtime and space complexities, e.g., O(1) term-completion with preprocessing complexity that is linear in the number of known terms of the matrix/tensor. From a practical perspective, the seamless ability of the framework to generalize to exploit high-dimensional structural relationships among key state variables, e.g., user and product attributes, offers a means for extracting significantly more information than is possible for alternative methods that cannot generalize beyond direct user-product relationships.

cs.IR↗
arXiv2023

Imposing Consistency Properties on Blackbox Systems with Applications to SVD-Based Recommender Systems

In this paper we discuss pre- and post-processing methods to induce desired consistency and/or invariance properties in blackbox systems, e.g., AI-based. We demonstrate our approach in the context of blackbox SVD-based matrix-completion methods commonly used in recommender system (RS) applications. We provide empirical results showing that enforcement of unit-consistency and shift-consistency, which have provable RS-relevant properties relating to robustness and fairness, also lead to improved performance according to generic RMSE and MAE performance metrics, irrespective of the initial chosen hyperparameter.

cs.IR↗
arXiv2023

An Admissible Shift-Consistent Method for Recommender Systems

In this paper, we propose a new constraint, called shift-consistency, for solving matrix/tensor completion problems in the context of recommender systems. Our method provably guarantees several key mathematical properties: (1) satisfies a recently established admissibility criterion for recommender systems; (2) satisfies a definition of fairness that eliminates a specific class of potential opportunities for users to maliciously influence system recommendations; and (3) offers robustness by exploiting provable uniqueness of missing-value imputation. We provide a rigorous mathematical description of the method, including its generalization from matrix to tensor form to permit representation and exploitation of complex structural relationships among sets of user and product attributes. We argue that our analysis suggests a structured means for defining latent-space projections that can permit provable performance properties to be established for machine learning methods.

cs.IR↗