arXiv Science⌕ Search

arXiv subjects

Liping Tang

Publications and source records attributed to Liping Tang.

At least 19 recordsLinked to original sources

An Inexact Proximal Gradient Method for Composite Multiobjective Optimization with Absolute and Relative Stopping Criteria

Proximal gradient methods are widely used for composite optimization. In multiobjective optimization, however, the resulting minimax subproblem generally requires an additional inner method to solve, even when the proximal mapping of each nonsmooth component is available in closed form. In addition, most existing convergence results are obtained only under the assumption that the involved subproblems are solved exactly, which is unrealistic and limits the applicability of the proximal gradient methods. To make the proximal gradient methods practical, we propose an implementable inexact proximal gradient method based on proximal computation. Considering an approximate optimality condition and a hybrid stopping criterion including absolute and relative rules for the subproblem, we address the computational challenge by developing an inner inexact projected gradient method that can exploit the proximal mappings of the individual components. The $\mathcal{O}(1/k)$ convergence rate is also established for our method under certain conditions. Numerical experiments illustrate the computational benefits of solving the subproblems inexactly.

math.OC↗

K2-Horizon: A Frontier Fleet of Open Source Language Models

We present K2-Horizon, a fleet of six fully open language models, from a 0.9B dense model to a 375B-A23B MoE, built for general-purpose reasoning and agentic work across math and science, software engineering, deep research, office automation, and beyond. The models lead or are competitive with strong open models of their size on agentic and coding benchmarks. On the modeling side, to strengthen reasoning, we introduce Mixture-of-Value Attention (MoVA), applying MoE to attention value projections, and softplus attention gating, a more flexible, theoretically motivated gate. Our 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters. We also introduce Uno, lossless diffusion adapters that speed up decoding by up to 2.2x without a separate draft model. Our release follows the LLM360 definition of open source for transparency and reproducibility: the full development history, including intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code, notably our production-scale xLLM framework for dense and MoE training and the RL360 reinforcement-learning stack. Beyond the data itself, we release and document in detail every pipeline that built it, the part of model development that open releases most rarely disclose. These pipelines produced roughly 10T synthetic pre-training tokens, showing that synthetic data can be a major ingredient of pre-training at scale. Four of the models share one 22T-token pre-training corpus and recipe, revealing consistent scaling behavior and enabling controlled studies of scale and sparsity. All artifacts are available at https://ifm.ai/k2/

cs.AI↗

Conference Design for Consensus under KL Acceptance

We study consensus formation and information acquisition in conferences where DeGroot-type opinion updating is constrained by an acceptance rule based on Kullback-Leibler divergence. Players choose private information experiments before deliberation, and these choices determine both their posterior distributions and acceptance tolerances. We establish sufficient conditions for global consensus under specified conference structures, including repeated bilateral conferences organized by a fully receptive mediator. Under full reimbursement of the mediator's conference costs, the information-acquisition game has a weighted potential and admits a pure-strategy Nash equilibrium. We develop a numerical procedure using verified payoff bounds over the continuous posterior domain to certify approximate equilibria and assess partial reimbursement policies that preserve a given equilibrium profile. The binary state, three-player examples illustrate how conference order interacts with state variance. At larger variances in the tested range, placing the demanding nonmediator last is associated with lower conference costs and lower certified sufficient planner reimbursement, but greater information acquisition expenditure by the mediator. These findings reveal a trade-off between the planner's fiscal expenditure and the mediator's private burden, and show that the conference structure matters economically.

econ.TH↗

A Tight Surrogate-Based Adaptive Step Size for the Multiobjective Conditional Gradient Method

The multiobjective conditional gradient (MCondG) method is an effective descent-type algorithm for solving constrained multiobjective optimization problems. In this algorithm, the adaptive step size is derived by minimizing a one-dimensional convex quadratic surrogate function, which guarantees a sufficient decrease in all objective functions at each iteration. However, we observe that this surrogate function is relatively conservative, producing overly small step sizes and consequently slowing the convergence of the algorithm. This conservativeness stems from the fact that the surrogate function is dominated by the largest Lipschitz constant among the gradients of all objective functions. Motivated by this observation, we develop a tighter quadratic surrogate function that exploits the individual gradient Lipschitz constants. Based on this surrogate, we propose a new adaptive step size strategy obtained by evaluating this surrogate over a finite candidate set. We prove that the proposed step size preserves the descent property of the MCondG method, and has an improved worst-case complexity bound compared with the original adaptive step size strategy. Numerical experiments on benchmark and real-world problems demonstrate the effectiveness of the proposed strategy in terms of computational efficiency and solution quality.

math.OC↗

K2-V2: A 360-Open, Reasoning-Enhanced LLM

We introduce K2-V2, a 360-open LLM built from scratch as a superior base for reasoning adaptation, in addition to functions such as conversation and knowledge retrieval from general LLMs. It stands as the strongest fully open model, rivals open-weight leaders in its size class, outperforms Qwen2.5-72B and approaches the performance of Qwen3-235B. We actively infuse domain knowledge, reasoning, long-context, and tool use throughout the training process. This explicitly prepares the model for complex reasoning tasks. We demonstrate this potential using simple supervised fine-tuning, establishing a strong baseline that indicates significant headroom for advanced alignment. By releasing the full training history and data composition, we maximize the effectiveness of continuous training, a key open source production scenario. We release the model weights and signature LLM360 artifacts, such as complete training data, to empower the community with a capable, reasoning-centric foundation.

cs.LG↗

Barzilai-Borwein Diagonal Quasi-Newton Method for Nonconvex Multiobjective Optimization Problems

This paper addresses the challenge of developing efficient algorithms for large-scale nonconvex multiobjective optimization problems (MOPs). While quasi-Newton methods are effective, their traditional application to MOPs is computationally expensive as they require maintaining and inverting separate Hessian approximations for each objective function. To overcome this limitation, we propose a novel Barzilai-Borwein diagonal-type Quasi-Newton method (BB-DQN). Our key innovation is the use of a single, shared, and modified BB-type matrix, updated iteratively using function and gradient information, to approximate the Hessians of all objectives simultaneously. We theoretically demonstrate that this approximation matrix remains positive definite throughout the iterative process. Furthermore, we establish the global convergence of the BB-DQN method without convexity assumptions and prove its R-linear convergence under mild conditions. Numerical experiments on a diverse set of test problems confirm that BB-DQN outperforms existing methods like M-BFGSMO, achieving superior performance in terms of computational time, iteration count, and reliability, especially for large-scale instances.

math.OC↗

First-order Methods for Unconstrained Vector Optimization Problems: A Unified Majorization-Minimization Perspective

In this paper, we develop a unified majorization-minimization scheme and convergence analysis with first-order surrogate functions for unconstrained vector optimization problems (VOPs). By selecting different surrogate functions, the unified method can be reduced to various existing first-order methods. The unified convergence analysis reveals that the slow convergence of the steepest descent method is primarily attributed to the significant gap between the surrogate and objective functions. Consequently, narrowing this surrogate gap can enhance the performance of first-order methods for VOPs. To strike a better trade-off in terms of surrogate gap and per-iteration cost, we reformulate the direction-finding subproblem and elucidate that selecting a tighter surrogate function is equivalent to using an appropriate base of the dual cone in the direction-finding subproblem. Building on this insight, we employ the Barzilai-Borwein method to narrow the surrogate gap and propose a Barzilai-Borwein descent method for VOPs (BBDVO) with polyhedral cones. By reformulating the corresponding subproblem, we provide a novel perspective on the Barzilai-Borwein descent method, bridging the gap between this method and the steepest descent method. Finally, several numerical experiments are presented to validate the efficiency of the BBDVO.

math.OC↗

Linguistic Indirectness in Public Cheap-Talk Games

We study linguistic indirectness when speakers attend to social ties. Social ties are modeled by a graph, and conferences are the sets of nodes that hear a message. Conference worth is a distance polynomial on the graph; allocations are given by the Myerson value of the conference-restricted worth, which yields the bargaining-power components for each participant. Aggregating these components gives an effective bias that, via a Partition-Threshold rule, pins down the number of equilibrium message partitions in a cheap talk game. Results: (i) among trees, stars maximize worth, leading to weakly fewer equilibrium partitions; (ii) on stars, we derive closed-form effective biases, with a witness-hub marginal effect of adding leaves changing sign at $δ^{\ast}=0.6$; (iii) for two stars joined by one link, two-star (hub-hub) vs big-star (hub-leaf) precision flips at 8/15 for the same number of nodes; private leaf-leaf conferences are most informative.

econ.TH↗

MegaMath: Pushing the Limits of Open Math Corpora

Mathematical reasoning is a cornerstone of human intelligence and a key benchmark for advanced capabilities in large language models (LLMs). However, the research community still lacks an open, large-scale, high-quality corpus tailored to the demands of math-centric LLM pre-training. We present MegaMath, an open dataset curated from diverse, math-focused sources through following practices: (1) Revisiting web data: We re-extracted mathematical documents from Common Crawl with math-oriented HTML optimizations, fasttext-based filtering and deduplication, all for acquiring higher-quality data on the Internet. (2) Recalling Math-related code data: We identified high quality math-related code from large code training corpus, Stack-V2, further enhancing data diversity. (3) Exploring Synthetic data: We synthesized QA-style text, math-related code, and interleaved text-code blocks from web data or code data. By integrating these strategies and validating their effectiveness through extensive ablations, MegaMath delivers 371B tokens with the largest quantity and top quality among existing open math pre-training datasets.

cs.CL↗

An accelerated gradient method with adaptive restart for convex multiobjective optimization problems

In this work, based on the continuous time approach, we propose an accelerated gradient method with adaptive residual restart for convex multiobjective optimization problems. For the first, we derive rigorously the continuous limit of the multiobjective accelerated proximal gradient method by Tanabe et al. [Comput. Optim. Appl., 2023]. It is a second-order ordinary differential equation (ODE) that involves a special projection operator and can be viewed as an extension of the ODE by Su et al. [J. Mach. Learn. Res., 2016] for Nesterov acceleration. Then, we introduce a novel accelerated multiobjective gradient (AMG) flow with tailored time scaling that adapts automatically to the convex case and the strongly convex case, and the exponential decay rate of a merit function along with the solution trajectory of AMG flow is established via the Lyapunov analysis. After that, we consider an implicit-explicit time discretization and obtain an accelerated multiobjective gradient method with a convex quadratic programming subproblem. The fast sublinear rate and linear rate are proved respectively for convex and strongly convex problems. In addition, we present an efficient residual based adaptive restart technique to overcome the oscillation issue and improve the convergence significantly. Numerical results are provided to validate the practical performance of the proposed method.

math.OC↗

LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch

We detail the training of the LLM360 K2-65B model, scaling up our 360-degree OPEN SOURCE approach to the largest and most powerful models under project LLM360. While open-source LLMs continue to advance, the answer to "How are the largest LLMs trained?" remains unclear within the community. The implementation details for such high-capacity models are often protected due to business considerations associated with their high cost. This lack of transparency prevents LLM researchers from leveraging valuable insights from prior experience, e.g., "What are the best practices for addressing loss spikes?" The LLM360 K2 project addresses this gap by providing full transparency and access to resources accumulated during the training of LLMs at the largest scale. This report highlights key elements of the K2 project, including our first model, K2 DIAMOND, a 65 billion-parameter LLM that surpasses LLaMA-65B and rivals LLaMA2-70B, while requiring fewer FLOPs and tokens. We detail the implementation steps and present a longitudinal analysis of K2 DIAMOND's capabilities throughout its training process. We also outline ongoing projects such as TXT360, setting the stage for future models in the series. By offering previously unavailable resources, the K2 project also resonates with the 360-degree OPEN SOURCE principles of transparency, reproducibility, and accessibility, which we believe are vital in the era of resource-intensive AI research.

cs.LG↗

Accelerated Proximal Gradient Method with Backtracking for Multiobjective Optimization

This paper proposes a new backtracking strategy based on the FISTA accelerated algorithm for multiobjective optimization problems. The strategy focuses on solving the problem of Lipschitz constant being unknown. It allows estimate parameter updates non-increasingly. Furthermore, the proposed strategy effectively avoids the limitation in convergence proofs arising from the non-negativity of the auxiliary sequence, thus providing a theoretical guarantee for its performance. We demonstrate that, under relatively mild assumptions, the algorithm achieves the convergence rate of $O(1/k2)$.

math.OC↗

Scaled Proximal Gradient Methods for Multiobjective Optimization: Improved Linear Convergence and Nesterov's Acceleration

Over the past two decades, descent methods have received substantial attention within the multiobjective optimization field. Nonetheless, both theoretical analyses and empirical evidence reveal that existing first-order methods for multiobjective optimization converge slowly, even for well-conditioned problems, due to the objective imbalances. To address this limitation, we incorporate curvature information to scale each objective within the direction-finding subproblem, introducing a scaled proximal gradient method for multiobjective optimization (SPGMO). We demonstrate that the proposed method achieves improved linear convergence, exhibiting rapid convergence in well-conditioned scenarios. Furthermore, by applying small scaling to linear objectives, we prove that the SPGMO attains improved linear convergence for problems with multiple linear objectives. Additionally, integrating Nesterov's acceleration technique further enhances the linear convergence of SPGMO. To the best of our knowledge, this advancement in linear convergence is the first theoretical result that directly addresses objective imbalances in multiobjective first-order methods. Finally, we provide numerical experiments to validate the efficiency of the proposed methods and confirm the theoretical findings.

math.OC↗

Generalized conditional gradient methods for multiobjective composite optimization problems with H{ö}lder condition

In this paper, we deal with multiobjective composite optimization problems, where each objective function is a combination of smooth and possibly non-smooth functions. We first propose a parameter-dependent conditional gradient method to solve this problem. The step size in this method requires prior knowledge of the parameters related to the H{ö}lder continuity of the gradient of the smooth function. The convergence properties of this method are then established. Given that these parameters may be unknown or, if known, may not be unique, the first method may encounter implementation challenges or slow convergence. To address this, we further propose a parameter-free conditional gradient method that determines the step size using a local quadratic upper approximation and an adaptive line search strategy, eliminating the need for any problem-specific parameters. The performance of the proposed methods is demonstrated on several test problems involving the indicator function and an uncertainty function.

math.OC↗

Conjugate gradient methods without line search for multiobjective optimization

This paper addresses unconstrained multiobjective optimization problems where two or more continuously differentiable functions have to be minimized. We delve into the conjugate gradient methods proposed by Lucambio Pérez and Prudente (SIAM J Optim, 28(3): 2690--2720, 2018) for such problems. Instead of the Wolfe-type line search procedure used in their work, we employ a fixed stepsize formula (or no-line-search scheme), which can mitigate the pressure of choosing stepsize caused by multiple inequalities and avoid the computational cost associated with function evaluations in specific applications. The no-line-search scheme is utilized to derive the condition of Zoutendijk's type. Global convergence encompasses the vector extensions of Fletcher--Reeves, conjugate descent, Dai--Yuan, Polak--Ribière--Polyak and Hestenes--Stiefel parameters, subject to certain mild assumptions. Additionally, numerical experiments are conducted to demonstrate the practical performance of the proposed stepsize rule, and comparative analyses are made with the multiobjective steepest descent methods using the Armijo line search and the multiobjective conjugate gradient methods using the Wolfe-type line search.

math.OC↗

Mirror descent method for stochastic multi-objective optimization

Stochastic multi-objective optimization (SMOO) has recently emerged as a powerful framework for addressing machine learning problems with multiple objectives. The bias introduced by the nonlinearity of the subproblem solution mapping complicates the convergence analysis of multi-gradient methods. In this paper, we propose a novel SMOO method called the Multi-gradient Stochastic Mirror Descent (MSMD) method, which incorporates stochastic mirror descent method to solve the SMOO subproblem, providing convergence guarantees. By selecting an appropriate Bregman function, our method enables analytical solutions of the weighting vector and requires only a single gradient sample at each iteration. We demonstrate the sublinear convergence rate of our MSMD method under four different inner and outer step setups. For SMOO with preferences, we propose a variant of MSMD method and demonstrate its convergence rate. Through extensive numerical experiments, we compare our method with both stochastic descent methods based on weighted sum and state-of-the-art SMOO methods. Our method consistently outperforms these methods in terms of generating superior Pareto fronts on benchmark test functions while also achieving competitive results in neural network training.

math.OC↗

Decomposition-Invariant Pairwise Frank-Wolfe Algorithm for Constrained Multiobjective Optimization

Recently the away-step Frank-Wolfe algoritm for constrained multiobjective optimization has been shown linear convergence rate over a polytope which is generated by finite points set. In this paper we design a decomposition-invariant pairwise frank-wolfe algorithm for multiobjective optimization that the feasible region is an arbitrary bounded polytope. We prove it has linear convergence rate of the whole sequence to a pareto optimal solution under strongly convexity without other assumptions.

math.OC↗

Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding

The vast applications of deep generative models are anchored in three core capabilities -- generating new instances, reconstructing inputs, and learning compact representations -- across various data types, such as discrete text/protein sequences and continuous images. Existing model families, like variational autoencoders (VAEs), generative adversarial networks (GANs), autoregressive models, and (latent) diffusion models, generally excel in specific capabilities and data types but fall short in others. We introduce Generalized Encoding-Decoding Diffusion Probabilistic Models (EDDPMs) which integrate the core capabilities for broad applicability and enhanced performance. EDDPMs generalize the Gaussian noising-denoising in standard diffusion by introducing parameterized encoding-decoding. Crucially, EDDPMs are compatible with the well-established diffusion model objective and training recipes, allowing effective learning of the encoder-decoder parameters jointly with diffusion. By choosing appropriate encoder/decoder (e.g., large language models), EDDPMs naturally apply to different data types. Extensive experiments on text, proteins, and images demonstrate the flexibility to handle diverse data and tasks and the strong improvement over various existing models.

cs.LG↗