arXiv subjects

Tianyi Zhang

Publications and source records attributed to Tianyi Zhang.

At least 325 records · Page 18Linked to original sources

arXiv2023

The Effect of Masking Strategies on Knowledge Retention by Language Models

Language models retain a significant amount of world knowledge from their pre-training stage. This allows knowledgeable models to be applied to knowledge-intensive tasks prevalent in information retrieval, such as ranking or question answering. Understanding how and which factual information is acquired by our models is necessary to build responsible models. However, limited work has been done to understand the effect of pre-training tasks on the amount of knowledge captured and forgotten by language models during pre-training. Building a better understanding of knowledge acquisition is the goal of this paper. Therefore, we utilize a selection of pre-training tasks to infuse knowledge into our model. In the following steps, we test the model's knowledge retention by measuring its ability to answer factual questions. Our experiments show that masking entities and principled masking of correlated spans based on pointwise mutual information lead to more factual knowledge being retained than masking random tokens. Our findings demonstrate that, like the ability to perform a task, the (factual) knowledge acquired from being trained on that task is forgotten when a model is trained to perform another task (catastrophic forgetting) and how to prevent this phenomenon. To foster reproducibility, the code, as well as the data used in this paper, are openly available.

cs.CL↗

arXiv2023

Automated Summarization of Stack Overflow Posts

Software developers often resort to Stack Overflow (SO) to fill their programming needs. Given the abundance of relevant posts, navigating them and comparing different solutions is tedious and time-consuming. Recent work has proposed to automatically summarize SO posts to concise text to facilitate the navigation of SO posts. However, these techniques rely only on information retrieval methods or heuristics for text summarization, which is insufficient to handle the ambiguity and sophistication of natural language. This paper presents a deep learning based framework called ASSORT for SO post summarization. ASSORT includes two complementary learning methods, ASSORT_S and ASSORT_{IS}, to address the lack of labeled training data for SO post summarization. ASSORT_S is designed to directly train a novel ensemble learning model with BERT embeddings and domainspecific features to account for the unique characteristics of SO posts. By contrast, ASSORT_{IS} is designed to reuse pre-trained models while addressing the domain shift challenge when no training data is present (i.e., zero-shot learning). Both ASSORT_S and ASSORT_{IS} outperform six existing techniques by at least 13% and 7% respectively in terms of the F1 score. Furthermore, a human study shows that participants significantly preferred summaries generated by ASSORT_S and ASSORT_{IS} over the best baseline, while the preference difference between ASSORT_S and ASSORT_{IS} was small.

cs.SE↗

arXiv2023

SAD: Semi-Supervised Anomaly Detection on Dynamic Graphs

Anomaly detection aims to distinguish abnormal instances that deviate significantly from the majority of benign ones. As instances that appear in the real world are naturally connected and can be represented with graphs, graph neural networks become increasingly popular in tackling the anomaly detection problem. Despite the promising results, research on anomaly detection has almost exclusively focused on static graphs while the mining of anomalous patterns from dynamic graphs is rarely studied but has significant application value. In addition, anomaly detection is typically tackled from semi-supervised perspectives due to the lack of sufficient labeled data. However, most proposed methods are limited to merely exploiting labeled data, leaving a large number of unlabeled samples unexplored. In this work, we present semi-supervised anomaly detection (SAD), an end-to-end framework for anomaly detection on dynamic graphs. By a combination of a time-equipped memory bank and a pseudo-label contrastive learning module, SAD is able to fully exploit the potential of large unlabeled samples and uncover underlying anomalies on evolving graph streams. Extensive experiments on four real-world datasets demonstrate that SAD efficiently discovers anomalies from dynamic graphs and outperforms existing advanced methods even when provided with only little labeled data.

cs.LG↗

arXiv2023

Existence of Optical Vortex Solitons in Photorefractive Media

Optical propagation and vortices in nonlinear media have been intensively studied in modern optical physics. In this paper, we establish constraints regarding the propagation constant and provide an existence theory and numerical computations for positive exponentially decaying solutions for a class of ring-profiled solitons in a type of nonlinear media known as a photorefractive nonlinearity. Our methods include constrained minimization and finite element formalism, and we study the vortex profile and its propagation by fixing the energy flux.

math.AP↗

arXiv2023

CDFI: Cross Domain Feature Interaction for Robust Bronchi Lumen Detection

Endobronchial intervention is increasingly used as a minimally invasive means for the treatment of pulmonary diseases. In order to reduce the difficulty of manipulation in complex airway networks, robust lumen detection is essential for intraoperative guidance. However, these methods are sensitive to visual artifacts which are inevitable during the surgery. In this work, a cross domain feature interaction (CDFI) network is proposed to extract the structural features of lumens, as well as to provide artifact cues to characterize the visual features. To effectively extract the structural and artifact features, the Quadruple Feature Constraints (QFC) module is designed to constrain the intrinsic connections of samples with various imaging-quality. Furthermore, we design a Guided Feature Fusion (GFF) module to supervise the model for adaptive feature fusion based on different types of artifacts. Results show that the features extracted by the proposed method can preserve the structural information of lumen in the presence of large visual variations, bringing much-improved lumen detection accuracy.

cs.CV↗

arXiv2023

AutoRepair: Automated Repair for AI-Enabled Cyber-Physical Systems under Safety-Critical Conditions

Cyber-Physical Systems (CPS) have been widely deployed in safety-critical domains such as transportation, power and energy. Recently, there comes an increasing demand in employing deep neural networks (DNNs) in CPS for more intelligent control and decision making in sophisticated industrial safety-critical conditions, giving birth to the class of DNN controllers. However, due to the inherent uncertainty and opaqueness of DNNs, concerns about the safety of DNN-enabled CPS are also surging. In this work, we propose an automated framework named AutoRepair that, given a safety requirement, identifies unsafe control behavior in a DNN controller and repairs them through an optimization-based method. Having an unsafe signal of system execution, AutoRepair iteratively explores the control decision space and searches for the optimal corrections for the DNN controller in order to satisfy the safety requirements. We conduct a comprehensive evaluation of AutoRepair on 6 instances of industry-level DNN-enabled CPS from different safety-critical domains. Evaluation results show that AutoRepair successfully repairs critical safety issues in the DNN controllers, and significantly improves the reliability of CPS.

cs.SE↗

arXiv2023

Giant room-temperature nonlinearities from a monolayer Janus topological semiconductor

Nonlinear optical materials possess wide applications, ranging from terahertz and mid-infrared detection to energy harvesting. Recently, the correlations between nonlinear optical responses and topological properties, such as Berry curvature and the quantum metric tensor, have stimulated great interest. Here, we report giant room-temperature nonlinearities in an emergent non-centrosymmetric two-dimensional topological material, the Janus transition metal dichalcogenides in the 1T' phase, which are synthesized by an advanced atomic-layer substitution method. High harmonic generation, terahertz emission spectroscopy, and second harmonic generation measurements consistently reveal orders-of-the-magnitude enhancement in terahertz-frequency nonlinearities of 1T' MoSSe (e.g., > 50 times higher than 2H MoS$_2$ for 18th order harmonic generation; > 20 times higher than 2H MoS$_2$ for terahertz emission). It is elucidated that such colossal nonlinear optical responses come from topological band mixing and strong inversion symmetry breaking due to the Janus structure. Our work defines general protocols for designing materials with large nonlinearities and preludes the applications of topological materials in optoelectronics down to the monolayer limit. This two-dimensional form of topological materials also constitute a unique platform for examining origin of the anomalous high-harmonic generation, with potential applications as building blocks for scalable attosecond sources.

cond-mat.mes-hall↗

arXiv2023

The $\textit{NuSTAR}$ Extragalactic Surveys: Source Catalogs from the Extended $\textit{Chandra}$ Deep Field-South and the $\textit{Chandra}$ Deep Field-North

We present a routinized and reliable method to obtain source catalogs from the $\textit{Nuclear Spectroscopic Telescope Array}$ ($\textit{NuSTAR}$) extragalactic surveys of the Extended $\textit{Chandra}$ Deep Field-South (E-CDF-S) and $\textit{Chandra}$ Deep Field-North (CDF-N). The $\textit{NuSTAR}$ E-CDF-S survey covers a sky area of $\approx30'\times30'$ to a maximum depth of $\sim$ 230 ks corrected for vignetting in the 3--24 keV band, with a total of 58 sources detected in our E-CDF-S catalog; the $\textit{NuSTAR}$ CDF-N survey covers a sky area of $\approx7'\times10'$ to a maximum depth of $\sim$ 440 ks corrected for vignetting in the 3--24 keV band, with a total of 42 sources detected in our CDF-N catalog that is produced for the first time. We verify the reliability of our two catalogs by crossmatching them with the relevant catalogs from the $\textit{Chandra}$ X-ray observatory, and find that the fluxes of our $\textit{NuSTAR}$ sources are generally consistent with that of their $\textit{Chandra}$ counterparts. Our two catalogs are produced following the exactly same method and made publicly available, thereby providing a uniform platform that facilitates further studies involving these two fields. Our source-detection method provides a systematic approach for source cataloging in other $\textit{NuSTAR}$ extragalactic surveys.

astro-ph.GA↗

arXiv2023

From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers

In this paper we provide, to the best of our knowledge, the first comprehensive approach for incorporating various masking mechanisms into Transformers architectures in a scalable way. We show that recent results on linear causal attention (Choromanski et al., 2021) and log-linear RPE-attention (Luo et al., 2021) are special cases of this general mechanism. However by casting the problem as a topological (graph-based) modulation of unmasked attention, we obtain several results unknown before, including efficient d-dimensional RPE-masking and graph-kernel masking. We leverage many mathematical techniques ranging from spectral analysis through dynamic programming and random walks to new algorithms for solving Markov processes on graphs. We provide a corresponding empirical evaluation.

cs.LG↗

arXiv2023

Cost Sharing under Private Valuation and Connection Control

We consider a cost sharing problem on a weighted undirected graph, where all the nodes want to connect to a special node called source, and they need to share the total cost (weights) of the used edges. Each node except for the source has a private valuation of the connection, and it may block others' connections by strategically cutting its adjacent edges to reduce its cost share, which may increase the total cost. We aim to design mechanisms to prevent the nodes from misreporting their valuations and cutting their adjacent edges. We first show that it is impossible for such a mechanism to further satisfy budget balance (cover the total cost) and efficiency (maximize social welfare). Then, we design two feasible cost sharing mechanisms that incentivize each node to offer all its adjacent edges and truthfully report its valuation, and also satisfy either budget balance or efficiency.

cs.GT↗

arXiv2023

DeepLens: Interactive Out-of-distribution Data Detection in NLP Models

Machine Learning (ML) has been widely used in Natural Language Processing (NLP) applications. A fundamental assumption in ML is that training data and real-world data should follow a similar distribution. However, a deployed ML model may suffer from out-of-distribution (OOD) issues due to distribution shifts in the real-world data. Though many algorithms have been proposed to detect OOD data from text corpora, there is still a lack of interactive tool support for ML developers. In this work, we propose DeepLens, an interactive system that helps users detect and explore OOD issues in massive text corpora. Users can efficiently explore different OOD types in DeepLens with the help of a text clustering method. Users can also dig into a specific text by inspecting salient words highlighted through neuron activation analysis. In a within-subjects user study with 24 participants, participants using DeepLens were able to find nearly twice more types of OOD issues accurately with 22% more confidence compared with a variant of DeepLens that has no interaction or visualization support.

cs.HC↗

arXiv2023

DeepSeer: Interactive RNN Explanation and Debugging via State Abstraction

Recurrent Neural Networks (RNNs) have been widely used in Natural Language Processing (NLP) tasks given its superior performance on processing sequential data. However, it is challenging to interpret and debug RNNs due to the inherent complexity and the lack of transparency of RNNs. While many explainable AI (XAI) techniques have been proposed for RNNs, most of them only support local explanations rather than global explanations. In this paper, we present DeepSeer, an interactive system that provides both global and local explanations of RNN behavior in multiple tightly-coordinated views for model understanding and debugging. The core of DeepSeer is a state abstraction method that bundles semantically similar hidden states in an RNN model and abstracts the model as a finite state machine. Users can explore the global model behavior by inspecting text patterns associated with each state and the transitions between states. Users can also dive into individual predictions by inspecting the state trace and intermediate prediction results of a given input. A between-subjects user study with 28 participants shows that, compared with a popular XAI technique, LIME, participants using DeepSeer made a deeper and more comprehensive assessment of RNN model behavior, identified the root causes of incorrect predictions more accurately, and came up with more actionable plans to improve the model performance.

cs.HC↗

arXiv2023

Human-in-the-Loop Schema Induction

Schema induction builds a graph representation explaining how events unfold in a scenario. Existing approaches have been based on information retrieval (IR) and information extraction(IE), often with limited human curation. We demonstrate a human-in-the-loop schema induction system powered by GPT-3. We first describe the different modules of our system, including prompting to generate schematic elements, manual edit of those elements, and conversion of those into a schema graph. By qualitatively comparing our system to previous ones, we show that our system not only transfers to new domains more easily than previous approaches, but also reduces efforts of human curation thanks to our interactive interface.

cs.HC↗

arXiv2023

GRANDE: a neural model over directed multigraphs with application to anti-money laundering

The application of graph representation learning techniques to the area of financial risk management (FRM) has attracted significant attention recently. However, directly modeling transaction networks using graph neural models remains challenging: Firstly, transaction networks are directed multigraphs by nature, which could not be properly handled with most of the current off-the-shelf graph neural networks (GNN). Secondly, a crucial problem in FRM scenarios like anti-money laundering (AML) is to identify risky transactions and is most naturally cast into an edge classification problem with rich edge-level features, which are not fully exploited by the prevailing GNN design that follows node-centric message passing protocols. In this paper, we present a systematic investigation of design aspects of neural models over directed multigraphs and develop a novel GNN protocol that overcomes the above challenges via efficiently incorporating directional information, as well as proposing an enhancement that targets edge-related tasks using a novel message passing scheme over an extension of edge-to-node dual graph. A concrete GNN architecture called GRANDE is derived using the proposed protocol, with several further improvements and generalizations to temporal dynamic graphs. We apply the GRANDE model to both a real-world anti-money laundering task and public datasets. Experimental evaluations show the superiority of the proposed GRANDE architecture over recent state-of-the-art models on dynamic graph modeling and directed graph modeling.

cs.LG↗

arXiv2023

Benchmarking Large Language Models for News Summarization

Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, and not model size, is the key to the LLM's zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LMM summaries are judged to be on par with human written summaries.

cs.CL↗

arXiv2023

Voltage-Controlled Magnon Transistor via Tunning Interfacial Exchange Coupling

Magnon transistors that can effectively regulate magnon transport by an electric field are desired for magnonics which aims to provide a Joule-heating free alternative to the conventional electronics owing to the electric neutrality of magnons (the key carriers of spin-angular momenta in the magnonics). However, also due to their electric neutrality, magnons have no access to directly interact with an electric field and it is thus difficult to manipulate magnon transport by voltages straightforwardly. Here, we demonstrated a gate voltage ($V_{\rm g}$) applied on a nonmagnetic metal/magnetic insulator (NM/MI) interface that bended the energy band of the MI and then modulated the possibility for conduction electrons in the NM to tunnel into the MI can consequently enhance or weaken the spin-magnon conversion efficiency at the interface. A voltage-controlled magnon transistor based on the magnon-mediated electric current drag (MECD) effect in a Pt/Y$_{\rm 3}$Fe$_{\rm 5}$O$_{\rm 12}$ (YIG)/Pt sandwich was then experimentally realized with $V_{\rm g}$ modulating the magnitude of the MECD signal. The obtained efficiency (the change ratio between the MECD voltage at $\pm V_{\rm g}$) reached 10%/(MV/cm) at 300 K. This prototype of magnon transistor offers an effective scheme to control magnon transport by a gate voltage.

cond-mat.mes-hall↗

arXiv2022

LADIS: Language Disentanglement for 3D Shape Editing

Natural language interaction is a promising direction for democratizing 3D shape design. However, existing methods for text-driven 3D shape editing face challenges in producing decoupled, local edits to 3D shapes. We address this problem by learning disentangled latent representations that ground language in 3D geometry. To this end, we propose a complementary tool set including a novel network architecture, a disentanglement loss, and a new editing procedure. Additionally, to measure edit locality, we define a new metric that we call part-wise edit precision. We show that our method outperforms existing SOTA methods by 20% in terms of edit locality, and up to 6.6% in terms of language reference resolution accuracy. Our work suggests that by solely disentangling language representations, downstream 3D shape editing can become more local to relevant parts, even if the model was never given explicit part-based supervision.

cs.CV↗

arXiv2022

CREATIVESUMM: Shared Task on Automatic Summarization for Creative Writing

This paper introduces the shared task of summarizing documents in several creative domains, namely literary texts, movie scripts, and television scripts. Summarizing these creative documents requires making complex literary interpretations, as well as understanding non-trivial temporal dependencies in texts containing varied styles of plot development and narrative structure. This poses unique challenges and is yet underexplored for text summarization systems. In this shared task, we introduce four sub-tasks and their corresponding datasets, focusing on summarizing books, movie scripts, primetime television scripts, and daytime soap opera scripts. We detail the process of curating these datasets for the task, as well as the metrics used for the evaluation of the submissions. As part of the CREATIVESUMM workshop at COLING 2022, the shared task attracted 18 submissions in total. We discuss the submissions and the baselines for each sub-task in this paper, along with directions for facilitating future work in the field.

cs.CL↗