arXiv subjects

Tianyi Zhang

Publications and source records attributed to Tianyi Zhang.

At least 343 records · Page 19Linked to original sources

arXiv2022

Coder Reviewer Reranking for Code Generation

Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions. Inspired by collaborative programming, we propose Coder-Reviewer reranking. We augment Coder language models from past work, which generate programs given language instructions, with Reviewer models, which evaluate the likelihood of the instruction given the generated programs. We perform an extensive study across six datasets with eight models from three model families. Experimental results show that Coder-Reviewer reranking leads to consistent and significant improvement (up to 17% absolute accuracy gain) over reranking with the Coder model only. When combined with executability filtering, Coder-Reviewer reranking can often outperform the minimum Bayes risk method. Coder-Reviewer reranking is easy to implement by prompting, can generalize to different programming languages, and works well with off-the-shelf hyperparameters.

cs.LG↗

arXiv2022

DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation

We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. Compared to prior works, DS-1000 incorporates three core features. First, our problems reflect diverse, realistic, and practical use cases since we collected them from StackOverflow. Second, our automatic evaluation is highly specific (reliable) -- across all Codex-002-predicted solutions that our evaluation accept, only 1.8% of them are incorrect; we achieve this with multi-criteria metrics, checking both functional correctness by running test cases and surface-form constraints by restricting API usages or keywords. Finally, we proactively defend against memorization by slightly modifying our problems to be different from the original StackOverflow source; consequently, models cannot answer them correctly by memorizing the solutions from pre-training. The current best public system (Codex-002) achieves 43.3% accuracy, leaving ample room for improvement. We release our benchmark at https://ds1000-code-gen.github.io.

cs.SE↗

arXiv2022

Multi-class Classification from Multiple Unlabeled Datasets with Partial Risk Regularization

Recent years have witnessed a great success of supervised deep learning, where predictive models were trained from a large amount of fully labeled data. However, in practice, labeling such big data can be very costly and may not even be possible for privacy reasons. Therefore, in this paper, we aim to learn an accurate classifier without any class labels. More specifically, we consider the case where multiple sets of unlabeled data and only their class priors, i.e., the proportions of each class, are available. Under this problem setup, we first derive an unbiased estimator of the classification risk that can be estimated from the given unlabeled sets and theoretically analyze the generalization error of the learned classifier. We then find that the classifier obtained as such tends to cause overfitting as its empirical risks go negative during training. To prevent overfitting, we further propose a partial risk regularization that maintains the partial risks with respect to unlabeled datasets and classes to certain levels. Experiments demonstrate that our method effectively mitigates overfitting and outperforms state-of-the-art methods for learning from multiple unlabeled sets.

cs.LG↗

arXiv2022

A Declarative Metamorphic Testing Framework for Autonomous Driving

Autonomous driving has gained much attention from both industry and academia. Currently, Deep Neural Networks (DNNs) are widely used for perception and control in autonomous driving. However, several fatal accidents caused by autonomous vehicles have raised serious safety concerns about autonomous driving models. Some recent studies have successfully used the metamorphic testing technique to detect thousands of potential issues in some popularly used autonomous driving models. However, prior study is limited to a small set of metamorphic relations, which do not reflect rich, real-world traffic scenarios and are also not customizable. This paper presents a novel declarative rule-based metamorphic testing framework called RMT. RMT provides a rule template with natural language syntax, allowing users to flexibly specify an enriched set of testing scenarios based on real-world traffic rules and domain knowledge. RMT automatically parses human-written rules to metamorphic relations using an NLP-based rule parser referring to an ontology list and generates test cases with a variety of image transformation engines. We evaluated RMT on three autonomous driving models. With an enriched set of metamorphic relations, RMT detected a significant number of abnormal model predictions that were not detected by prior work. Through a large-scale human study on Amazon Mechanical Turk, we further confirmed the authenticity of test cases generated by RMT and the validity of detected abnormal model predictions.

cs.SE↗

arXiv2022

Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?

Autonomous driving has shown great potential to reform modern transportation. Yet its reliability and safety have drawn a lot of attention and concerns. Compared with traditional software systems, autonomous driving systems (ADSs) often use deep neural networks in tandem with logic-based modules. This new paradigm poses unique challenges for software testing. Despite the recent development of new ADS testing techniques, it is not clear to what extent those techniques have addressed the needs of ADS practitioners. To fill this gap, we present the first comprehensive study to identify the current practices and needs of ADS testing. We conducted semi-structured interviews with developers from 10 autonomous driving companies and surveyed 100 developers who have worked on autonomous driving systems. A systematic analysis of the interview and survey data revealed 7 common practices and 4 emerging needs of autonomous driving testing. Through a comprehensive literature review, we developed a taxonomy of existing ADS testing techniques and analyzed the gap between ADS research and practitioners' needs. Finally, we proposed several future directions for SE researchers, such as developing test reduction techniques to accelerate simulation-based ADS testing.

cs.SE↗

arXiv2022

Scenario-Based Test Reduction and Prioritization for Multi-Module Autonomous Driving Systems

When developing autonomous driving systems (ADS), developers often need to replay previously collected driving recordings to check the correctness of newly introduced changes to the system. However, simply replaying the entire recording is not necessary given the high redundancy of driving scenes in a recording (e.g., keeping the same lane for 10 minutes on a highway). In this paper, we propose a novel test reduction and prioritization approach for multi-module ADS. First, our approach automatically encodes frames in a driving recording to feature vectors based on a driving scene schema. Then, the given recording is sliced into segments based on the similarity of consecutive vectors. Lengthy segments are truncated to reduce the length of a recording and redundant segments with the same vector are removed. The remaining segments are prioritized based on both the coverage and the rarity of driving scenes. We implemented this approach on an industry level, multi-module ADS called Apollo and evaluated it on three road maps in various regression settings. The results show that our approach significantly reduced the original recordings by over 34% while keeping comparable test effectiveness, identifying almost all injected faults. Furthermore, our test prioritization method achieves about 22% to 39% and 41% to 53% improvements over three baselines in terms of both the average percentage of faults detected (APFD) and TOP-K.

cs.SE↗

arXiv2022

Shuffle Instances-based Vision Transformer for Pancreatic Cancer ROSE Image Classification

The rapid on-site evaluation (ROSE) technique can signifi-cantly accelerate the diagnosis of pancreatic cancer by im-mediately analyzing the fast-stained cytopathological images. Computer-aided diagnosis (CAD) can potentially address the shortage of pathologists in ROSE. However, the cancerous patterns vary significantly between different samples, making the CAD task extremely challenging. Besides, the ROSE images have complicated perturbations regarding color distribution, brightness, and contrast due to different staining qualities and various acquisition device types. To address these challenges, we proposed a shuffle instances-based Vision Transformer (SI-ViT) approach, which can reduce the perturbations and enhance the modeling among the instances. With the regrouped bags of shuffle instances and their bag-level soft labels, the approach utilizes a regression head to make the model focus on the cells rather than various perturbations. Simultaneously, combined with a classification head, the model can effectively identify the general distributive patterns among different instances. The results demonstrate significant improvements in the classification accuracy with more accurate attention regions, indicating that the diverse patterns of ROSE images are effectively extracted, and the complicated perturbations are significantly reduced. It also suggests that the SI-ViT has excellent potential in analyzing cytopathological images. The code and experimental results are available at https://github.com/sagizty/MIL-SI.

eess.IV↗

arXiv2022

Fusion rules for pastures and tracts

Baker and Bowler defined a category of algebraic objects called tracts which generalize both partial fields and hyperfields. They also defined a notion of weak and strong matroids over a tract $F$, and proved that if $F$ is perfect, meaning that $F$-vectors and $F$-covectors are orthogonal for every matroid over $F$, then the notions of weak and strong $F$-matroids coincide. We define the class of strongly fused tracts and prove that such tracts are perfect. We in fact prove a more general result which implies that given a tract $F$, there is a tract $σ(F)$ with the same 3-term additive relations as $F$ such that weak $F$-matroids coincide with strong $σ(F)$-matroids. We also show that both partial fields and stringent hyperfields are strongly fused; in this way, our criterion for perfection generalizes results of Baker-Bowler and Bowler-Pendavingh.

math.CO↗

arXiv2022

Faster Cut-Equivalent Trees in Simple Graphs

Let $G = (V, E)$ be an undirected connected simple graph on $n$ vertices. A cut-equivalent tree of $G$ is an edge-weighted tree on the same vertex set $V$, such that for any pair of vertices $s, t\in V$, the minimum $(s, t)$-cut in the tree is also a minimum $(s, t)$-cut in $G$, and these two cuts have the same cut value. In a recent paper [Abboud, Krauthgamer and Trabelsi, 2021], the authors propose the first subcubic time algorithm for constructing a cut-equivalent tree. More specifically, their algorithm has $\widetilde{O}(n^{2.5})$ running time. In this paper, we improve the running time to $\hat{O}(n^2)$ if almost-linear time max-flow algorithms exist. Also, using the currently fastest max-flow algorithm by [van den Brand et al, 2021], our algorithm runs in time $\widetilde{O}(n^{17/8})$.

cs.DS↗

arXiv2022

Faster Min-Plus Product for Monotone Instances

In this paper, we show that the time complexity of monotone min-plus product of two $n\times n$ matrices is $\tilde{O}(n^{(3+ω)/2})=\tilde{O}(n^{2.687})$, where $ω< 2.373$ is the fast matrix multiplication exponent [Alman and Vassilevska Williams 2021]. That is, when $A$ is an arbitrary integer matrix and $B$ is either row-monotone or column-monotone with integer elements bounded by $O(n)$, computing the min-plus product $C$ where $C_{i,j}=\min_k\{A_{i,k}+B_{k,j}\}$ takes $\tilde{O}(n^{(3+ω)/2})$ time, which greatly improves the previous time bound of $\tilde{O}(n^{(12+ω)/5})=\tilde{O}(n^{2.875})$ [Gu, Polak, Vassilevska Williams and Xu 2021]. Then by simple reductions, this means the following problems also have $\tilde{O}(n^{(3+ω)/2})$ time algorithms: (1) $A$ and $B$ are both bounded-difference, that is, the difference between any two adjacent entries is a constant. The previous results give time complexities of $\tilde{O}(n^{2.824})$ [Bringmann, Grandoni, Saha and Vassilevska Williams 2016] and $\tilde{O}(n^{2.779})$ [Chi, Duan and Xie 2022]. (2) $A$ is arbitrary and the columns or rows of $B$ are bounded-difference. Previous result gives time complexity of $\tilde{O}(n^{2.922})$ [Bringmann, Grandoni, Saha and Vassilevska Williams 2016]. (3) The problems reducible to these problems, such as language edit distance, RNA-folding, scored parsing problem on BD grammars. [Bringmann, Grandoni, Saha and Vassilevska Williams 2016]. Finally, we also consider the problem of min-plus convolution between two integral sequences which are monotone and bounded by $O(n)$, and achieve a running time upper bound of $\tilde{O}(n^{1.5})$. Previously, this task requires running time $\tilde{O}(n^{(9+\sqrt{177})/12}) = O(n^{1.859})$ [Chan and Lewenstein 2015].

cs.DS↗

arXiv2022

Pancreatic Cancer ROSE Image Classification Based on Multiple Instance Learning with Shuffle Instances

The rapid on-site evaluation (ROSE) technique can significantly ac-celerate the diagnostic workflow of pancreatic cancer by immediately analyzing the fast-stained cytopathological images with on-site pathologists. Computer-aided diagnosis (CAD) using the deep learning method has the potential to solve the problem of insufficient pathology staffing. However, the cancerous patterns of ROSE images vary greatly between different samples, making the CAD task extremely challenging. Besides, due to different staining qualities and various types of acquisition devices, the ROSE images also have compli-cated perturbations in terms of color distribution, brightness, and contrast. To address these challenges, we proposed a novel multiple instance learning (MIL) approach using shuffle patches containing the instances, which adopts the patch-based learning strategy of Vision Transformers. With the re-grouped bags of shuffle instances and their bag-level soft labels, the approach utilizes a MIL head to make the model focus on the features from the pancreatic cancer cells, rather than that from various perturbations in ROSE images. Simultaneously, combined with a classification head, the model can effectively identify the gen-eral distributive patterns across different instances. The results demonstrate the significant improvements in the classification accuracy with more accurate at-tention regions, indicating that the diverse patterns of ROSE images are effec-tively extracted, and the complicated perturbations of ROSE images are signifi-cantly eliminated. It also suggests that the MIL with shuffle instances has great potential in the analysis of cytopathological images.

eess.IV↗

arXiv2022

TempLM: Distilling Language Models into Template-Based Generators

While pretrained language models (PLMs) have greatly improved text generation, they have also been known to produce unfaithful or inappropriate content. In contrast, classic template-based systems provide strong guarantees of faithfulness at the cost of fluency. We propose TempLM, which achieves the best of both worlds by distilling a PLM into a template-based generator. On the E2E and SynthBio data-to-text datasets, we show that TempLM is more faithful than the original PLM and is more fluent than prior template systems. Notably, on an out-of-domain evaluation, TempLM reduces a finetuned BART model's unfaithfulness rate from 83% to 0%. In a human study, we find that TempLM's templates substantially improve upon human-written ones in BERTScore.

cs.CL↗

arXiv2022

Learning Cross-Scale Visual Representations for Real-Time Image Geo-Localization

Robot localization remains a challenging task in GPS denied environments. State estimation approaches based on local sensors, e.g. cameras or IMUs, are drifting-prone for long-range missions as error accumulates. In this study, we aim to address this problem by localizing image observations in a 2D multi-modal geospatial map. We introduce the cross-scale dataset and a methodology to produce additional data from cross-modality sources. We propose a framework that learns cross-scale visual representations without supervision. Experiments are conducted on data from two different domains, underwater and aerial. In contrast to existing studies in cross-view image geo-localization, our approach a) performs better on smaller-scale multi-modal maps; b) is more computationally efficient for real-time applications; c) can serve directly in concert with state estimation pipelines.

cs.CV↗

arXiv2022

Autonomous Investigations over WS$_2$ and Au{111} with Scanning Probe Microscopy

Individual atomic defects in 2D materials impact their macroscopic functionality. Correlating the interplay is challenging, however, intelligent hyperspectral scanning tunneling spectroscopy (STS) mapping provides a feasible solution to this technically difficult and time consuming problem. Here, dense spectroscopic volume is collected autonomously via Gaussian process regression, where convolutional neural networks are used in tandem for spectral identification. Acquired data enable defect segmentation, and a workflow is provided for machine-driven decision making during experimentation with capability for user customization. We provide a means towards autonomous experimentation for the benefit of both enhanced reproducibility and user-accessibility. Hyperspectral investigations on WS$_2$ sulfur vacancy sites are explored, which is combined with local density of states confirmation on the Au{111} herringbone reconstruction. Chalcogen vacancies, pristine WS$_2$, Au face-centered cubic, and Au hexagonal close packed regions are examined and detected by machine learning methods to demonstrate the potential of artificial intelligence for hyperspectral STS mapping.

cond-mat.mtrl-sci↗

arXiv2022

When Cyber-Physical Systems Meet AI: A Benchmark, an Evaluation, and a Way Forward

Cyber-physical systems (CPS) have been broadly deployed in safety-critical domains, such as automotive systems, avionics, medical devices, etc. In recent years, Artificial Intelligence (AI) has been increasingly adopted to control CPS. Despite the popularity of AI-enabled CPS, few benchmarks are publicly available. There is also a lack of deep understanding on the performance and reliability of AI-enabled CPS across different industrial domains. To bridge this gap, we initiate to create a public benchmark of industry-level CPS in seven domains and build AI controllers for them via state-of-the-art deep reinforcement learning (DRL) methods. Based on that, we further perform a systematic evaluation of these AI-enabled systems with their traditional counterparts to identify the current challenges and explore future opportunities. Our key findings include (1) AI controllers do not always outperform traditional controllers, (2) existing CPS testing techniques (falsification, specifically) fall short of analyzing AI-enabled CPS, and (3) building a hybrid system that strategically combines and switches between AI controllers and traditional controllers can achieve better performance across different domains. Our results highlight the need for new testing techniques for AI-enabled CPS and the need for more investigations into hybrid CPS systems to achieve optimal performance and reliability.

cs.SE↗

arXiv2022

Channel Estimation and Projection for RIS-assisted MIMO Using Zadoff-Chu Sequences

The reconfigurable intelligent surface (RIS) technology is a promising enabler for millimeter wave (mmWave) wireless communications, as it can potentially provide spectral efficiency comparable to the conventional massive multiple-input multiple-output (MIMO) but with significantly lower hardware complexity. In this paper, we focus on the estimation and projection of the uplink RIS-aided massive MIMO channel, which can be time-varying. We propose to let the user equipments (UE) transmit Zadoff-Chu (ZC) sequences and let the base station (BS) conduct maximum likelihood (ML) estimation of the uplink channel. The proposed scheme is computationally efficient: it uses ZC sequences to decouple the estimation of the frequency and time offsets; it uses the space-alternating generalized expectation-maximization (SAGE) method to reduce the high-dimensional problem due to the multipaths to multiple lower-dimensional ones per path. Owing to the estimation of the Doppler frequency offsets, the time-varying channel state can be projected, which can significantly lower the overhead of the pilots for channel estimation. The numerical simulations verify the effectiveness of the proposed scheme.

cs.IT↗

arXiv2022

Model-Based Neural Network and Its Application to Line Spectral Estimation

This paper presents the concept of "model-based neural network"(MNN), which is inspired by the classic artificial neural network (ANN) but for different usages. Instead of being used as a data-driven classifier, a MNN serves as a modeling tool with artfully defined inputs, outputs, and activation functions which have explicit physical meanings. Owing to the same layered form as an ANN, a MNN can also be optimized using the back-propagation (BP) algorithm. As an interesting application, the classic problem of line spectral estimation can be modeled by a MNN. We propose to first initialize the MNN by the fast Fourier transform (FFT) based spectral estimation, and then optimize the MNN by the BP algorithm, which automatically yields the maximum likelihood (ML) parameter estimation of the frequency spectrum. We also design a method of merging and pruning the hidden-layer nodes of the MNN, which can be used for model-order selection, i.e., to estimate the number of sinusoids. Numerical simulations verify the effectiveness of the proposed method.

eess.SP↗

arXiv2021

MSHT: Multi-stage Hybrid Transformer for the ROSE Image Analysis of Pancreatic Cancer

Pancreatic cancer is one of the most malignant cancers in the world, which deteriorates rapidly with very high mortality. The rapid on-site evaluation (ROSE) technique innovates the workflow by immediately analyzing the fast stained cytopathological images with on-site pathologists, which enables faster diagnosis in this time-pressured process. However, the wider expansion of ROSE diagnosis has been hindered by the lack of experienced pathologists. To overcome this problem, we propose a hybrid high-performance deep learning model to enable the automated workflow, thus freeing the occupation of the valuable time of pathologists. By firstly introducing the Transformer block into this field with our particular multi-stage hybrid design, the spatial features generated by the convolutional neural network (CNN) significantly enhance the Transformer global modeling. Turning multi-stage spatial features as global attention guidance, this design combines the robustness from the inductive bias of CNN with the sophisticated global modeling power of Transformer. A dataset of 4240 ROSE images is collected to evaluate the method in this unexplored field. The proposed multi-stage hybrid Transformer (MSHT) achieves 95.68% in classification accuracy, which is distinctively higher than the state-of-the-art models. Facing the need for interpretability, MSHT outperforms its counterparts with more accurate attention regions. The results demonstrate that the MSHT can distinguish cancer samples accurately at an unprecedented image scale, laying the foundation for deploying automatic decision systems and enabling the expansion of ROSE in clinical practice. The code and records are available at: https://github.com/sagizty/Multi-Stage-Hybrid-Transformer.

eess.IV↗