Support Thresholds, Not Algorithms, Limit Rare-Association Recovery in Co-Purchase Networks

Abstract

The support threshold of the Apriori algorithm involves a trade-off in conducting market basket analysis: the associations that occur frequently are noted with high threshold; however, the low ones lead to generating the large amount of rules. The paper compares five methods for co-purchase edge filtration on two grocery datasets: i.e., Instacart (3.2 million baskets) and Dunnhumby (208 thousand baskets), including Apriori, Apriori + lift post-filtering, top-$K$ ranking based on lift, and two methods based on networks, noise-corrected (NC) and disparity filter (DF). The top-$K$ method ensures the maximum average lift, while the NC achieves similar lift level by means of a single value of the significance parameter ($α$). These two methods recover substantially more rare high-lift associations than Apriori (80-100% against 22-28%). NC and top-$K$ select meaningfully different edges (18-29% non-overlapping): NC retains statistically validated pairs, while top-$K$ retains rare pairs with high lift but low statistical significance. A rolling-origin holdout evaluation shows that top-$K$ edges recur at higher rates at every split, but NC edges are ~12 pp more likely to remain statistically significant in the held-out network.