Ryan Amos | arXiv Science

arXiv2022

Reviews in motion: a large scale, longitudinal study of review recommendations on Yelp

The United Nations Consumer Protection Guidelines lists "access ... to adequate information ... to make informed choices" as a core consumer protection right. However, problematic online reviews and imperfections in algorithms that detect those reviews pose obstacles to the fulfillment of this right. Research on reviews and review platforms often derives insights from a single web crawl, but the decisions those crawls observe may not be static. A platform may feature a review one day and filter it from view the next day. An appreciation for these dynamics is necessary to understand how a platform chooses which reviews consumers encounter and which reviews may be unhelpful or suspicious. We introduce a novel longitudinal angle to the study of reviews. We focus on "reclassification," wherein a platform changes its filtering decision for a review. To that end, we perform repeated web crawls of Yelp to create three longitudinal datasets. These datasets highlight the platform's dynamic treatment of reviews. We compile over 12.5M reviews--more than 2M unique--across over 10k businesses. Our datasets are available for researchers to use. Our longitudinal approach gives us a unique perspective on Yelp's classifier and allows us to explore reclassification. We find that reviews routinely move between Yelp's two main classifier classes ("Recommended" and "Not Recommended")--up to 8% over eight years--raising concerns about prior works' use of Yelp's classes as ground truth. These changes have impacts on small scales; for example, a business going from a 3.5 to 4.5 star rating despite no new reviews. Some reviews move multiple times: we observed up to five reclassifications in eleven months. Our data suggests demographic disparities in reclassifications, with more changes in lower density and low-middle income areas.

cs.CY↗

arXiv2020

Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset

Automated analysis of privacy policies has proved a fruitful research direction, with developments such as automated policy summarization, question answering systems, and compliance detection. Prior research has been limited to analysis of privacy policies from a single point in time or from short spans of time, as researchers did not have access to a large-scale, longitudinal, curated dataset. To address this gap, we developed a crawler that discovers, downloads, and extracts archived privacy policies from the Internet Archive's Wayback Machine. Using the crawler and following a series of validation and quality control steps, we curated a dataset of 1,071,488 English language privacy policies, spanning over two decades and over 130,000 distinct websites. Our analyses of the data paint a troubling picture of the transparency and accessibility of privacy policies. By comparing the occurrence of tracking-related terminology in our dataset to prior web privacy measurements, we find that privacy policies have consistently failed to disclose the presence of common tracking technologies and third parties. We also find that over the last twenty years privacy policies have become even more difficult to read, doubling in length and increasing a full grade in the median reading level. Our data indicate that self-regulation for first-party websites has stagnated, while self-regulation for third parties has increased but is dominated by online advertising trade associations. Finally, we contribute to the literature on privacy regulation by demonstrating the historic impact of the GDPR on privacy policies.

cs.CY↗