arXiv subjects

Laura Dabbish

Publications and source records attributed to Laura Dabbish.

16 recordsLinked to original sources

arXiv2026

Towards Apples to Apples for AI Evaluations: From Real-World Use Cases to Evaluation Scenarios

AI measurement science has a wide variety of methodologies and measurements for comparing AI systems, resulting in what often appear to be "apples-to-oranges" comparisons across AI evaluations. To move toward "apples-to-apples" comparisons in real-world AI evaluations, this work advocates for methodological transparency in evaluation scenarios, operational grounding, and human-centered design (HCD) principles. We propose a repeatable process for transforming high-level use cases to detailed scenarios by eliciting use cases from subject matter experts (SMEs) via a structured AI Use Case Worksheet with six key elements: use case, sector, user (direct and indirect), intended outcomes, expected impacts (positive and negative), and KPIs and metrics. We demonstrate utility of the worksheet and process in the U.S. financial services sector. This paper reports on example high-level AI use cases identified by financial services sector SMEs: cyber defense enablement, developer productivity, financial crime aggregation, suspicious activity report (SAR) filing, credit memo generation, and internal call center support. These AI use cases provided are illustrative of the process and not exhaustive. Central to our work is a three-stage expansion pipeline combining LLM prompting with human reviews to generate 107 scenarios from those use cases elicited from SMEs. This process integrates iterative human reviews at every juncture to ensure operational grounding: for scenario titles and descriptions; for core scenario elements like users, benefits and risks, and metrics; and for scenario narratives and evaluation objectives. Human checkpoints ensure scenarios remain reflective of real-world usage and human needs. We describe a validation rubric to assess scenario quality. By defining key scenario components, this work supports a more consistent and meaningful paradigm for human-centered AI evaluations.

cs.HC↗

arXiv2026

Supporting Informed Self-Disclosure: Design Recommendations for Presenting AI-Estimates of Privacy Risks to Users

People candidly discuss sensitive topics online under the perceived safety of anonymity; yet, for many, this perceived safety is tenuous, as miscalibrated risk perceptions can lead to over-disclosure. Recent advances in Natural Language Processing (NLP) afford an unprecedented opportunity to present users with quantified disclosure-based re-identification risk (i.e., "population risk estimates", PREs). How can PREs be presented to users in a way that promotes informed decision-making, mitigating risk without encouraging unnecessary self-censorship? Using design fictions and comic-boarding, we story-boarded five design concepts for presenting PREs to users and evaluated them through an online survey with N = 44 Reddit users. We found participants had detailed conceptions of how PREs may impact risk awareness and motivation, but envisioned needing additional context and support to effectively interpret and act on risks. We distill our findings into four key design recommendations for how best to present users with quantified privacy risks to support informed disclosure decision-making.

cs.HC↗

arXiv2025

Worker Discretion Advised: Co-designing Risk Disclosure in Crowdsourced Responsible AI (RAI) Content Work

Responsible AI (RAI) content work, such as annotation, moderation, or red teaming for AI safety, often exposes crowd workers to potentially harmful content. While prior work has underscored the importance of communicating well-being risk to employed content moderators, designing effective disclosure mechanisms for crowd workers while balancing worker protection with the needs of task designers and platforms remains largely unexamined. To address this gap, we conducted individual co-design sessions with 15 task designers, 11 crowdworkers, and 3 platform representatives. We investigated task designer preferences for support in disclosing tasks, worker preferences for receiving risk disclosure warnings, and how platform representatives envision their role in shaping risk disclosure practices. We identify design tensions and map the sociotechnical tradeoffs that shape disclosure practices. We contribute design recommendations and feature concepts for risk disclosure mechanisms in the context of RAI content work.

cs.HC↗

arXiv2025

Locating Risk: Task Designers and the Challenge of Risk Disclosure in RAI Content Work

As AI systems are increasingly tested and deployed in open-ended and high-stakes domains, crowdworkers are often tasked with responsible AI (RAI) content work. These tasks include labeling violent content, moderating disturbing text, or simulating harmful behavior for red teaming exercises to shape AI system behaviors. While prior research efforts have highlighted the risks to worker well-being associated with RAI content work, far less attention has been paid to how these risks are communicated to workers by task designers or individuals who design and post RAI tasks. Existing transparency frameworks and guidelines, such as model cards, datasheets, and crowdworksheets, focus on documenting model information and dataset collection processes, but they overlook an important aspect of disclosing well-being risks to workers. In the absence of standard workflows or clear guidance, the consistent application of content warnings, consent flows, or other forms of well-being risk disclosure remains unclear. This study investigates how task designers approach risk disclosure in crowdsourced RAI tasks. Drawing on interviews with 23 task designers across academic and industry sectors, we examine how well-being risk is recognized, interpreted, and communicated in practice. Our findings highlight the need to support task designers in identifying and communicating risks not only to support crowdworker well-being but also to strengthen the ethical integrity and technical efficacy of AI development pipelines.

cs.HC↗

arXiv2025

Letters from Future Self: Augmenting the Letter-Exchange Exercise with LLM-based Agents to Enhance Young Adults' Career Exploration

Young adults often encounter challenges in career exploration. Self-guided interventions, such as the letter-exchange exercise, where participants envision and adopt the perspective of their future selves by exchanging letters with their envisioned future selves, can support career development. However, the broader adoption of such interventions may be limited without structured guidance. To address this, we integrated Large Language Model (LLM)-based agents that simulate participants' future selves into the letter-exchange exercise and evaluated their effectiveness. A one-week experiment (N=36) compared three conditions: (1) participants manually writing replies to themselves from the perspective of their future selves (baseline), (2) future-self agents generating letters to participants, and (3) future-self agents engaging in chat conversations with participants. Results indicated that exchanging letters with future-self agents enhanced participants' engagement during the exercise, while overall benefits of the intervention on future orientation, career self-concept, and psychological support remained comparable across conditions. We discuss design implications for AI-augmented interventions for supporting young adults' career exploration.

cs.HC↗

arXiv2025

Understanding the Challenges of Maker Entrepreneurship

The maker movement embodies a resurgence in DIY creation, merging physical craftsmanship and arts with digital technology support. However, mere technological skills and creativity are insufficient for economically and psychologically sustainable practice. By illuminating and smoothing the path from ``maker" to ``maker entrepreneur," we can help broaden the viability of making as a livelihood. Our research centers on makers who design, produce, and sell physical goods. In this work, we explore the transition to entrepreneurship for these makers and how technology can facilitate this transition online and offline. We present results from interviews with 20 USA-based maker entrepreneurs {(i.e., lamps, stickers)}, six creative service entrepreneurs {(i.e., photographers, fabrication)}, and seven support personnel (i.e., art curator, incubator director). Our findings reveal that many maker entrepreneurs 1) are makers first and entrepreneurs second; 2) struggle with business logistics and learn business skills as they go; and 3) are motivated by non-monetary values. We discuss training and technology-based design implications and opportunities for addressing challenges in developing economically sustainable businesses around making.

cs.HC↗

arXiv2024

Paradoxes of Openness and Trans Experiences in Open Source Software

In recent years, concerns have increased over the lack of contributor diversity in open source software (OSS), despite its status as a paragon of open collaboration. OSS is an important form of digital infrastructure and part of a career path for many developers. While there exists a growing body of literature on cisgender women's under-representation in OSS, the experiences of contributors from other marginalized groups are comparatively absent from the literature. Such is the case for trans contributors, a historically influential group in OSS. In this study, we interviewed 21 trans participants to understand and represent their experiences in the OSS literature. From their experiences, we theorize two related paradoxes of openness in OSS: the paradox of openness and display and the paradox of openness and governance. In an increasingly violent world for trans people, we draw on our theorizing to build recommendations for more inclusive and safer OSS projects for contributors.

cs.HC↗

arXiv2024

Power and Play: Investigating "License to Critique" in Teams' AI Ethics Discussions

Past work has sought to design AI ethics interventions--such as checklists or toolkits--to help practitioners design more ethical AI systems. However, other work demonstrates how these interventions may instead serve to limit critique to that addressed within the intervention, while rendering broader concerns illegitimate. In this paper, drawing on work examining how standards enact discursive closure and how power relations affect whether and how people raise critique, we recruit three corporate teams, and one activist team, each with prior context working with one another, to play a game designed to trigger broad discussion around AI ethics. We use this as a point of contrast to trigger reflection on their teams' past discussions, examining factors which may affect their "license to critique" in AI ethics discussions. We then report on how particular affordances of this game may influence discussion, and find that the hypothetical context created in the game is unlikely to be a viable mechanism for real world change. We discuss how power dynamics within a group and notions of "scope" affect whether people may be willing to raise critique in AI ethics discussions, and discuss our finding that games are unlikely to enable direct changes to products or practice, but may be more likely to allow members to find critically-aligned allies for future collective action.

cs.CY↗

arXiv2023

Beyond Fairness: Alternative Moral Dimensions for Assessing Algorithms and Designing Systems

The ethics of artificial intelligence (AI) systems has risen as an imminent concern across scholarly communities. This concern has propagated a great interest in algorithmic fairness. Large research agendas are now devoted to increasing algorithmic fairness, assessing algorithmic fairness, and understanding human perceptions of fairness. We argue that there is an overreliance on fairness as a single dimension of morality, which comes at the expense of other important human values. Drawing from moral psychology, we present five moral dimensions that go beyond fairness, and suggest three ways these alternative dimensions may contribute to ethical AI development.

cs.CY↗

arXiv2023

Nip it in the Bud: Moderation Strategies in Open Source Software Projects and the Role of Bots

Much of our modern digital infrastructure relies critically upon open sourced software. The communities responsible for building this cyberinfrastructure require maintenance and moderation, which is often supported by volunteer efforts. Moderation, as a non-technical form of labor, is a necessary but often overlooked task that maintainers undertake to sustain the community around an OSS project. This study examines the various structures and norms that support community moderation, describes the strategies moderators use to mitigate conflicts, and assesses how bots can play a role in assisting these processes. We interviewed 14 practitioners to uncover existing moderation practices and ways that automation can provide assistance. Our main contributions include a characterization of moderated content in OSS projects, moderation techniques, as well as perceptions of and recommendations for improving the automation of moderation tasks. We hope that these findings will inform the implementation of more effective moderation practices in open source communities.

cs.HC↗

arXiv2022

Experimental Evidence for Using a TTM Stages of Change Model in Boosting Progress Toward 2FA Adoption

Behavior change ideas from health psychology can also help boost end user compliance with security recommendations, such as adopting two-factor authentication (2FA). Our research adapts the Transtheoretical Model Stages of Change from health and wellness research to a cybersecurity context. We first create and validate an assessment to identify workers on Amazon Mechanical Turk who have not enabled 2FA for their accounts as being in Stage 1 (no intention to adopt 2FA) or Stages 2-3 (some intention to adopt 2FA). We randomly assigned participants to receive an informational intervention with varied content (highlighting process, norms, or both) or not. After three days, we again surveyed workers for Stage of Amazon 2FA adoption. We found that those in the intervention group showed more progress toward action/maintenance (Stages 4-5) than those in the control group, and those who received content highlighting the process of enabling 2FA were significantly more likely to progress toward 2FA adoption. Our work contributes support for applying a Stages of Change Model in usable security.

cs.HC↗

arXiv2022

Do They Accept or Resist Cybersecurity Measures? Development and Validation of the 13-Item Security Attitude Inventory (SA-13)

We present SA-13, the 13-item Security Attitude inventory. We develop and validate this assessment of cybersecurity attitudes by conducting an exploratory factor analysis, confirmatory factor analysis, and other tests with data from a U.S. Census-weighted Qualtrics panel (N=209). Beyond a core six indicators of Engagement with Security Measures (SA-Engagement, three items) and Attentiveness to Security Measures (SA-Attentiveness, three items), our SA-13 inventory adds indicators of Resistance to Security Measures (SA-Resistance, four items) and Concernedness with Improving Compliance (SA-Concernedness, three items). SA-13 and the subscales exhibit desirable psychometric qualities; and higher scores on SA-13 and on the SA-Engagement and SA-Attentiveness subscales are associated with higher scores for security behavior intention and for self-reported recent security behaviors. SA-13 and the subscales are useful for researchers and security awareness teams who need a lightweight survey measure of user security attitudes. The composite score of the 13 indicators provides a compact measurement of cybersecurity decisional balance.

cs.CR↗

arXiv2021

Significant Otter: Understanding the Role of Biosignals in Communication

With the growing ubiquity of wearable devices, sensed physiological responses provide new means to connect with others. While recent research demonstrates the expressive potential for biosignals, the value of sharing these personal data remains unclear. To understand their role in communication, we created Significant Otter, an Apple Watch/iPhone app that enables romantic partners to share and respond to each other's biosignals in the form of animated otter avatars. In a one-month study with 20 couples, participants used Significant Otter with biosignals sensing OFF and ON. We found that while sensing OFF enabled couples to keep in touch, sensing ON enabled easier and more authentic communication that fostered social connection. However, the addition of biosignals introduced concerns about autonomy and agency over the messages they sent. We discuss design implications and future directions for communication systems that recommend messages based on biosignals.

cs.HC↗

arXiv2020

Decentralized is not risk-free: Understanding public perceptions of privacy-utility trade-offs in COVID-19 contact-tracing apps

Contact-tracing apps have potential benefits in helping health authorities to act swiftly to halt the spread of COVID-19. However, their effectiveness is heavily dependent on their installation rate, which may be influenced by people's perceptions of the utility of these apps and any potential privacy risks due to the collection and releasing of sensitive user data (e.g., user identity and location). In this paper, we present a survey study that examined people's willingness to install six different contact-tracing apps after informing them of the risks and benefits of each design option (with a U.S.-only sample on Amazon Mechanical Turk, $N=208$). The six app designs covered two major design dimensions (centralized vs decentralized, basic contact tracing vs. also providing hotspot information), grounded in our analysis of existing contact-tracing app proposals. Contrary to assumptions of some prior work, we found that the majority of people in our sample preferred to install apps that use a centralized server for contact tracing, as they are more willing to allow a centralized authority to access the identity of app users rather than allowing tech-savvy users to infer the identity of diagnosed users. We also found that the majority of our sample preferred to install apps that share diagnosed users' recent locations in public places to show hotspots of infection. Our results suggest that apps using a centralized architecture with strong security protection to do basic contact tracing and providing users with other useful information such as hotspots of infection in public places may achieve a high adoption rate in the U.S.

cs.HC↗

arXiv2019

Animo: Sharing Biosignals on a Smartwatch for Lightweight Social Connection

We present Animo, a smartwatch app that enables people to share and view each other's biosignals. We designed and engineered Animo to explore new ground for smartwatch-based biosignals social computing systems: identifying opportunities where these systems can support lightweight and mood-centric interactions. In our work we develop, explore, and evaluate several innovative features designed for dyadic communication of heart rate. We discuss the results of a two-week study (N=34), including new communication patterns participants engaged in, and outline the design landscape for communicating with biosignals on smartwatches.

cs.HC↗

arXiv2014

Transparency and Coordination in Peer Production

This paper examines coordination in transparent work environments - environments where the content of work artifacts, and the actions taken on these artifacts, are fully visible to organizational members. Our qualitative study of a community of open source software developers revealed a coordination system characterized by interest-based, asynchronous interaction and knowledge transfer. At the core of asynchronous knowledge transfer, lies the concept of quasi-codification, which occurs when rich process knowledge is implicitly encoded in work artifacts. Our findings suggest that members are able to more selectively form dependencies, monitor the trajectory of projects, and make their work understandable to others which facilitates coordination. We discuss two important characteristics that enable coordination activities in a transparent environment: the presence of an imagined audience that dictates the way artifacts are crafted, and experience within the environment, that allows individuals to derive knowledge from these artifacts. By showing how transparency influences coordination, this research challenges previous conceptions of coordination for complex, collaborative work.

cs.HC↗