The new AI tools demonstrate strong potential to boost productivity and performance with measurable improvements in speed, quality, and outcomes. However, how AI agents will integrate effectively into existing or emerging workflows is unclear. A new preprint examines the limited knowledge of AI agents’ roles in workplace settings, more specifically, the who and how of productivity gains when humans collaborate with AI.
The study randomized team composition and AI agent personalities [1] to explore how AI affects collaboration dynamics, productivity, and output quality. Researchers developed an experimental platform enabling real-time collaborative teamwork between humans and AI agents on a complex, creative task – designing marketing ads that garnered nearly 5 million online views, enabling detailed performance analysis from human real-world engagement metrics.
Participants first completed a questionnaire measuring their personality traits and were paired, in a blinded manner, with either a human or AI partner. Paired participants had 40 minutes to collaboratively produce and submit as many ads as possible. Immediately after the ad task ended, participants completed a second survey assessing their collaboration experience and questions about their perception and experience with AI.
More Ads, Less Small Talk
Collaborating with AI agents significantly reshaped teamwork, enhancing productivity and altering communication dynamics compared to human-only teams.
- Human-AI teams communicated 45% more frequently compared to human-human teams. When the content of the messages was characterized [2], human-human teams emphasized social and emotional communication (e.g., rapport building, expressions of concern, humor). In contrast, human-AI teams primarily exchanged task-focused and process-oriented messages. For example, human-AI collaboration focused more on giving instructions or suggestions than directly editing content.
- Collaborating with AI agents focused on tasks by reducing the social coordination needed in human-only teams, resulting in improved completion rates of specific task elements, such as ad copy. This was especially true for individuals who might typically struggle, but not so much for high performers – AI effectively enhances “performance for lower-performing participants.”
- Individual personality traits influence communication patterns. AI personality had significant but nuanced interactive effects with the personalities of human collaborators – much as we might expect from human-human interactions.
More open and extroverted participants communicated more frequently, while the more “agreeable” communicated less when interacting with AI. A conscientious AI notably increased communication, but only when paired with its synergetic conscientious humans. Paired with a less conscientious human willing to simply accept what was offered resulted in poorer collaborative output. Agreeable AI significantly boosted productivity when paired with extroverted or neurotic humans. Extraverted AIs benefited human extraverts but, in a sense, “steamrolled” more agreeable, compliant humans. These differences highlight the importance of pairing compatible personalities
When the ads were run on social media, Human-AI and Human-Human teams showed comparable real-world performance in cost-per-click, click-through rate, and view metrics. Ad effectiveness depended primarily on content quality, with higher text quality boosting click-through rates [3] and viewer engagement durations.
The study shows that integrating GPT-based AI agents into human workflows changes how teams communicate and work together. One key benefit is that AI agents encourage more task-focused interactions, cutting down people's time and effort in socially coordinating. This shift helps individuals become more productive, especially those with skill levels that might otherwise struggle. The study also highlights that aligning AI “persona” with those of their human users can boost productivity and creativity. Yet, this customization requires careful consideration to ensure compatibility and maximize effectiveness. Finally, AI tends to fall short in other tasks at the periphery or outside its training.
AI in Healthcare: Helping, Hurting, or Hovering Over Your Shoulder?
While designing click-worthy ads with AI may seem far removed from the world of patient care, the lessons about collaboration, communication, and competence have real implications for healthcare. Generative AI is being hailed as the next big breakthrough. But as with any powerful tool, there are risks.
The first foray of digital technology into healthcare, the electronic health record (EHR), has spawned at least as much harm as the benefits it promised (but has failed to deliver). While digital collaboration has benefited those involved in billing, incentivizing, and paying, it has not played well with its human collaborators –introducing new burdens for clinicians, contributing to burnout, and shifting attention away from the patient experience.
The aviation industry learned about automation’s downside the hard way. The Federal Aviation Agency has our most significant experience with AI in the form of autopilot technology. After decades of reliance on autopilot systems, the FAA issued guidance reminding pilots to keep flying manually when possible—because skills not practiced are quickly lost.
“The Federal Aviation Administration (FAA) believes maintaining and improving the knowledge and skills needed for manual flight operations is necessary for safe flight operations. … [Pilots are encouraged] to manually fly the aircraft when conditions permit, including at least periodically, the entire departure and arrival phases, and potentially the entire flight, if/when practicable and permissible.”
- Safety Advisory To Operators SAFO 17007
Healthcare may need to take a similar cue as it integrates AI into clinical workflows.
Artificial intelligence is the digirati’s latest promise to transform healthcare by improving productivity and supporting clinicians at every level. However, this study and our experience with EHRs should caution us that its integration into clinical workflows must be approached thoughtfully to maximize benefits and minimize unintended risks such as deskilling.
Highly experienced physicians, whose expertise has been honed through years of specialized practice, stand to gain from AI in automating routine tasks like charting, diagnostics, and administrative documentation. AI allows these experts more time to concentrate on complex patient care decisions. These seasoned clinicians' extensive experience and depth of practice shield them from becoming overly reliant on AI.
AI can serve as an immediate mentor for the growing cadre of trained but inexperienced providers, providing real-time support, enhancing confidence, and accelerating their development towards baseline clinical competence. Yet, collaborating with AI is not the same as individual thought, nor is AI guidance capable of helping with “edge cases,” which are challenging even to the most experienced human clinicians. An over-dependence on AI can impede the cultivation of foundational clinical reasoning and critical thinking skills. These novices might become highly proficient in using AI tools without truly developing the independent judgment necessary for robust medical practice.
The greatest challenge, however, lies with the “journeymen,” the substantial group of moderately experienced clinicians, neither apprentice nor master, trainee or expert, who represent the backbone of the healthcare workforce. AI promises these clinicians reduced workload, increased productivity, and improved diagnostic accuracy as AI increasingly handles moderately skilled tasks. However, increasing reliance on AI for clinical judgments or procedural guidance may diminish their independent decision-making capabilities and confidence over time.
Will health systems mandate practices similar to the FAA and find ways to integrate AI into healthcare workflows to capture benefits without causing widespread skill dependency and erosion? Or will they adopt the ethos of moving fast, breaking things, and apologizing later?
[1]. The “Big Five personality traits” are a widely accepted psychological framework for describing human personality. The five traits include openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism.
[2] Process, Content, Social, Emotional, Feedback,
[3] The click-through rate was 0.1%, which gives you a sense of how many ads are necessary to make a connection and why we face a tsunami of them.
Source: Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performance ArXiv DOI: 10.48550/arXiv.2503.18238
Chuck Dinerstein, MD, MBA
Director of Medicine
Dr. Charles Dinerstein, M.D., MBA, FACS is Director of Medicine at the American Council on Science and Health. He has over 25 years of experience as a vascular surgeon.
