A physician looked at my demo screen and asked a question that changed everything.
“So you’re building a tool that turns me into a widget on a factory line?”
We were showing an analytics dashboard. Productivity metrics. Documentation completion times. Patient throughput per hour. On paper, it made perfect sense. Healthcare administrators love data.
But that doctor saw something I’d missed.
We’d accidentally built surveillance disguised as support.
The Technology That Works On You
I’ve been in healthcare technology for 2 years. Built MediLogix from the ground up. Won awards for AI-driven documentation solutions.
And I almost made the worst mistake possible.
The line between helpful AI and invasive surveillance isn’t about the technology itself. It’s about intention. When we capture physician conversations in real-time, analyze tonality, extract clinical context, we’re seeing people at their most human and vulnerable.
The question isn’t can we capture this data.
The question is should we, and what happens when we do.
That physician’s “widget” comment revealed something fundamental. Doctors didn’t go to medical school to become productivity units. They went to heal people. When technology treats them like assembly line workers, it threatens their professional identity at the core.
The psychological contract is simple: “I’ll adopt your technology if it helps me be a better doctor, not if it turns me into a better employee.”
There’s a massive difference.
When Personalization Becomes Surveillance
Here’s what nobody talks about in the AI personalization race.
57% of consumers are anxious that brands using AI would put their personal data at risk. Yet every AI company is chasing more data points, better predictions, deeper personalization.
The economic pressure is enormous. Every conversation we record makes our AI smarter. Every emotional pattern we track improves accuracy. Our competitors are absolutely using aggregated physician and patient data to train their models.
It gives them a technical advantage.
But it’s a Faustian bargain.
We made a deliberate architectural decision early on. Our AI learns from anonymized clinical language patterns, medical terminology, specialty-specific workflows. But not from individual physician behavior or emotional patterns.
The tonality analysis happens in real-time during transcription. It helps our human medical transcriptionists smooth language so documentation reads professionally while preserving clinical accuracy.
Then that emotional data disappears.
It’s not retained. Not fed back into training datasets. Not tracked over time. Not reported to administrators.
That decision costs us. We could build more sophisticated systems if we tracked how individual physicians communicate over time, identified their patterns, predicted their preferences. The AI would get better faster.
But we’d be creating a data asset that could be subpoenaed, hacked, or sold if the company ever changed hands.
I ask myself one question: “If a physician knew exactly how we were using their data, would they still trust us in the exam room?”
If the answer is anything other than an absolute yes, we don’t do it.
The Deal I Walked Away From
A large hospital system came to us a few years back. The CFO wanted real-time productivity tracking. Alerts when physicians were “falling behind schedule.” Dashboards showing who was documenting fastest.
He wanted to turn our system into a performance management tool.
I told him: “If you implement it that way, your physicians will hate it. Within six months they’ll either stop using it or start looking for jobs elsewhere.”
He kept saying, “We need accountability. We need to know our physicians are being efficient.”
When administrators use words like “accountability” and “visibility into behavior,” they’re often asking for control and predictability in an environment that feels chaotic. Healthcare is complex, margins are thin, pressure is enormous.
But they’re trying to apply manufacturing principles to human relationships.
“Accountability” usually means “I want to measure whether physicians are working hard enough.” “Visibility” means “I want to see if they’re wasting time.”
It’s rooted in fundamental mistrust.
We offered an alternative. Give physicians time savings and autonomy. Track outcomes like claim denial rates and patient satisfaction instead of speed metrics. Let efficiency gains speak for themselves.
He wasn’t interested.
We walked away from that deal. It was a significant contract. My VP of Sales came into my office afterward and said, “Mark, we have payroll to meet. You just turned down six figures because of a philosophical disagreement.”
He wasn’t wrong.
But my father taught me something in business: you can’t unring a bell.
Once you build surveillance features, once you cross that line, you can’t go back and say, “Actually, we’ve decided to respect privacy now.” Your code is out there. Your reputation is set. You’ve attracted clients who want those features.
You become trapped serving a market you never wanted to serve.
The Bells Already Rung
Some bells have already been rung in healthcare AI, and we’re going to regret them.
First: predictive analytics for patient risk stratification that determines who gets care and who doesn’t. AI systems score patients based on likelihood to comply with treatment, probability of readmission, projected healthcare costs.
Hospitals use those scores to decide who gets access to programs, who gets flagged for monitoring, who gets deprioritized.
The problem? Those algorithms are trained on historical data reflecting decades of healthcare disparities. Research shows that at a given risk score, Black patients are considerably sicker than White patients because the algorithm predicts healthcare costs rather than illness.
We’ve automated discrimination and called it efficiency.
Second: ambient clinical documentation that records everything in the exam room without explicit, ongoing consent. Some systems capture entire patient visits. The clinical conversation, the small talk, the family member in the background, the patient’s emotional reactions.
All of it gets transcribed, analyzed, stored.
Patients often don’t fully understand what they’ve consented to. There’s no easy way to say “stop recording now” in the middle of a vulnerable conversation.
We’ve normalized surveillance in the one place that should be absolutely sacred.
Third: the consolidation of health data into massive commercial databases. Companies aggregate EHR data, claims data, prescription data, social determinants of health. They sell access to researchers, pharma companies, insurers.
The data is “de-identified,” but re-identification is possible with enough data points.
We’ve turned patient health information into a commodity. Most patients have no idea their data is being bought and sold.
Why Trust Beats Data Extraction
Here’s what I tell investors when they ask why we’re “handicapping” our AI by not using all available data.
Accuracy isn’t just about algorithmic performance. It’s about whether the system gets used at all.
You can have the most accurate AI in the world. But if physicians don’t trust it enough to adopt it, your accuracy is irrelevant.
We’ve seen this with EHR systems. Technically sophisticated products that physicians despise and work around because they were built without regard for trust or workflow.
84% of physicians reported that AI scribes positively affected communication with patients. Their overall work satisfaction improved. But only when the technology served them rather than monitored them.
In healthcare, trust is the ultimate competitive moat. It’s harder to build and easier to lose than any technical advantage.
Five years from now, if competitors have marginally better AI performance because they extracted more data, but physicians are raising privacy concerns and regulators are investigating their data practices, we’ll be positioned as the trusted alternative.
Healthcare is moving toward stricter data privacy regulations. HIPAA is just the beginning. The EU’s AI Act, state-level privacy laws, patient rights movements. The regulatory environment is tightening, not loosening.
Companies that built their competitive advantage on data extraction are going to face a reckoning.
We’re building a company that can survive and thrive in that future because our foundation is trust, not data exploitation.
The Physician Burnout We’re Creating
Here’s the uncomfortable truth about AI efficiency gains.
When we save a physician 1-3 hours a day on documentation, what happens to that time?
If volume-driven financial incentives remain unchanged, that freed-up time gets used to see more patients. The efficiency improvement becomes a productivity increase. The physician still goes home exhausted, just having processed more people through the system.
We’ve optimized the wrong thing.
Physician burnout affects 50% of physicians. Excessive workloads and administrative burdens related to electronic medical records hamper productivity. They spend between 34 and 55 percent of their workday compiling clinical documentation.
AI can help. But only if we’re honest about what we’re optimizing for.
Are we optimizing for sustainable, high-quality care? Or short-term productivity metrics that burn people out and ultimately cost more?
When we built MediLogix, we made a choice. We lead with the physician experience because that’s where trust is built or broken. But we make sure administrators see measurable ROI.
Physicians document faster and more accurately. Claim denials drop. Patient satisfaction goes up. Turnover decreases.
We don’t navigate the conflict by choosing sides. We align the incentives.
The Bells We Haven’t Rung Yet
The bells I’m most worried about are the ones we haven’t rung yet.
AI systems that can predict physician burnout based on documentation patterns. Algorithms that detect “problem” physicians based on tonality analysis. Systems that optimize patient scheduling based on profitability rather than clinical need.
Those are coming.
We have a narrow window to decide whether we’re going to allow them or draw a line.
That’s why I’m adamant about building MediLogix the right way. Every company that chooses trust over extraction makes it harder for the surveillance model to become the default.
I’ve been asked what my plan B is if the market moves toward normalizing surveillance-style medicine. If privacy concerns become quaint and the market rewards whoever has the most data.
Here’s my answer: I won’t suddenly pivot and start harvesting data just because everyone else is doing it.
Once you cross certain ethical lines, you can’t uncross them. Your company culture changes. Your decision-making framework changes. You become something different.
My plan B isn’t to compromise. It’s to serve the subset of the market that still values privacy and trust.
There will always be physicians who remember what it felt like to practice medicine before it became a data business. There will always be patients who want their most vulnerable moments protected.
That might be a smaller market. But it’s a market worth serving.
What Happens Next
The personalization paradox in healthcare AI comes down to a simple question.
Are we building technology that serves people, or are we building technology that extracts value from people?
The line between the two is often invisible until you cross it.
I’ve seen that line. I’ve almost crossed it. I’ve walked away from deals because staying would have meant building something I couldn’t be proud of.
The truth is, you can optimize for maximum data extraction and model performance, or you can optimize for trust and ethical boundaries.
You can’t do both.
We chose trust, even though it’s the harder and less profitable path in the short term.
Because once you lose trust in healthcare, you don’t get it back.
And because I’d rather fail doing the right thing than succeed doing the wrong thing.
The bells keep ringing. The question is whether we’re paying attention to the sound.
