This Expert Panel is Archived.

This Expert Panel is no longer active as of December 2018. Thanks to those who posted here and made this information available to others visiting the site.

Quality Improvement Strategies for Ambulatory Care Settings

Posted: 27 Oct, 2014   Recommendations: 30   Replies: 48

While the majority of health care takes place in ambulatory care facilities, the majority of quality improvement efforts continue to be focused in acute care settings such as hospitals and emergency rooms. Common problems in outpatient care settings include delayed or missed diagnoses, medication errors, ineffective patient-physician communication, lack of follow up on lab tests and results, and delayed preventative interventions. The improvement of care coordination and preventative care in ambulatory settings has the potential to decrease usage of acute care resources and overall patient visits, improving patient outcomes, and decreasing health care costs nationwide.

In this virtual Expert Panel, panelists will share their experiences designing and implementing quality improvement programs in ambulatory care settings. To address these important issues, we are pleased to welcome our panelists for this discussion:

     • Tara Bishop, MD – Assistant Attending Physician at New York-Presbyterian Hospital and Weill Cornell Medical Center
     • Sandhya K. Rao, MD – Medical Internist at Massachusetts General Hospital and Associate Medical Director for the MGH Physicians Organization
     • Andrew Ellner, MD – Associate Physician in the Division of Global Health Equity at Brigham and Women’s Hospital, Primary Care Physician at the Phyllis Jen Center for Primary Care, and Assistant Professor of Medicine at Harvard Medical School, and Co-Director of the HMS Center for Primary Care
     • Amy L. Billett, MD - Director of Safety and Quality, Division of Pediatrics Hematology/Oncology and Associate Professor of Pediatrics at Harvard Medical School

Our panelists will offer insight into the following questions:

     1. What kinds of quality improvement efforts have you implemented or researched for ambulatory care settings? What’s unique about implementing QI programs in these settings?
     2. What are the key factors to successful quality improvement efforts in ambulatory care settings? How do you lay the groundwork for success when developing a QI program specific to ambulatory care settings?
     3. What kinds of metrics or indicators do you find most informative in these settings? How did you build consensus for these indicators?
     4. Who are the key stakeholders for developing QI programs in ambulatory care settings, and what incentives have you found useful for keeping these stakeholders engaged?
     5. With so many competing priorities, limited resources and time, how do you ensure QI efforts are sustainable?

This panel is part of our US Communities Initiative, which is supported by the Agency for Healthcare Research and Quality (AHRQ), and aims to foster discussions between health care professionals on evidence-based practices, and translating these practices across disparate settings, to improve health care delivery in under-served populations in the US.

In an effort to understand the impact of our Expert Panels, we’ve created a short (4 question) survey. Your responses are greatly appreciated—please take the survey before the discussion begins:

We look forward to a rich discussion next week – please join the conversation and share your questions or comments!



Lisa Buck Replied at 9:26 AM, 27 Oct 2014

I am particularly interested in using Motivational Interviewing skills with patients and any evidence that supports changes in patient behavior resulting from this intervention.

Winnie Nhlengethwa Replied at 7:47 AM, 29 Oct 2014

I will be happy to participate on November 3&4 .

Elizabeth Glaser Replied at 9:35 AM, 29 Oct 2014

Thanks , look forward to it.

Thomas Flood Replied at 9:54 AM, 29 Oct 2014

What have any of your facilities done to improve the air quality? Any UV systems in use to reduce the airborne bacteria?

Sandeep Saluja Replied at 10:01 AM, 29 Oct 2014

A couple of issues on my mind which I would like to learn please:

How to assess if clinicians are not missing diagnosing relatively rare diseases in the heavy rush of routine patients?
How to ensure and assess if they are able to treat difficult cases?
How to ensure and assess if evidence based medicine is being practised and unnecessary use of antibiotics etc. is minimised?

Adel Bakouch Replied at 11:49 AM, 29 Oct 2014

Thanks , look forward to it.

Catherine Anderton Replied at 12:43 PM, 29 Oct 2014

Role of peer review via manual chart audit in the EMR era?
Thank you.

Yudha Saputra Replied at 1:05 PM, 30 Oct 2014

I strongly believe ambulatory care not just part of hospital or clinic service but also in pharmacy or drug store. Monitoring patient is one of difficult thing to measure the quality of its based on my own experience. Double-job sometimes make us lose our concern on monitoring their goal therapy. In my opinion I think there should be one unit or division who focus only on monitoring quality of ambulatory care, without double-job.

Adel Salah Replied at 1:12 PM, 30 Oct 2014

Thanks , look forward to it.

Angela López Replied at 1:33 PM, 30 Oct 2014

I look forward the discussion. My concerns areas are:
1. How long (or short) should 1 patient stay with a doctor, knowing we have in developing countries, too many patients per day. Im in Mozambique.
2. What tools can i use to make sure I don't misdiagnose, or not diagnose other conditions present with the same patient?
3. Can I still do a `good job`, even though I don`t have time to ask about toilet training and sleeping patterns because of time?

Thank u,

Rebecca Jurbala Replied at 1:34 PM, 30 Oct 2014

Thank you all for sharing these great questions and topics for the Expert Panel.

I am very pleased to welcome a fourth panelist, Amy L. Billett, MD - Director of Safety and Quality, Division of Pediatrics Hematology/Oncology and Associate Professor of Pediatrics at Harvard Medical School. It's great to have you join us!

We are looking forward to a rich discussion next week.

Rebecca Jurbala Replied at 1:42 PM, 30 Oct 2014

Here is a relevant resource from AHRQ on external supports for QI programs in primary care practices.

Attached resource:

Kashif Khan Replied at 3:54 PM, 31 Oct 2014

My inquiry involves responses to the 5 questions posed above in terms of low and middle income countries.

We are working on a project to understand how to measure and improve the quality and effectiveness of primary health care delivery systems in low and middle income countries at the national or sub-national level (province, district or facility).

Does anyone have experiences to share or places we should be looking including success stories in:

1. Delivering high quality primary health care, or
2. Improvement approaches which have been proven to be effective.


Andrew Ellner, MD Replied at 7:33 PM, 2 Nov 2014

I am a primary care doctor at the Phyllis Jen Center for Primary Care in Boston, and the Co-Director of the Harvard Medical School Center for Primary Care. Over my career, I’ve focused on improving health systems at both the local level, in Boston, as well as the global level, through work with the Clinton HIV/AIDS Initiative, the World Health Organization and the Global Health Delivery Project.

I currently lead a collaborative that works across 20 primary care clinics within 7 health systems, serving about 275,000 patients. We are focused on optimizing the work of primary care teams and systems to provide care that is comprehensive, accessible, highly reliable and safe. In the first two years of work together, we had a deep focus on transforming towards working together in highly-effective, multi-disciplinary teams (involving doctors, nurses, social workers, medical assistants, administrative staff and others). This involved many fundamental changes in how our clinics were organized, managed and led. In our next phase of work we are building on this foundation of team-based care to focus on creating highly reliable systems for diagnosing important conditions, like colorectal and breast cancer, and caring for patients with complex care needs due to multiple medical and psychosocial co-morbidities.

Andrew Ellner, MD Replied at 7:35 PM, 2 Nov 2014

There are many important differences between quality improvement (QI) work in the ambulatory and inpatient setting, but I believe the most salient is that in the hospital, patients are basically captive and there is a high degree of control by the medical team over what happens over the relatively short time that a patient is admitted. In the ambulatory setting, care proceeds over longer time periods, and most of the important health-related actions (adhering to treatment and/or diagnostic plans, lifestyle choices like diet and exercise, etc.) are almost completely up to the patient and family and out of the control of the primary medical team. Some of the safety issues are similar (such as medication interactions and errors), but the most important safety issue – which, in the United States, is missed and delayed diagnoses of important diagnoses like cancer and heart disease – are different.

I think we are early in understanding how best to conduct QI and other systems improvement work in the ambulatory setting, but this fundamental difference – that patients and families are the most important “actors” or “workers” in this space – requires us to take a very different approach to how we conduct QI work. One way we’ve tried to address this in our collaborative is to mandate that the teams participating formally include patients and family members on their “transformation teams” – the multi-disciplinary team that meets regularly and focuses on transforming how the practice works and conducting QI activities. Doing so seems counter-intuitive and challenging at first (how can we show patients our “dirty laundry”?) but quickly helps to transform the teams’ focus towards making changes that are truly patient-centered. My intuition is that, in the future, the best way to do this will be to seek much more frequent and actionable feedback from patients and families in ways that are common in high-functioning businesses like Uber or Amazon.

Julia Hallisy Replied at 7:58 PM, 2 Nov 2014

Dear Dr. Ellner,

It is wonderful to hear you focus on patients and families as partners and to seek to include them as team members. My work at The Empowered Patient Coalition has focused on this very goal and our new companion website at offers dozens of free patient education and engagement tools including inpatient and outpatient SBAR forms for the public, rounding forms for families to use, daily progress sheets and many checklists and fact sheets written by and for patients. Including patients and families involves providing information, tools, resources and support for them to develop the skills and the confidence they need to contribute to healthcare safety and quality.

Julia Hallisy
The Empowered Patient Coalition

IDA CHAPUMA Replied at 2:53 AM, 3 Nov 2014

1. Improve/employee Human Resource Personnel who are qualified in
Ambulatory Case Settings;

2. Availability and sustainability of transport;

3. Availability and sustainability of drugs and medical resources;

4. Sustainability of finance to pay employees;

5. You have to have correct medical history;

6. Proper physical examination in order to come up with right

7. Right diagnosis with right treatment;

8. Right laboratory investigation to rule out disease progression;

9. Right prophylaxis for some medical conditions i.e. tuberculosis,
measles and whooping cough etc.

Tara Bishop Replied at 5:57 AM, 3 Nov 2014

Hello everyone. I am very happy to be on this panel and look forward to an interesting on-line discussion. It looks like there are already many interesting topics.

I am an Assistant Professor of Healthcare Policy and Research and Medicine at Weill Cornell Medical School and a practicing general internist. I also serve as the Director of Quality Improvement for the Department of Medicine at my medical college.

My research focuses on quality and safety in the outpatient setting. I like to think of quality broadly and often refer to the Institute of Medicine's (IOM) categories of quality to help me keep that broad framework. The IOM's categories are safety, effectiveness, efficiency, equity, patient-centeredness, and timeliness. This framework helps me remember that quality must be defined broadly.

The outpatient setting, in my opinion, can be a more challenging setting than the inpatient setting. Patients may be seen at different offices by different providers and care is delivered over long periods of time. In the outpatient setting we have to provide preventative care, acute care, and chronic disease management. We are also seeing more and more procedures being done in the outpatient setting.

Some of my research has identified problems in access to specialists, the relatively under-discussed problem of medical errors, and the challenges of chronic disease management. I'm looking forward to discussing these topics and others.

Sandhya Rao Replied at 8:58 AM, 3 Nov 2014

HI, my name is Sandhya Rao, and I am honored to be a part of this panel. I currently serve a number of different role at Partners Healthcare, in Boston, all aimed at engaging clinicians, in particular our specialists, in efforts to improve the quality, safety, and efficiency of care we provide. At Massachusetts General Hospital, one of the two academic health centers of Partners Healthcare, I am the associate medical director for quality improvement. I also serve as the the medical director for specialty care for Population Health Management at Partners Healthcare. In both of these roles, I oversee the establishment of PCP/Specialist collaborations, "e-consult" programs, virtual visit programs, and projects related to the "appropriateness" of specialty care. Finally, I founded and currently direct Partners' quality improvement training program for clinicians, which is called the Partners Clinical Process Improvement Leadership Program. (CPIP)
I agree with what has been said about the key differences between hospital based and ambulatory quailty improvement. Here are some factors that I have seen lead to great success in ambulatory care QI.
1) Establishment of a broad team that includes frontline providers: As has been mentioned, the tricky part of ambulatory care redesign is the number of transitions and handoffs that patients experience as they move through the system. It is very critical that QI leaders establish broad problem specific teams that include a frontline expert on every piece of the process that relates to the question at hand. Often we hesitate to do this because of logistic complexity,the urgency (and impatience) of leadership to find a solution, and the worry that we will never achieve consensus if we involve everyone. Over and over again, I have seen teams bypass this step and pay the price at the end.
2) Make a diagnosis first!: As clinicians, we were trained to consider a broad differential and gather data before jumping to conclusions. The same rules apply in QI. With limited resources, it is critical that we take the time to use QI tools such as process mapping, observations, and MEASUREMENT, to find objective data that identifies the key drivers of our quality gaps. This can be done through observing a process for a few days or following a couple of patients around, but without it, we risk wasting resources on solutions developed based on anecdotes.
3) Think big, test small: don't be afraid to try something! With the complexity of processes involved with ambulatory care, it is very useful to break your big problem down into small chunks, and test your ideas locally, perhaps at one site that is well suited for change. Because ambulatory care redesign often involves broad roll out to multiple sites, it is useful to just start with a small "lab.


A/Prof. Terry HANNAN Replied at 5:28 PM, 3 Nov 2014

In response to Q2 and possible Q3. “What are the key factors to successful quality improvement efforts in ambulatory care settings? How do you lay the groundwork for success when developing a QI program specific to ambulatory care settings?” I believe that we need to build any QI improvements on the core principle of health care delivery as defined within the WHO charter. This is, “there is no health without management, and there is no management without information”.[ Reference: Leao B. Terms of reference for designing the requirements of the health information system of the Maputo Central Hospital and preparation of the tender specifications. Geneva: World Health Organization; 2007.]

So what are the ”tools” that help us deliver this health information management? In the current era we would like them all to be “e-“ however this is currently not possible but is achievable in the future.
There are significant technical aspects of these “e-tools” that are essential but the most critical component of any successful quality health information management toll is end user involvement and this MUST include patients. As Mamlin and Biondich documented if an e-health system is not clinically useful it will not be used. (1, 2)

Based on this knowledge it is of interest to note that despite the proliferation of mHealth technologies in low and middle income nations (and even high income nations) there are few studies demonstrating significant benefits to care delivery. So the question needs to be asked do we have the ‘focus’ wrong?
From Professor Alain Labrique

1. Mamlin BW, Biondich PG. AMPATH Medical Record System (AMRS): collaborating toward an EMR for developing countries. AMIA Annu Symp Proc. 2005:490-4. Epub 2006/06/17.
2. Mamlin BW, Biondich PG, Wolfe BA, Fraser H, Jazayeri D, Allen C, et al. Cooking up an open source EMR for developing countries: OpenMRS - a recipe for successful collaboration. AMIA Annu Symp Proc. 2006:529-33. Epub 2007/01/24.

Tara Bishop Replied at 6:20 PM, 3 Nov 2014

What great questions, insights, and thoughts. With regard to the first set of questions - here are some thoughts:
Our panelists will offer insight into the following questions:

1. What kinds of quality improvement efforts have you implemented or researched for ambulatory care settings? What’s unique about implementing QI programs in these settings?

- I have implemented small QI projects in an academic practice and have studied several large interventions by the New York City Department of Health in small practice around New York City. There are so many unique aspects of these two kinds of practices. First, these practices tend to serve underserved patients who may have no insurance or Medicaid and who may have language barriers. These are particularly vulnerable groups of patients and have particular challenges when implementing QI projects.

2. What are the key factors to successful quality improvement efforts in ambulatory care settings? How do you lay the groundwork for success when developing a QI program specific to ambulatory care settings?

-I think key factors for successful ambulatory QI are similar to inpatient QI. We have to engage providers and patients. We have to change incentives (if possible). We have to build in resources and standard processes. And we have to change culture. I also think evaluation should be built into any QI project - we have to be able to measure the impact of QI interventions.

3. What kinds of metrics or indicators do you find most informative in these settings? How did you build consensus for these indicators?

-Simple metrics that can come from electronic records are easy and don't require more data collection but often other metrics are necessary to show impact in areas that matter but are hard to measure. Using standard measures (like those endorsed by the National Quality Forum) or validated survey tools might help build consensus.

4. Who are the key stakeholders for developing QI programs in ambulatory care settings, and what incentives have you found useful for keeping these stakeholders engaged?

-We are all harried in the ambulatory setting. No one seems to have enough time so QI projects that can ease the workload of doctors and other providers seem to work well. For example, we have a dedicated Coumadin clinic that is run by a nurse practioner who uses a standard algorithm for adjusting Coumadin doses. It not only creates a safer care but eases the workload of the doctors in the practice. It's a win-win.

5. With so many competing priorities, limited resources and time, how do you ensure QI efforts are sustainable?

-Leadership matters and often leadership buy-in is tied to incentives like payment or reporting requirements. If QI efforts can leverage the incentives by payers and regulators then I think things will be sustainable. QI that leverages IT can also help make things sustainable - if something is built into the EMR then even if there is an upfront cost, upkeep costs are low.

Lindsay Jubelt Replied at 6:21 PM, 3 Nov 2014

Tara makes a great point about the growing recognition of missed diagnoses and medical errors as important areas of quality and safety in the ambulatory setting.

Are others working on this issue? If so, how have you started to measure or address it?

Tara -- what has your work revealed to date?

Lindsay Jubelt Replied at 6:33 PM, 3 Nov 2014

In Sandhya's earlier post (#19), she mentions the importance of getting front line staff involved in QI projects. This is a challenge given the volume and efficiency pressures in the ambulatory setting.

Some strategies to promote staff involvement in QI include allocating time for QI work or paying incentives for quality improvement gains. Not all delivery systems are supportive of these strategies. I'm curious what experiences others have had in soliciting front line involvement in ambulatory quality improvement. What has worked? What has not?

Andrew Ellner, MD Replied at 7:52 PM, 3 Nov 2014

I want to thank Julia for the great tip about tools for engaging and empowering patients. Again, I think that this is the most critical and underexplored aspect of ensuring quality and safety in the ambulatory setting.

In response to the comments about missed diagnoses, this is an area where I am focused. Studies point to missed cancer diagnoses as among the most commonly missed diagnoses (or at least the most consequential, as they are the greatest source of malpractice claims). To address this issue, we are beginning to work in my collaborative on preventing and ideally eliminating missed and delayed diagnosis of colorectal cancer (CRC). Our practices are mapping and working to improve their processes not only for CRC screening, but also for responding to suggestive symptoms and signs such as rectal bleeding or iron deficiency anemia. Some emerging early themes are the importance of closed looped systems for tracking referrals to specialists (like GI) or tests (like colonoscopies); the need to dramatically reduce barriers that patients experience in accessing and navigating the system (eg low literacy; transportation; ability to adhere to the bowel prep regimen); and the potential for non-physician members of primary care teams to help with tracking and following up on abnormal tests and helping patients navigate the system.

There are some systems beginning to experiment with big data approaches, such as electronically scanning patient populations for worrisome signs and symptoms (rectal bleeding; breast lumps; iron deficiency anemia) that have not been followed up, but, although this makes sense as a helpful approach, not yet good evidence that this will work.

Sandeep Saluja Replied at 8:06 PM, 3 Nov 2014

Does it help if an experienced clinician is assigned the task of randomly
visiting clinics to assess and help with difficult cases?

Andrew Ellner, MD Replied at 8:07 PM, 3 Nov 2014

In response to today's discussion question: 2. What are the key factors to successful quality improvement efforts in ambulatory care settings? How do you lay the groundwork for success when developing a QI program specific to ambulatory care settings?

One of the biggest challenges that I have experienced in ambulatory care is getting adequate data in real-time to know whether a change we are testing is an improvement. This is because most EHRs in use were not designed with this type of functionality in mind. In the short-term, to get around this challenge, we have to not be afraid to start with small tests of change and very simple metrics, often looking at processes of care rather than outcomes. In the longer term, I do think we need to work towards using structured fields and other approaches to EHR design and use that allow for much easier data capture and analysis and real-time metrics.

Aside from this data issue, another key is meaningfully engaging frontline staff and all other team members in identifying opportunities for improvement and trying tests of change. This requires some time to meet initially and on a regular basis, engaging the full team early on in planning the effort, and really committing to giving everyone a voice during project planning and implementation.

And the final key piece, which I will say once more, is finding ways to involve patients in the planning process and in giving regular feedback about whether a project is working.

Sandhya Rao Replied at 11:41 PM, 4 Nov 2014

HI all,

In response to some of the interesting questions raised today:
Sandeep:Does it help if an experienced clinician is assigned the task of randomly
visiting clinics to assess and help with difficult cases?

At our institution, there was a pilot study in which a dermatologist was available on demand to visit primary care clinics in which there was a question of cellulitis, to help make the diagnosis and suggest treatment. The short study showed that this physician was able to prevent several hospital admissions and emergency room visits by avoiding erroneous diagnoses of cellulitis. In addition, at our primary care sites that are lucky enough to have co-located specialty practices, we observe that there is more informal consultation, fewer formal specialty referrals, that often involved long waits, and high levels of satisfaction from both primary care physicians and specialists. Rather than random, from a systems perspective, I would suggest doing a small test of change, and if positive, implementing the concept of access to experienced or specialized clinicians in a systematic way, rather than randomly. At our institution, we are working to use simple technology like email as well as video based technology to facilate "e-consultation" between clinicians when appropriate.

Lindsay's question:I'm curious what experiences others have had in soliciting front line involvement in ambulatory quality improvement. What has worked? What has not?

This is a tricky one. If we take the long view that ultimately we aim to create a culture of continuous quality improvement, in which all members of the care team view improvement as part of the role on their team, then short term financial incentives can be problematic. Although I have observed several examples of projects in which small financial incentives have been useful in grabbing the attention of care team members, the most successful long term efforts I have seen rely on establishing team culture and setting attainable, measurable goals, and posting them in shared workspace to create momentum among teams. On a light note, I have seen competition and gaming as a very effective method for engaging front line clinicians in the adoption of process improvements.


Tara Bishop Replied at 7:13 AM, 5 Nov 2014

On Monday Lindsay brought up the issue of diagnostic errors in the outpatient setting. This is, I think, an important issue but difficult to measure. We know that most malpractice claims in the u.s. in the ambulatory setting are for a diagnostic errors. These errors are hard to measure because they can sometimes be caused by cognitive errors. One reasonable area to consider studying and addressing are things that fall through the cracks. For example, abnormal lab results that are not addressed or patients not getting to a recommended specialist may be feasible problems to measure and fix.

Andrew Ellner, MD Replied at 8:31 AM, 5 Nov 2014

To respond further to the thread about diagnostic errors, while the literature does suggest that many of these errors are a combination of systems and cognitive failures, I think it's important to put the cognitive failures in context. My sense from reading more about actual malpractice claims cases is that, even where there may seem to be an error in judgment on the clinicians' part, it almost always happens within the context of great complexity and chaos that is a function of limited organization and management in ambulatory practices, unrealistic expectations of provider volume (too many patients in a day; not enough time) and/or medical and psychosocial complexity for patients (many co-morbidities competing for priority and time). I agree with Tara's comment that a necessary (although perhaps insufficient) step in creating high reliability in the ambulatory setting is closing all the loops related to referrals, tests ordered, and test results. On top of that, I think there will be population management approaches that help to do ongoing surveillance and create a safety net by identifying and proactively helping clinicians address potential misses and delays in diagnosis.

Derek Ritz Replied at 10:38 AM, 5 Nov 2014

The issue of diagnostic errors is a difficult one for many reasons. This issue is made more difficult by the fact that cognitive errors on the part of the physician can and do lead to diagnostic errors -- but it is (presently) very difficult to address such cognitive errors as a root cause because of the (present) care culture and the (present) compensation structure. Although the care culture is evolving, the physician is still very often a single point of failure regarding care management, including diagnosis. Although the compensation schemes are evolving, fee-for-service payments still incent physicians to practice within the chaotic, time-constrained environments that come with a focus on high patient traffic. In this light, it starts to look more like a system design issue than a cognitive errors issue.

My sense is that the system design has to be revisited to address the root causes of our present safety/quality issues in healthcare delivery. Such system design changes will present challenges; we will need to evolve the power structures and the compensation schemes. To not take on these challenges, however, will require our physicians to exhibit super-human capabilities in order for safety/quality to be improved from its present level to the kind of metrics found in high-reliability industries that enjoy strong safety records and low error rates. I don't think that's a reasonable expectation and, perhaps more importantly, it is not fair to physicians.

Patients are going to continue to have co-morbidities. There are going to be competing priorities and there will be complexities that need to be addressed, and will require sufficient time and attention span to be addressed successfully. Humans are fallible. In the present system design, we can bring a malpractice suit against someone because they're just being human. That's not a design we should continue to favour. We need to be willing to re-design for system-level quality and system-level safety.

Bistra Zheleva Replied at 11:06 AM, 5 Nov 2014

What a fascinating discussion! I am afraid I can't contribute much to it because most of the work I am involved in happens in the secondary and tertiary levels of health care but I am intrigued by the last comment as it raised a question for me - Derek, what changes would you ideally see happen in the present day care culture and compensation structure to allow for better quality of diagnosis to happen?

I myself had the experience once where I was misdiagnosed and I self-diagnosed and had to strongly press on my provider (with whom I had a great relationship) to provide further testing. That experience taught me about the power of the informed patient but I am curious about what other things would help to improve this process.

Tara Bishop Replied at 12:14 PM, 5 Nov 2014

Andrew - great comments on diagnostic errors. I couldn't agree more that harried doctors with less time and less support are likely prone to more diagnostic errors. There is a really great article in this month's Harvard Business Review that discusses the importance of staff and support systems in healthcare and argues that they shouldn't be cut for cost savings:

Measuring diagnostic errors in the ambulatory setting is hard - how have others done this?

William Martinez Replied at 5:17 PM, 5 Nov 2014

I also could not agree more that we need to be creating work enviornments that bring out the best in people, not the worst. Structural flaws like those discussed here set up clinicians for errors. This is very clear and should be a central focus of improvement efforts. It is also important to not to ignore individual contributions to errors and to strive for personal improvement in diagnostic acumen. Many clinicians work in suboptimal, harried conditions, yet some of those clinicians may make a disproportionate amount of errors (or have other performance issues) relative to similarly matched colleagues working in similar environments (see attached WebM&M article). It is important that we have a system for identifying and assisting these colleagues. Studying high performing physicians may also be of value as there is lots that can be learned from their approach (see attached blog by Bob Wachter) and their work environment that allows then to function at a high level. Creating better systems and work environments to prevent errors is critical and part of that system is monitoring workforce performance and creating a culture where feedback on performance is expected and welcomed as part of a commitment to continued improvement.

Attached resources:

Andrew Ellner, MD Replied at 9:29 AM, 6 Nov 2014

Fantastic discussion. A couple of thoughts.

In terms of a systems design to address diagnostic error, I strongly agree that we need to leave the paradigm of the superhuman doctor who knows everything, does everything, and has all of the power behind. This will require thoughtfully redistributing responsibility and power not only to other care givers on the team, but also to patients and families. See Gordy Schiff's wonderful 2013 article for a great vision around this. Where there are clear algorithms to guide appropriate care (screening, immunizations, etc), we should be designing a system that ensures adherence through a combination of non-physician work, patient empowerment, and supportive IT systems. See also, Richard Bohmer's work for important ideas about standardization vs. customization in healthcare operations. Physicians will always have an important role, but it should be around situations involving complexity, variation, and building a relationship with patients and families that has both therapeutic and diagnostic power.

There are many important corollaries to an approach in this paradigm. One is that we need to dramatically lower barriers to access, which feels scary and counter-intuitive in a strictly visit-based model of delivery where people need appointments, need to travel to clinics, etc. But breaking out of this approach through volume based payments, re-designed workflows that protect clinician time for non-visit-based patient care and interaction, and much more sophisticated, functional IT systems, should enable us to dramatically improve our diagnostic accuracy and timeliness. My prediction is that such an approach could also greatly enhance patient experience, productive and timely interaction between generalists and specialists, and provider work-life.

Dan Jackson TWIZELIMANA Replied at 10:33 AM, 6 Nov 2014

It is good to do evidence based medicine every where we are including in
ambulatory care settings so that we offer best care to the patints .UTD is
one of the best tool to use for ambulatory setting because it is easy and
effective tool to use.

Sandhya Rao Replied at 12:11 PM, 6 Nov 2014

Thanks for such a great discussion.

As a transition into question 3: What kinds of metrics or indicators do you find most informative in these settings? How did you build consensus for these indicators?, that relates to the practice of evidence-based medicine and standardization of care, where appropriate, I wanted to mention the use of "variation data". Measuring variation in clinical decision making has been a way for us to engage clinicians in a conversation about standardization of care without applying judgement about decision failures or overuse/underuse of testing, etc. It is not easy, but we have identified cohorts of doctors, such as primary care physicians, and a common clinical scenario, like ordering a head CT for a headache, and showed physicians how they relate to their peers in terms of ordering. We share the data with physician groups, bringing a specialist along to discuss tricky cases, and even demonstrate physical exam maneuvers they use in these common scenarios. I agree with Andy that these types of efforts still overemphasize the role of the doctor in adhering to guidelines. In the long term, systems should really help eliminate unwarranted variation where possible. For that, consensus on practice standards, even on local processes that would enable care team members to order overdue screening tests, is critical. We have found that these variation reports have been a good way to start that conversation.

Tara Bishop Replied at 12:21 PM, 6 Nov 2014

William Martinez - great links to articles on diagnostic error

Tara Bishop Replied at 2:46 PM, 6 Nov 2014

For today's question what kinds of metrics or indicators do you find most informative in these settings? How did you build consensus for these indicators?

A few things to think about:
1. Does the measure actually matter to patients and providers - does it really represent high quality care. Can you reliably get the data for the measure?
2. Can the measure be captured reliably and relatively easily?
3. What is the risk of unintended consequences by choosing a measure?

There is a great article worth reading - What make a good quality measure:

I think it is easy to start using standard measures like HEDIS, PQRS or NQF certified measures. These are generally vetted measures that have gone through a validation process. Of course, there are domains of quality that are not well-captured in these sources but I think they offer a good starting point.

Clemens Hong Replied at 2:26 AM, 7 Nov 2014

What a terrific, wide-ranging discussion. I see threads above that raise the issues of culture and incentives. I would love hear panelists (and community members) focused comment in these two areas:

1) How have you gone about building a culture of continuous, data-driven quality improvement in your settings? I think the issue of culture is a critical one in QI, and it would be great to have some practical tips (e.g. best practices) on how you all build culture.

2) What is the appropriate role for incentives in QI, and how do you structure incentives? This is a very complex pair of question, so I'll try to give a few more specific examples here. If you a multidisciplinary team is accountable for a measure, does the team get the incentive or the provider? What if you have specialists and primary care providers working together - e.g. shared measures? How do you mitigate against the potential unintended consequences (e.g. that incentive dollars are more likely to land in the "pockets" of providers working with "easier to manage" populations, rather than providers that need the most help? Do you do use case-mix adjustment? Do you provide incentives for reaching measure thresholds, for improvement, for hybrids of the two?

My bias is that performance incentives only really have any hope of working for the simplest process measures (e.g. documentation of smoking status in the EMR, but not for attainment of a HbA1C<8 in a patient with diabetes). Maybe one useful incentive is an incentive to hold a regular multidisciplinary QI meeting, but there a host of other measures for which I would be very skeptical. Anyway, I don't know if there is an answer, but I would love to hear your thoughts on this complex issue.

Thanks again for a rich discussion.

Andrew Ellner, MD Replied at 11:12 AM, 7 Nov 2014

Thanks, Clemens. These are great questions. My $0.02:

Culture: I couldn't agree more that this is critical. In my reading and thinking about this, the best material about creating culture comes from the management literature about teams/teaming and organizational behavior. See in particular Richard Hackman's books and framework about teams and Patrick Lencioni's books The Five Dysfunctions of a Team and The Four Obsessions of an Extraordinary CEO. Long story short, it's the role of the leader in a team or organization to set and reinforce cultural norms. In my experience, it is virtually impossible to create the type of culture that you describe above without having a leader who is both engaged and effective at empowering all team members to contribute to their full capacity. Given the existing very hierarchical power dynamics in healthcare, this really requires over-communicating the importance of everyone speaking up and then clearly demonstrating that peoples' opinions, ideas, feedback etc. are being taken seriously and lead to changes in process and structure. It may also require a leader confronting individuals who are disruptive to this type of culture.

Andrew Ellner, MD Replied at 11:23 AM, 7 Nov 2014

On incentives: this is an important and really challenging question. I don't know good literature that addresses this beyond some of the findings about the limits and unintended consequences of pay-for-performance. I agree with your skeptical stance, and would really underscore the importance of intrinsic motivation among healthcare workers to care for people effectively (which has much overlap with professionalism) and the dangers of actually dampening this with poorly designed incentives. (See the book The Penguin and the Leviathan for a good basic description of this phenomenon.) My intuition, and based on thoughtful comments from highly effective healthcare leaders and managers like Doug Eby, CMO of Southcentral Foundation, is that we should limit financial incentives (and instead aim to people a fair salary for their level of qualification) and do our best to regularly measure and publicly report the best process and outcomes metrics we can identify. This will provide most of the motivation people need, particularly if we address the organizational dysfunction that makes healthcare and particularly primary care so difficult to practice right now. I acknowledge that we still don't have great metrics, but agree with Tara's comments above that we need to move towards much more patient reporting of experience and outcomes of care.

Andrew Kahr Replied at 11:43 AM, 7 Nov 2014

I believe the concern here is first of all with decisions made by individual deliverers of health care--decisions as to what questions to ask, what tests to apply, what diagnosis to reach, what followup to propose--and when to end the visit.

There is a copious literature (Kahneman, Ariely, etc.) showing that in a vast range of recurring decision settings, those making decisions have a tremendously excessive confidence that they are making good decisions. But they aren't, because they haven't considered the alternatives, gathered the information, and evaluated possible consequences (including unintended ones).

This is one major source of medical errors.

These errors can be reduced by providing an interactive expert system, delivered through the cell phone, which directs the visit step by step. When there's an exception, an expert can be put on the line.

The place to start this is in settings with the least qualified (non-physician, non-nurse) deliverers of care--for instance, "level 3" in India, people working alone in villages. Data shows that these people deliver terrible care first of all because they don't spend enough time with the patient, don't ask enough questions, and don't do enough tests. They jump to conclusions and make errors.

How about testing this in N. India?

Tara Bishop Replied at 12:28 PM, 7 Nov 2014

Clemens - great answer to the two questions of the day regarding culture and incentives.

Individual behavior is influenced by a myriad of factors including culture and incentives but also time, experience, and resources. Dan Pink's popular book, Drive, highlights that motivation is not always about carrots and sticks. Dean Harlin at Yale has also written about this. I think QI that makes high quality care easy (like some of the behavioral economics work on defaults and choice architecture) and standard processes that take simple decision-making out of human hands (like algorithms for coumadin management) hold a lot of promise.

Culture is absolutely important but slow to change. And incentives can definitely help but right now are not designed in such that is very effective.

Lindsay Jubelt Replied at 1:48 PM, 7 Nov 2014

I'm so glad that Tara brought up Daniel Pink's Drive. One important point made in the book is that incentives seem to have more impact when they come unexpectedly. When they become routinized and an expected component of one's salary, the effect may be diminished.

Perhaps it would be helpful to expand our thinking. Incentives can be positive or negative and they can be monetary or non-monetary. For example, incentives can come in the form of academic prestige, professional development, or personal fulfillment. To dates, there has been limited research on the impact of various incentives and incentive structures on physicians, particularly in the non-monetary category. One paper from this past year of particular note on this subject is Jonathan Kolstad's work examining the impact of public report cards vs. financial incentives on cardiac surgeons in Pennsylvania. Koldstad found that the intrinsic incentive to do well compared to one's peers was four times more powerful on surgeon's performance on quality scores than financial incentive.

I'd be curious to hear what other's experience with incentives have been, and what innovative ideas people have for incentives in QI.

Attached resource:

christophe millien Replied at 4:54 AM, 8 Nov 2014

Several must be adressed to improve quality for ambulatory care.
1- engagement of the patient is very important in his own care. Education
of him about seval things like screening for disease, prevention, his
responsibility in promotion of health to people around him, let the
patient know what that means the best quality in fonction of thr realy.
2- having enough infrastructures, materials and human ressources to take
care of them. Beside medical staff like primary physician, RN, midewife
etc. , community health workers, social worker, should be an important
part of this strategia.
3- guidelines and reseaches are very important. The epidemiologic aspect is
necessairy for planning. A health map of the population by considering the
epidemiologic aspect can give you an idea in what part of the population
you have to do a specific action.
4- proccess must be well established.
5- technology is important element.we can use it for: Increase the
diagnosis capacity, improve the management of the patient, doing community
education, favor accesibility to the care etc..

stephen hendricks Replied at 11:32 AM, 8 Nov 2014

My name is Stephen Hendricks from South Africa, a Public Health Specialist and a Harvard University Graduate where I completed the graduate degrees , that is , a MPH degree . at the School of Public Health, the MPA degree at the Kennedy School of Government and the Doctorate in Medical Sciences degree with majors in Health Policy, Epidemiology and Biostatistics jointly between the School of Public Health and the Medical School. I am currently back in South Africa where I hold a full time Professor appointment In Health Policy and Management, in the School of Health Sciences and Public Health at University of Pretoria. I joined the GHD online for the richness of views which are expressed by colleagues from all over the US and elsewhere. The 5 Qs that Tara Bishop put on the discussion platform have particular significance for South Africa as we are at the point of sorting the QI platform across the SA health care system (having just completed a QI audit for 4000 clinics) to become the platform on which the Minister of Health, Dr Aaron Motsoaledi , wants to launch UHC or NHI in SA, though colleagues on this GHD online team would probably immediately caution us on ensuring that all our "ducks" are in a row and not land up where Obama Care seemingly finds itself today moving at a very slow pace. The issues of culture and incentives are of significance because in SA apart from the culture of medical work and the asymmetry of information that physicians and health care providers have relative to patient seeking care, there is also the traditional culture and practices which impact often negatively of QI implementation because of the indigenous nature of the health belief model view held by the majority of South Africans irrespective of how sophisticated they may be because of western education exposure. So I would really like to ask the colleagues on the GHD QI to assist me in this challenge. Secondly, I am facilitating a QI Course 17-21 November 2014 at University of Pretoria School of Public Health with graduate health students most of who are physicians , medical specialists and other health care providers, CEOs of Hospitals and District Health Managers. if any members of the GHD QI online panel would like to participate eg you may have a QI video clip for about 2 to 3 minutes in which you are teaching an aspect of QI, then please let me know and I can use it during the class teaching.

Rebecca Jurbala Replied at 10:02 AM, 10 Nov 2014

As our Expert Panel has now ended, I’d like to thank our wonderful expert panelists for sharing their insights with us and all of our community members for contributing to this incredibly rich discussion. We look forward to continuing our focus on these important issues in the community.

We will be working on a Discussion Brief to summarize the key points from this Expert Panel, and will share details as soon as that is available on the website.

In the meantime, we would be grateful for your feedback in our short, (9 question) follow-up survey. These surveys help us understand the impact of our Expert Panel discussions, and your feedback is incredibly valuable to us. Please take the survey now, by visiting:

Sudip Bhandari Replied at 9:12 AM, 26 Jan 2015

Many thanks again to our panelists and members who participated in this Expert Panel discussion in November 2014!

To help us understand the longer-term impact of these Expert Panels and plan future events, we have created a very short, 5 question, follow-up survey. This survey will only take 2-3 minutes of your time—please take the survey now at:

Many thanks,

This Expert Panel is Archived.

This Expert Panel is no longer active as of December 2018. Thanks to those who posted here and made this information available to others visiting the site.