Reliability does not equal predictive value

I guess because we're in the midst of Lyme season here in the Lyme rampant states today I googled "lyme disease" + test + research. The second "scholarly article" return was this one which I found interesting enough to read in full text (and to follow several linked references to and from).

Klempner, M. S., C. H. Schmid, L. Hu, A. C. Steere, G. Johnson, B. McCloud, R. Noring, and A. Weinstein. 2001. Intralaboratory reliability of serologic and urine testing for Lyme disease. Am. J. Med. 110:217-219.
Article Outline
Laboratory testing for Lyme disease is controversial because of problems with test sensitivity and specificity, the lack of standardized reagents, and interlaboratory and intralaboratory variability [see article for hyper-refs]. We determined the reliability of a serologic test and a urine test for Lyme disease, each performed in a reference laboratory, in control subjects and patients with Lyme disease who had posttreatment symptoms.

Material and Methods
Study Subjects
We studied 10 healthy control subjects who had never had Lyme disease and 21 patients with a history of acute Lyme disease, as defined by the Centers for Disease Control and Prevention [see article for hyper-refs], who had chronic (>6 month’s duration) fatigue, musculoskeletal pain, or neurocognitive impairment despite treatment with recommended antibiotics.

Sample Collection
Serum samples were obtained from all 21 patients and the 10 control subjects. One aliquot was immediately analyzed; duplicate aliquots were frozen at −70°C and tested within 6 months after collection.
...
Results
Serologic Test
In all 10 control subjects, the initial western blot analysis yielded negative results. In three of four duplicate specimens analyzed, the same immunoreactive bands seen in the original aliquot were present; 1 duplicate specimen contained a 41-kDa band that was not present in the original aliquot.

In the 21 patients with Lyme disease, the results of the initial western blot analysis were positive in 14 cases and negative in 7. Analysis of duplicate specimens yielded identical results in all 21 patients (κ = 1.0, Table 1). The same immunoreactive bands identified in the first analysis were present in 7 of the 14 seropositive duplicate samples; 5 samples had 1 additional band, and 2 samples had 2 additional bands. Repeat testing of the 7 seronegative samples showed fewer than 5 reactive bands in all samples.
...
To sum, using the "five band" criteria, the test-retest outcome was great when we consider just the 10 control subjects. Moreover, one might conclude that a negative blood test reliably predicts/detects the absence of disease in a person who does not have the disease. Further, a positive blood test reliably predicts/detects the presence of disease in people with it. However, a negative blood test is not so great a diagnostic tool for someone with the disease. That is, there are false negatives among the 21 experimental subjects, 7 false negatives in both the test and retest conditions (- reliably). That's one third of experimental condition subjects who tested negative twice despite being diagnosed with Lyme. How do our authors interpret these findings? In explicit terms, very narrowly.

Discussion
Our study showed that testing of duplicate serum specimens from 21 patients with Lyme disease and 10 healthy controls by a single reference laboratory using a commercially available immunoglobulin G western blot kit gave 100% concordant results for seroreactivity and highly reproducible results for the identification of individual bands.

Yes, the results were "reproducible", thus the test is deemed reliable. What does this mean for practice? That is, should we extrapolate that the western blot is a good test for confirming (or disconfirming) a suspected diagnosis of Lyme Disease? The authors make a cited claim that "In patients with chronic symptoms of at least 6 months’ duration, the most appropriate serologic test for prior infection with B. burgdorferi is the immunoglobulin G western blot, which is recommended by the Centers for Disease Control as the final basis for determination of seroreactivity." which, given its context in the publication, serves to link the reliability of the western blot with it being a good tool for diagnosis.

So I guess if you think you have Lyme, you'd better hope you're not a member of portion of the population who may get a false negative test result.

For more (and quite interesting) reading on predictive value of tests as diagnostic tools (which is different from the reliability of such a test), search terms "sensitivity" and "reliability". A nice overview of the topics as they relate to diagnostic testing is given in this piece by Tze-Wey Loong titled "Understanding sensitivity and specificity with the right side of the brain".

Lyme-tube 1

I've been hoping to make a Lyme public service announcement for the radio, oh for a while now. If you have chronic health problems, you know what it's like to find time for these "extra" things. Anyhow, I am determined to do it this year, mostly because I still run into people in my area who do not know even the basics about Lyme disease. Where I live is an extremely high Lyme disease area. I just went to a work party-thing where the host chided me and another guest for not wanting to sit outside. Three days later, we found out the host had been bitten by a tick. How's your poetic justice? The guest who had taken the warnings of Lyme seriously has since dubbed such events "tick-nics". An apt name for any outdoor event here in Lyme-ville CT.

I'm hoping my host doesn't catch Lyme from this bite - although the tick was on for a while and as I said, the area has a very high rate of Lyme disease infection.

Don't find out about Lyme after you get bit. Some people are diagnosed quickly and treated effectively, and they are fortunate enough to be cured. I know a few of them. But there are people whose symptoms fail to respond to treatment. I'm one of them.

Here are some folks who posted their Lyme stories on Youtube.
Laura



Jamie


Heather


Jady

fuel for the fire

Another Lyme headline. The part that spooks me is the third paragraph.

(excerpted from the Stamford Advocate)
Medical board approves probation, fine for controversial doctor
Associated Press
December 18 2007
HARTFORD, Conn. -- A New Haven pediatrician who has been praised by patients but criticized by the medical establishment for the way he treats Lyme disease was reprimanded, fined $10,000 and placed on two years probation by state regulators Tuesday.

The Connecticut Medical Examining Board voted unanimously to impose the sanctions after concluding that Dr. Charles Ray Jones violated care standards by diagnosing Lyme disease in a boy and his sister and prescribing antibiotics based on a phone conversation with their mother, months before he examined them in May 2004.

The board also found that Jones broke standards by failing to reconsider his diagnoses of the children after lab tests came up negative for the tick-borne disease, which can cause painful arthritis, meningitis and other serious illnesses if not treated promptly.

Board members further concluded that Jones was wrong to prescribe antibiotics for nearly a year without repeat exams and without any arrangement with another doctor, because the children lived in Nevada, to monitor for any side effects of long-term antibiotic therapy.

Hartford lawyer Elliott Pollack, who is representing the 77-year-old Jones, said he will appeal the board's decision.
...

Other things aside for a moment, I know it's hard but try, consider that part of the board's ruling was based on the finding that Dr. Jones "broke standards by failing to reconsider his diagnoses of the children after lab tests came up negative for the tick-borne disease". I thought that a Lyme diagnosis was supposed to be more based on clinical presentation than lab values. What crazy Lyme-loving website did I get that from? The CDC, among others.

Lyme disease is diagnosed based on symptoms, objective physical findings (such as erythema migrans, facial palsy, or arthritis), and a history of possible exposure to infected ticks. Validated laboratory tests can be very helpful but are not generally recommended when a patient has erythema migrans.

On the CDC Lyme website, this passage continues with a helpful link to the current, detailed recommendations on serological testing for Lyme. If you do click there, what you get is a report called "Recommendations for Test Performance and Interpretation from the Second National Conference on Serologic Diagnosis of Lyme Disease".

The Second National Conference on Serological Diagnosis of Lyme Disease was in October of 1994.

I'm going to let you all do the math. You'll do it faster than me. I nearly had to count it out on my fingers...good thing it wasn't much older or I'd have run out of fingers!

My point. There are two, or one somewhat bifurcated one depending on how you look at it. My point is that the CT board's finding engenders and perpetuates too high a level of confidence in serological testing for the clinician dealing with the realities of Lyme presentation. The board's level of confidence would be well founded if the testing had improved much in the over 13 years since the "current" test performance and interpretation recommendations were made. It has not. I wonder now as I so often do when I read the latest Lyme fueled controversy story (or maybe more appropriately, the latest controversy fueled Lyme story) - Can't we do better than this?

(A) Little Miss Informed

I'd say this is good evidence that the Lyme Disease awareness among the general population is not just limited but BAD. My guess is this kid got her information from internet searches and from friends. I don't think internet medical information (or searching) is inherently bad, but I do think not knowing how to gauge scientific and medical information presented on the internet plus the free form and flow of information on the internet makes for a pretty horrible combination (from a public health/disease awareness perspective at least).

The medium is new but the issue isn't. There's always been "folk" diagnostics, word of mouth, a friend of a friend's aunt, etc. which are often seen as at least as informative as scientifically sound medical information. And these sources are closer, easier to access, and have an affective link which characterizes few if any patient/provider contexts or relationships.

Where's my lollipop?

Or, "A long ranty rant about medical insincerity by someone who is likely a bit dopamine depleted"

I've been thinking about some of my least favorite contradictions in medicine and health perspectives. I suppose these could be called hypocrisies, but I feel this term implies belief which I am very reluctant to attribute. Contradiction, I think, as a word is more neutral on the existence and legitimacy of beliefs underlying the observables I am reacting to.

A recent round of Lyme related news items in the mainstream press as well as my own attempt not to smoke this week (been 7 days without) have brought these contradictions to light.

Let's take the smoking first since it's a shorter rant, er, argument.

I pay taxes on my cigarettes which are meant to discourage my smoking (supposedly). I'm not allowed to smoke in most places, or sometimes even near places. Again, that is meant to discourage my unhealthy habit. There are companies which are charging smokers extra fees on top of the employees' contributions to cover their premiums, the reasoning being something like this: In the case of smokers, companies and their advocates say it's a matter of promoting health and of fairness, given that smokers, as a group, generate higher costs for employers through direct health care expenses and lost productivity. 'It's an equity argument,' said Helen Darling, president of the National Business Group on Health. 'Part of my costs going up is because of the behavior of other people.'" (Spouses, Smokers Earning Penalties - Hartford Courant, 10/14/07)

I accept the reasoning behind these rationales. I don't know that I believe they are what is truly the driving force for some policies and behaviors, but I at least accept the stated reasoning as sound. What I wonder though is where's my state subsidized lollipops? Where's my insurance coverage for smoking cessation therapy, devices, assistance? The funding could come out of the massive sin taxes I've paid for the last 10+ years I've been a smoker. So the government, my employer, and my insurance companies want to encourage me to quit smoking so much so that they will charge me extra for continuing to smoke. But they are unwilling to put anything INTO helping me stop smoking. I kinda think if someone wants to regulate what legal, addictive substances people do and where and how they do them, those someones had better be willing to pony up when the miserable addicts try to comply.

And then there's Lyme. This one can be summed up as the logical proposition - It's not Lyme or you're crazy. This is the diagnostic choice you will face if you have Lyme Disease and you do not feel well after 3 weeks of oral antibiotics. The contradiction lies in the statements of compassionate skepticism espoused by various champions of this proposition.

Over at the Hartford Courant, my local but not very good news paper, some fellow has written these four articles in the last 2 weeks.
Debate Deepens Over Lyme Treatment
Study Casts Doubt on Lyme Disease Treatment
Lyme Disease Diagnosis Challenged
No Basis for Chronic Lyme Disease

Reading them, I find myself wondering if the writer was spawned in a lab by a mad scientist who had obtained genetic material from both Jerry Springer and Geraldo Rivera. This would explain so very many things, not the least of which is the author's apparent grudge against science and even a sheen of objectivity.

While I'd like to think the reporter is guilty not just of perpetuating but also of fabricating the disparity between the "sides" of the controversy, I know he is not. The phenomenon at the heart of the controversy comes down to a poorly defined yet nonetheless associated set of people (or as they are less humanly called, "presentations" or "cases").

- Some people diagnosed with Lyme Disease do not get better after the accepted, recommended minimal treatment, a 3 week course of oral doxycycline.
- Some of these people's diagnoses were based on symptoms, signs, and labwork which meet even the rigid surveillance criteria (as well as the less rigid clinical/diagnostic criteria).
- Some of these people's diagnoses were based on signs, symptoms, and labwork which meet clinical criteria accepted by most main stream doctors and medical providers.
- Some of these people's diagnoses were based on signs, symptoms, and labwork which meet the clinical criteria accepted by only some doctors (often called "LLMDs" but that category is another catch all).
- Some of these people were diagnosed and treated soon after the probable time of infection.
- Some waited years be diagnosed and treated.
- The types of symptoms all of these people continue to experience post treatment fall into roughly three categories (skeletomuscular, neuro/cognitive, constitutional).
- Some people report overlapping classes of continuing symptoms post treatment.
- Some people report only one type of these continuing symptoms post treatment.
- All of these people are grouped as a single population when placed in the context of the highly divisive and divided Lyme Controversy.
- By the Chronic Lyme people thinking, all of these people are actively infected with the bacteria which causes Lyme disease and will get better with more or the right antibiotics.
- By the anti-Chronic Lyme people thinking, all of these people do not have Lyme Disease. Further, some (perhaps many) never had it.

It doesn't take a genius to know considering only the parameters of this population that the set of (dubious) chronic Lyme patients is hardly a homogeneous group. Then one must consider the heterogeneity of treatment among any super or subset of that population (i.e., Who got IV antibiotics for 21 days? 28 days? Which antibiotics? Who got more than one course? Who got 2 months of oral antibiotics? 3? 4? Do I hear 5?) .

Clearly there is an impossibly confounded bundle of factors in treating this group and their complaints as any thing like an even functionally defined set. Moreover, you'd think it would be equally obvious that any conclusions made based on some just can't be legitimately extended to all. And yet they are. That's part one of the contradiction in the Lyme Controversy.

Part two is what happens when anyone who falls into that poorly defined group seeks medical attention for what ails them. The mainstream reasoning is that these people do not have Lyme, BUT, goes the mainstream reasoning, we are not dismissing their symptoms. We are just saying it's not an active chronic Lyme infection causing the symptoms. BUT we won't develop or advocate research protocols and plans to see what IS causing the symptoms. And many of us will toss you out on your ass if you have the temerity to continue to come into our offices complaining of feeling "tired all the time" or "achy all over" or "foggy" after we have issued our "it's not Lyme" proclamation.

I'll buy the argument of "it's not an active infection" for me and people who have very similar histories to my own. I will buy it for the sake of argument and, for myself given my family history of autoimmune disease, I'll buy it even for the sake of medical practice.

What I don't buy is that the mainstream, anti-chronic Lyme people actually believe that the set of people we are calling here the (dubious) chronic Lyme patients have anything physically wrong with them. If that were a sincere and genuine belief on the part of the practitioner, these patients wouldn't be turning to any number of alternative or non-mainstream and sometimes quite dangerous treatments. They'd be taken care of by their own mainstream medical providers. If it were a sincere and genuine belief on the pat of the researcher, I'd be seeing calls for research participants for autoimmune studies on people with a documentable history of past acute Lyme infections. I look. Believe me. I don't see those studies.

In short, if the "there is no such thing as chronic Lyme" declaration wasn't equivalent with "and you sick people aren't really sick", we'd see at least some research on what is causing some of these people to get sick and others of these people to stay sick.

Hey, if you see that, you let me know. Also, send lollipops.

bitter pill

When I was taking doxycycline for my acute Lyme infection many years ago, I developed a serious aversion to it. I had to force myself to swallow each twice daily dose. So great was the desire not to take it that I sometimes gagged on it and had to spit out into my hand, take a moment to visualize that it was a large tic tac, and then pop it in for another go. I often forgot to take it when I was supposed to. There were many, many nights when I had to get up out of bed, eat some crackers, and take the damned pill because I had totally "forgotten" to take it before I brushed my teeth and settled down for sleep. How did I remember after I got into bed? I had finally taped a sign to my bedroom ceiling to remind me to take the pills.

Why was it so bad? Because within two hours of taking it, I would feel HORRIBLE. It wasn't an upset stomach. It's hard to describe...I felt like my brain had been scrambled. My head felt like I had taken a blow from a 2x4 right dead center on the forehead. My eyes ached. I couldn't stand light and sitting up or standing made me feel like I was going to puke and pass out.

This happened every single time I took a pill. The time frame on it might be off. It's been a while since I was on the doxy and I tried so hard to forget the details. Some of them will stay with me for a while I'm sure but stuff like "did it start in two hours or was it peaked and falling off by two?" is information I can and therefore do forget.

When I was on the IV rocephin a year later, I had another sort of odd psychological reaction. This one wasn't based in any physical aversion training. It was the creepiness of knowing that little port not far above my elbow was the leading end of a line that went smack into the superior vena cava. The exact placement of the end of the line was quite apparent to me since for a few days after I had it placed, every time I crossed my right arm over my chest - say to brush the teeth in the left side of my mouth - my heart would go "ka-chu-u-unk" instead of the usual "ka-thunk". I had them pull it back some. (Yeah, like I said, I'm that patient.)

Point is, I knew it was there and I sometimes had to think really hard about not thinking about it so I wouldn't get skeeved out. About the second week in, I noticed that if I successfully accomplished this goal, I'd forget about the line maintenance routine. So it's flushed twice a day, once after the drug and once again about 12 hours later. On the days when I'd managed to not be thinking overly much of my PICC and where it went, I'd forget to bring my supplies with me if I was out. It was sort of a catch 22 - think about it and fret but remember to be prepared to take care of it or not think about it and be so blissfully unaware that I got caught more than once without my syringes.

These days, when I miss a pill or forget if I took one, I can't help wondering if there's some kind of aversion issue happening. The ones I forget now have none of the obvious displeasures associated with the doxy or the PICC line, but there are two in particular which I consistently forget to take or forget if I've taken. Unlike my elavil, which I seem to hardly ever forget to take and never forget if I have taken, I far too often find myself standing there looking at the package of PPIs or bottle of NSAIDs in my hand and wondering "crap...did I take one of these already?" or realizing many hours later (usually when the symptoms get raging) that I can't remember having taken one.

I've considered that there may be some regularity to this pattern. The one which has worked the best and without which have an eventual visible, marked effect (weight loss) is the one I am good about taking. It's not a simple cause and effect thing though. I don't feel immediately or even directly better after I take the elavil. It's a cumulative thing on a much longer time scale than I would have thought necessary to induce such an association. So what's going on with the others then that makes them so readily forgotten?

I was thinking about this today after I got off the phone with a friend who's been doing chemo every other week for about many months now. She's been switched to a once every three weeks course now, but says it still sucks a bunch. I also thought of another friend who had to administer Hepatitis C treatment to herself at home (this woman hates needles) and my brother who's taken various HIV and psych meds, all with various side effects and the occasional adverse event.

How the hell do they face each dose?

my new favorite t-shirt

I had been thinking of making one myself. Tonight I did a quick search and found the folks at the California Lyme Disease Association have gone and done it for me. I hope they don't mind my uploading a picture of their "Ticks Suck" t-shirt.



















How great is that? I'll tell ya how great. It's super great with great big sprinkles on top. I love creative education/awareness tools. I still have a PSA in the works for my local radio station (hoping to find someone to do a spanish version too), and now I'm planning on getting me one of these snappy t-shirts as well. And maybe some as Christmas presents for a few of the many folks I know who have or had Lyme.

Oh yeah, and one for my advisor, who has had to live with MY Lyme Disease and the effects of it on my progress as her PhD student. I went from having two very productive years which resulted in contributing to 3 publications, and now I'm barely inching my way towards a dissertation proposal draft.

Lyme time

Here's one of what I'm sure will be many posts on Lyme Disease.

I live in the state where infection with the bacteria Borrelia Burgdorferi (borreliosis) got it's more common name - Lyme Disease. According to the state's public health website, "the percentage of deer ticks carrying B. burgdorferi can reach 23% in some areas" of this state.

Therefore, Lyme is part of my environment, both biologically and socially. Since getting Lyme in 2002, I do tick checks whenever I've spent time outside or when I've spent time around pets who go outside. The tick checks are just part of how my life has changed since I contracted Lyme Disease, but they are the most obvious sign. Some people think I obsess. These are people who have never had Lyme Disease.

I know a lot of people who have or had Lyme Disease. The stats say I also know a lot of people who will have Lyme Disease. My experiences in this Lyme Disease saturated area indicate that public education and provider education need to be stepped up a bit. But there are barriers to any Lyme education and awareness campaign. The most notable is resistance fueled by already in place misinformation. Attitudes based on conclusions drawn not as much from experience or science but from poorly phrased facts, over-generalizations, over-reactions, under-reactions, and what amounts to professional gossip.

Lyme is a quite unfortunate illness, not just because it totally sucks physically. What makes it so unfortunate is the misinformation I spoke of. This misinformation is both a product of and a source for the "Lyme Controversy" (insert ominous music here if you have a sense of humor).

The controversy has arisen among and divided both patient advocacy groups and health care providers. Variety in approach to research can be good for scientific progress. E.g., good things can come out of a little good natured rivalry between research teams perhaps and a healthy peer review process which includes diverse views and perspectives. But the sides on the Lyme controversy are not good natured about each other's views. There's nothing healthy about the relationship here. It's downright contentious.

Once a thing becomes so controversial, views of the sides tend to get further and further away from each other. It's that whole "in group/out group" thing I think. One unfortunate outcome for patients with Lyme Disease (or sick people suspected of having Lyme) is that a common set of good, empirically based clinical practices has become increasingly hard to find. In its place, you too often find practices and presentations colored by prejudice, provider-patient relationships marred with contention due to sometime passionately unyielding preconceptions. Such encounters reinforce the divisions between the sides. They get passed on through word of mouth, perpetuating the cycle of distrust and adding ever increasing layers of generalized vilification of members of "the other side".

In short, once a controversy arises, people start acting silly. And Lyme is nothing to be silly about.

An example of what I mean when I say "silly" is when a doctor who sees you years after you had a diagnosis of Lyme Disease questions the validity of the diagnosis because he believes Lyme is an overdiagnosed illness. "I don't believe in chronic Lyme" a doctor once told me. "What?" I never said "chronic Lyme". I never even hinted at it. I just said I had had a Lyme diagnosis 6 months earlier and was still feeling crappy most of the time.

Silly is also when a friend or colleague sends you all sorts of alarming information saying oral antibiotics never work and insists that one can only be adequately treated by taking a year of IV antibiotics, undergoing hyperbaric treatments, and all but stalking some unfortunate local infectious disease doctor. "Otherwise you could DIE!!!! It eats your brain!!!! It's eating your brain RIGHT NOW!!!!" (Ok, I'm exaggerating on this last bit, but truly only very slightly)

A case of Lyme or suspected Lyme is even more unfortunate when it happens to someone who already has existing medical problems. Pre-existing medical problems add another layer of difficulty to a situation which is already potentially nasty enough.

Which brings me to the point of my post. A good friend who already has some pretty significant health problems has a bite, a rash, and is having symptoms of Lyme. In talking to her, I gave her some peer-based (i.e. non-medical) advice which I want to share. I don't presume to know the best plan of action for any particular individual who suspects that they may have Lyme Disease, other than to seek medical attention. How to proceed is up to you and your doctor(s). But here's my one small piece of peer-based, non-medical advice for anyone who thinks they might have Lyme.

If you have a rash, TAKE PICTURES OF IT. Get copies, and keep them with your medical records. Give one to your primary, keep one (at least) at home.

I found it to be useful. I had a low positive ELISA, a positive (for diagnostic standards) western blot (IgG and IgM bands), and a borderline spinal tap. And still, I have run into doctors who were skeptical that I ever had Lyme. Then I show my stunning Lyme rash picture and the skepticism melts away, replaced by statements like "Yep, that's Lyme".

It's been my experience as a patient that doubt of a prior diagnosis can get in the way of adequate assessment and treatment, not just for the Lyme Disease but for later medical conditions.

Here's a copy of my stunning lyme rash - more technically called an "erythema migrans" or "EM" rash.













There are (at least) two small very dark marks which I think are picture scan artifacts
(one mark is at just after 12 o'clock near the inner portion of the rim, the other is on my abdomen outside the rash border but at about 2ish relative to the rash)
I think the technical term is "scanner schumutz"