Yeah, well in my field, oncology, meta-analyses are somewhat irrelevant. As you can imagine, the bar for completing a phase 3 randomised trial in oncology is pretty high. Meta-analyses are mostly there for trainees to notch up a paper.
Another fine example of the author's point is the ivermectin in covid meta-analytic nonsense (which I cannot even bring myself to link), where a bunch of small rubbish trials are meta-analysed into a 'flawless' body of evidence while double blind randomised trials are impugned.
Disagree. I change practice on Monday after a single quality trial. Pick up any society guideline, only a small amount of the recommendations rely on meta-analyses. Look at immunotherapy or antibody drug conjugates, revolutionary therapies that arrived one trial at a time.
For example, the majority diagnostic testing guidelines are based on meta-analyses.
FYI since you asked to pick one, let's take a look at NCCN. Not sure where you're drawing your conclusion that most evidence is from phase III RCTs:
"[Q] Quality and quantity of evidence refers to the number and types of clinical trials relevant to a particular intervention. To determine a score, panel members may weigh the depth of the evidence, i.e., the numbers of trials that address this issue and their design. The scale used to measure quality of evidence is:
2 (Low quality): Case reports or extensive clinical experience
1 (Poor quality): Little or no evidence"
"The overall quality of the clinical data and evidence that exist within the field of cancer research is highly variable, both within and across cancer types. Large, well designed, randomized controlled trials (RCTs) may provide high-quality clinical evidence in some tumor types and clinical situations. However, much of the clinical evidence available to clinicians is primarily based on data from indirect comparisons among randomized trials, phase II or non-randomized trials, or in many cases, on limited data from multiple smaller trials, retrospective studies, or clinical observations. In some clinical situations, no meaningful clinical data exist and patient care must be based upon clinical experience alone. Thus, in the field of oncology, it becomes critical and necessary (where the evidence landscape remains sparse or suboptimal) to include input from the experience and expertise of cancer specialists and other clinical experts."
From an article:
"We identified 1124 potential systematic reviews from our survey of the 49 NCCN guidelines for the treatment of cancer by site. Five NCCN guidelines did not cite any systematic reviews."
Well you discount the most important thing in the first line. 'Cutting-edge' is a funny way of saying 'most effective', as if it were somehow irrelevant.
I didn't say most evidence is from phase III RCTs, particularly if you include everything that happens in oncology as the denominator, only that meta-analyses were not that relevant. Most of the critical patient facing interventions have the backing of good quality trials, at least where it is reasonable and possible to do a trial. Also one of your citations is seemingly casting doubt on the value of meta-analyses in oncology, so somewhat confused about your point.
That paragraph from NCCN is quite interesting. It is describing medicine in general really, and belies the fact that oncology has probably one of the strongest evidence base across all medical fields. Take for example how many stents cardiologists have inserted long after contradictory evidence was available, or how many pointless back operations have been done, or how many people have sat through fruitless psychoanalysis.
> Well you discount the most important thing in the first line. 'Cutting-edge' is a funny way of saying 'most effective', as if it were somehow irrelevant.
I'm discounting it for this discussion because your argument is:
"meta-analyses are somewhat irrelevant" and "Meta-analyses are mostly there for trainees to notch up a paper." which is completely false.
Note a single clinical trial is still only considered "good quality" while multiple trials or meta-analyses are considered "high quality".
To address this new point you raised, when something has very promising early results we start using it in treatment (e.g. 3rd gen TKIs in adjuvant NSCLC) but until this weekend we had no 5 year OS survival for adjuvant use.
It's entirely possible something one thinks is "most effective" is later proven to not be (gen 1-2 TKIs, HIPEC, etc).
> That paragraph from NCCN is quite interesting. It is describing medicine in general really, and belies the fact that oncology has probably one of the strongest evidence base across all medical fields.
> Take for example how many stents cardiologists have inserted long after contradictory evidence was available, or how many pointless back operations have been done, or how many people have sat through fruitless psychoanalysis.
I'm not sure what point you are trying to make by addressing other specialties.
The National Comprehensive Cancer Network, comprised of multidisciplinary experts from 33 of the leading cancer centers in the country, is unequivocally the authority in oncology and is incredibly well respected. I'm going to defer to their opinion on the quality of evidence available and the hierarchy of evidence.
> Also one of your citations is seemingly casting doubt on the value of meta-analyses in oncology, so somewhat confused about your point.
The JAMA article states that the methodology in many studies does not meet NCCN/PRISMA criteria which is a well known, this says nothing about the relative value of good-quality meta-analyses (which are far more common now with the PRISMA update).
I'm really not sure why you think systematic reviews are irrelevant, this is a very radical viewpoint that I've seen no evidence of. Good meta-analysis > good RCT. The reality is that good quality studies of both types are uncommon in medicine, but the goal is still to use good SRs.
I don't think it is false. I can only speak of my experience as an oncology healthcare provider. I spend many hours each week digesting the literature, and <5% of that involves meta-analyses. In the multidisciplinary meetings I chair, we rarely discuss evidence from meta-analyses, but we are always talking about clinical trials. The NCCN guidelines were useful when I was a trainee, but otherwise they are too US-centric, and they are always out of date due to the frequency they are updated. This is why ASCO keeps issuing rapid updates in breast cancer for example (https://old-prod.asco.org/practice-patients/guidelines/breas...). There are 2 such updates this year already. If the primacy of meta-analyses were so great, why would they bother to issue rapid updates of what you class as low quality evidence?
But to give a concrete example, the problem with meta-analyses is well illustrated in the recent EBCTG meta-analysis published in the Lancet, a top tier journal. This involved over 100,000 patients, and explored concurrent chemotherapy regimens in breast cancer. The problem is that such regimens are not used anymore. The authors acknowledge in their own conclusion that this massive meta-analysis contradicts their own previous meta-analysis showing the superiority of sequential therapy. What exactly does one do with this? How does this help a patient get the right therapy? The treatment of various breast cancer subtypes has also evolved so much that the trials they meta-analyse are mostly obsolete. Hence my point, that meta-analyses are just not that useful in oncology, even truly massive well conducted ones published in prestigious journals. So it is not so simple as meta-analysis > RCT, that is merely lazy dogma. I find it hard to believe that anyone actually treating cancer patients would hold this view.
Of course most meta-analyses in oncology are not 100,000 patient behemoths conducted by consortia. They are much smaller studies, which usually don't bother to get patient level data, and just copy numbers from tables in the original papers while running through the Cochrane systematic review template.
And yet, here I am dubbed 'radical' at the bottom of a comment thread on Hacker News. Unfortunately the dogma around systematic reviews and EBM has exceeded its usefulness by quite some margin. The meta-analytic method was developed by psychologists trying to compile evidence about extra sensory perception of all things - an inauspicious beginning if there ever was one for the supposed cornerstone of medicine.
Speaking of ASCO, their methodology for recommendations is conducting their own systematic review and they start with Cochrane.
>Upon approval of the Protocol, a systematic review of the medical literature is conducted. ASCO staff use the information entered into the Protocol, including the clinical questions, inclusion/exclusion criteria for qualified studies, search terms/phrases, and range of study dates, to perform the systematic review. Literature searches of selected databases, including The Cochrane Library and Medline (via PubMed) are performed.
>After the systematic review is completed, a GRADE evidence profile and summary of findings table is developed to provide the guideline panels with the information about the body of evidence, judgments about the quality of evidence, statistical results, and certainty of the evidence ratings for each pre-specified included outcome.
Rapid criteria:
The criteria for a rapid recommendation update are:
1. that the identified evidence is of high methodological quality,
2. there is high certainty among experts that results are clinically meaningful to practice,
3. the identified evidence represents a significant shift in clinical practice from a recommendation in an existing ASCO guideline (e.g., change from recommending against the use of a particular therapy to recommending the use of that therapy; or a reversal to a recommendation) such that it should not wait for a scheduled guideline update.
A systematic literature review focused on the updated recommendation will be conducted by ASCO staff. Specifically, the immediate past guideline literature search strategy will be updated and filtered by search criteria specific to evidence informing the recommendation under review. All identified evidence will be quality- appraised using the GRADE methodology as outlined in Section 10 of this ASCO Guideline Methods Manual. The procedures used to draft the rapid recommendation update and deliberations by the expert panel will follow routine methods for all guidance products as outlined in this ASCO Guideline Methods Manual.
ASCO position on meta-analyses:
All these reasons can be used to make the excuse that a systematic review and meta -analysis should not be done, especially if resources aren’t available to hand search all journals of all languages, etc.
The solution is not to avoid doing a systematic review or meta-analysis, but to reveal to the reader what short cuts were taken (e.g., we included only peer reviewed published studies, or restricted our eligibility to studies published in English). This shows transparency, and then the readers can decide how important this problem is in applying the results of your meta -analysis to their situation.
Myths about meta-analyses
• A literature-based meta-analysis is not worth doing
So systematic reviews and meta-analyses are no longer useful?
That you're arguing with anecdotes is proof itself of why EBM is important.
>There are 2 such updates this year already. If the primacy of meta-analyses were so great, why would they bother to issue rapid updates of what you class as low quality evidence?
"A targeted electronic literature search was conducted to identify any additional phase III randomized controlled trials in this patient population. No additional randomized controlled trials were identified. The original guideline Expert Panels reconvened to review evidence from EMERALD and to review and approve the revised recommendations."
Where is the meta-analysis? Where is the funnel plot? What are you even arguing about? They issued an update because of one trial.
Here is another one from June 2022, a major change to how one type of breast cancer is managed, in the methods:
"A targeted electronic literature search was conducted to identify phase III clinical trials pertaining to the recommendation on immune checkpoint inhibitors in this patient population. No additional randomized trials were identified. The original Expert Panel was reconvened to review the key evidence from KEYNOTE-522 and to review and approve the revision to the recommendation."
Where is the meta-analysis? Again, what are you trying to argue? They issued an update because of one trial.
There are two updates this year, one about HER2 testing, and one about ESR1.
Perhaps you’re unaware but not every systematic review is or can be a meta-analysis. Meta-analyses can and are also included in systematic reviews released by ASCO.
The criteria for a rapid update is listed in the comment you replied to, I have no misunderstanding of why they publish them but you are trying to misrepresent this as deviating from EBM.
Your words were: “Unfortunately the dogma around systematic reviews and EBM has exceeded its usefulness by quite some margin.”
I’m not sure what point you’re trying to make but seeing as you’re attempting to argue ASCO’s own position and methodology with anecdotes and conjecture based on one specific area of breast cancer treatment and extrapolating to the entire field of oncology I’m not sure there’s a point in engaging further.
You can refer to the full ASCO statement I linked which discusses meta-analyses for their arguments.
The president of Medicens sans frontieres, Rowan Gillies, gave a speech once. He had a drawing of a patient and the doctor inside a circle. His comment was 'Other people keep trying to climb into this circle. They can all fuck off.' Excuse his profanity, but I offer this response to you, who knows nothing about what I do, and understands nothing about quality healthcare.
Of course if you ask a clincian if they should be regulated more, they will say no. The fewer stakeholders involved in the relationship between patient and doctor, the more power the doctor has. The truth is there should be others involved, at the very least the government health agencies.
Actually doctors would be universally happy to get more help and support to deliver better healthcare. That you don't talk about that is quite telling, or is that what you mean by 'regulation'?
Casting the patient doctor relationship in terms of power dynamics is a bogus sociological construct divorced from reality. The true division of power is between those who fund healthcare and those who receive it, I would start there if you think things need to be improved.
Yes - regulation is not the same as better support for the healthcare system and I am surprised you do not get that distinction. Regulation is about putting rules into place to prevent fraud, malpractice and anything where the clinician may abuse the system to the detriment of patients or other healthcare professionals around them. In addition to regulation, the government should also support the healthcare system through better funding and (patient and clinical) education. It can be true that both more regulation and better support are needed.
To say there is no power dynamic between patient and doctor is wildly ignorant. There obviously is one, even if the patient is not paying the doctor directly for their services. Not sure when (if?) you went to medical school, but this is taught to all medical graduates. The field of medical ethics exists for a reason.
What if the patient wants other people in the circle? (I do, when I'm a patient. I've never really trusted doctors I've had due to my perception of competing motivations, and they've given me reason for that distrust more than once).
Sorry to hear you have suffered poor quality interactions with doctors. Being honest, if there is no trust in the relationship between patient and doctor, then nothing else matters much as the experience will be poor on both sides.
Patient can of course bring whoever they want into the circle. The problem is the intrusions that neither healthcare provider nor patient want.
From the point of view of the experienced patient it's not a single circle, it's a Venn diagram with the patient at the center of multiple overlapping circles.
But sure, uninvited third parties shouldn't butt their head in often, except for the occasional regulator.
I can't speak to parent's practice but the cancer centers I've worked at follow the NCCN guidelines (apart from patients enrolled in trials, although this is also a NCCN recommendation) and many cases are reviewed in multi-disciplinary conferences to homogenize practice within an institution.
Although guidelines are just that (i.e. not mandatory to adhere to) I really doubt an oncologist in US/Canada "practices as they please".
The biggest issue, IMHO, is that clinical trials are often unethical. This is both in theory and especially in practice. I say this as a physician and clinical trial investigator.
EBM deals with this by saying ‘there is no viable alternative’, a remarkable statement of epistemological nihilism that enables much low quality snd pointless research.
Can you give an example of unethical trials where “there was no viable alternative” was what got the trial past an IRB? I’m more familiar with inverse complaints that trials are blocked by red tape and hypothetical concerns that are objectively small in actual QALY harm.
(I’m sure this varies by jurisdiction too; I have only heard bad thing about US IRBs)
IRBs do not evaluate the value of a research endeavour. They are in fact unable to do this due to lack of knowledge and expertise. They approve trials which fit the mold of trials they have seen before. Why does a clinical trial get done? The main reason is that someone, usually a pharmaceutical or device company, is willing to pay for it.
Some recent examples of problems with clinical trials:
I'd have assumed it was when they give sick people the placebo when they have a reasonable hunch (but not a published study that people trust) that the real medication would actually save them
Some people aren't worth it. Some people are just miserable. Not depressed, just miserable. Some people are insecure, vindictive, manipulative, querulous, disagreeable. Having a relationship with them will be as much as 100x more difficult than another person. This is the hard lesson I learned after 4 years in a thoroughly depleting relationship where I tried everything to make it work. When I look at my friends in their 40s, some have great partners and some don't, and the ones that don't have a truly execrable life despite good health, jobs, income, cars, holidays etc. Choosing a good partner matters more than almost anything else.
Don't get me wrong - I am not advocating we put all the bad partners in a camp somewhere. They are fundamentally in a state of suffering, and benefit from help and support. But the way to deliver that support is not by being in a romantic relationship with them.
If you are good partner material, consider the following asymmetry: A good partner can be in a relationship with a poor partner, but two poor partners will almost never be in a relationship together. This means that being a good partner increases your chance of ending up with a poor partner.
I think this is a great comment, with much broader implications for living well. Unmooring the sense of self from external factors is one important aspect of spirituality.
This reference [1] tends to support what you are saying:
> For this thesis, an atmospheric propagation code named ANCHOR (Atmospheric NPS Code for High Energy Laser Optical pRopagation) was developed and utilized to study the propagation of high energy lasers in various atmospheric conditions and for numerous laser configurations. The ANCHOR code accesses existing industry databases to obtain relevant optical properties for various atmospheres and then uses scaling laws to simulate laser propagation through the defined environments.
ANCHOR accounts for the effects of atmospheric diffraction, turbulence, platform jitter and thermal blooming on the laser beam, and outputs on-target irradiance and power-in-the-bucket profiles for a wide range of laser wavelengths. Several known physical trends associated with laser propagation will be reproduced, and the results will be compared to the industry accepted propagation code Wavetrain.
The results of ANCHOR studies will indicate that the 100 kW-class high energy laser can effectively engage slow-moving targets at ranges greater than five kilometers in clear weather by delivering enough energy to melt 0.1 liters of one millimeter-thick aluminum aircraft skin in five seconds. For hazy, turbulent, and rainy conditions, the laser can effectively engage targets from ranges closer than three kilometers, but reasonable dwell times are only achieved for ranges closer than two kilometers.
Yes, find a local sequencing provider and arrange to do a SNP chip or whole genome sequencing. In the contract ask that they delete your data after delivering it to you. This will be:
1. Expensive - probably at least 2 - 3 thousand dollars.
2. Require you to do your own analysis.
Obviously you can't be 100% sure they will delete your genomic data, but they have no incentive to keep it.
Or find a European genetics laboratory that offers this service.
Even though you may not be European, the lab will need to follow the much stricter data protection policies of the EU and will probably not have a different handling for non EU customers (but ask them).
I write this as someone who has worked at one of those in the EU.
Since you are in the field, would you give some names of such labs in the EU? The quality of results when searching for "European genetics laboratory" aren't great.
> the lab will need to follow the much stricter data protection policies of the EU
But will they still retain it indefinitely on a server somewhere? If yes, it will leak eventually. Or the government will lobby to get access to it to "protect the children".
Re 2: do you know of tools that don't involve the cloud and allow you to do such analysis? Ideally FLOSS. I could only find DbSNP and SNPedia but they are datasets, not sure if there are tools built on top of that like Promethease.
Re WGS there are a lot of well established tool chains that are FLOSS (eg https://github.com/bcbio/bcbio-nextgen). You could run alignment and variant calling on a beefy workstation. A laptop would potentially work. Easy to test this with publicly available raw data. Another option: The sequencing provider often will run alignment and some default variant calling for you. Annotating and analysing these variants can be done on pretty much any computer, all with open source software. A SNP chip is even easier to deal with as the computational requirements are less.
Interpreting the results is a more manual process. Really depends on what you are interested in.
There are thousands of sequenced human genomes available to access for research purposes (1000 genomes project, UK biobank etc) so one additional genome adds no marginal value.
Why to delete anything sellable? I always do all kinds of illegal copies of somebody's PD on all places I ever worked as employee. Typically by stealing some papers from garbage because photographing takes too much time per one 'drop'. With enough wit of choosing workplaces I use to have some incomes from my strategy.
> Viewers will be able to "order up" films—for example, "I want a film about a panda and a unicorn who save the world in a rocket ship. And put Bill Murray in it."
> From there I believe viewers will be given the ability to be digitally scanned themselves, and pay extra to have themselves inserted in these custom films. You'll also start to see licensing deals made with studios, so that viewers can order up older films like "Star Wars" and put their face on Luke Skywalker's body, and their ex-wife's face on Darth Vader's body, and so on.
This is pretty far from what most people want, I think. How are you going to chat about your AI generated custom personal vendetta movie around the watercooler?
Imagine a movie about divorce, except instead of Kramer vs Kramer it's you vs your wife, with your kids, in your city. It could elevate the human experience in a way previously unfathomable. The AI would be a sort of mix between a psychologist, lawyer, and film producer, stringing together a creation that tugs at your deepest emotional heartstrings.
Or imagine a horror where you get to slaughter your worst enemies.
Is it a Pandora's box? Certainly. But I think it's coming.
It's a red herring, only reaction videos would do that. Studios are going to double down on the current quantity over quality gambit they're running and not mention AI after this because AI tears are the first real entertaining thing to come from them in a decade. AI will not divide us.
I expect a lot more meandering miniseries MMO style where tangents for quality are pay-per-view. Mainly because if they try to starve out their audience with dither and delay tactics pirate AI will win.
Although quite uncharitable, I can believe this has some truth to it. She underwent a visible transformation for the trial, wearing her hair down and changing her entire mien. Quite a contrast to her appearance during the Theranos years.
I feel that alignment is not just hard but impossible, at least if you want something truly useful. Maybe the only thing you can do is let an AI develop and observe its nature from a distance, say in a simulated world running at high speed which it does not know is simulated. You can hope it will develop principles that do align with your own, that its essential nature will be good. Sometimes I wonder if that is what a greater intelligence is doing to us.
Another fine example of the author's point is the ivermectin in covid meta-analytic nonsense (which I cannot even bring myself to link), where a bunch of small rubbish trials are meta-analysed into a 'flawless' body of evidence while double blind randomised trials are impugned.