The IMJ, HPAT and bad science

2Scoops

What is it about the IMJ that it casually admits all sorts of poorly-designed studies on HPAT, that are really little more than anti-HPAT opinion pieces? First, we had this effort from Beaumont, upon which I noted at the time:

2Scoops wrote:

I think the biggest problem is that their results are exactly meaningless. Bizarrely, they didn't administer the HPAT itself, just a handful of sample questions with no test validity. And let's ignore the fact that these people would have matriculated anyway once you factor in their presumably high LC scores. Perhaps most damningly of all, there were no significant differences between consultant surgeons, NCHDs or med students - didn't stop the authors arguing the exact opposite!

There are heaps of other scientific flaws in this 'study' that limit the ability to generalize to virtually nothing, but I think they show their true colours in the discussion where their contempt for HPAT is laid bare. These guys had a very clear agenda against the test, then cobbled together a weak scientific investigation to fit around their preconceived biases. It's a sad reflection on science and looks very bad, IMO, for the Irish Medical Journal whose reviewer(s) let it through without asking for a discussion, at minimum, of its scientific limitations. It's practically an opinion piece.

For the record, I'm not a fan of HPAT. But then, I'm not a fan of the LC either. I'm just tired of of people using a thin veneer of science, the noble art of impartial investigation, to add authority to their unfounded opinions.

Edit: Study is here btw:
http://www.imj.ie//ViewArticleDetails.aspx?ArticleID=5733

Now, we are treated to round 2 from AMNCH and this equally poorly-designed study:
http://www.imj.ie//ViewArticleDetails.aspx?ArticleID=6655

Once again, I’m brushing over the many bizarre methodical concerns to cut straight to the issue at hand: thinly veiled attacks on HPAT. In any other world, the HPAT having independent predicative power [although some serious statistical concerns that the authors can even say say this, btw] would be a good thing; clearly it’s measuring something the LC is not. But in the IMJ you can argue that this means that it is redundant and LC can be used on its own! A similar argument could be advanced that since socioeconomic background is an independent predictor of college success, then why bother with the LC? Just give college places based on what job the student's father has. A modest proposal, I’m sure you’ll agree. :pac:

The IMJ is quickly losing credibility. At least some headline-hungry SpRs are getting their names in the newspapers…

Now, can we get some real science on HPAT, please?

HQvhs

I have to agree. I read that study and I was shocked at hoe poor the quality was. I was even more shocked that the normally reputable Dr. Muiris Houston gave it credence in his piece in the Irish Times yesterday. If doctors, journalists or newspapers do not like the HPAT, that's fine (I'm not the biggest fan either), but I would much rather they come out and say so (and give their opinion and reasons) rather than hiding behind shoddy 'scientific studies'.

Piste

2Scoops wrote:

Perhaps most damningly of all, there were no significant differences between consultant surgeons, NCHDs or med students - didn't stop the authors arguing the exact opposite!

The way I understood their argument was that consultants- by virtue of the fact that they have risen to the level of consultant surgeon- have the skills purported to be tested for by the HPAT:

A Comparison of Performances of Consultant Surgeons, NCHDs and Medical Students in a Modified HPAT Examination wrote:

"The ability to interpret data along with the more intuitive “wait and see” decisions, combined with the ability to make a decision based on suboptimal knowledge and change as the situation evolves, are critical determinants in the performance of a doctor. These are skills gained with time and experience, and would be akin to the traits purported to be tested by the aptitude tests. If a true measure of these skills, consultants should clearly have scored higher. "

That's the argument they're making, that consultant already have these skills but the HPAT results dont reflect this, meaning that many people who have the potential to become great surgeons will be screened out by this test, which is inherently unfair.

2Scoops

Piste wrote: »

The way I understood their argument was that consultants- by virtue of the fact that they have risen to the level of consultant surgeon- have the skills purported to be tested for by the HPAT:

I understand their argument perfectly well, it's just not supported by the data. In HPAT, which I am unaware is meant to select for consultant surgeons btw, the students score equally to qualified consultant surgeons, so couldn't I equally use that to support the fact that HPAT is measuring a consistent factor that doesn't change over time, or that we are selecting students with similar profiles to consultant surgeons? And isn't HPAT purported to measure a relatively consistent factor that shouldn't change much over time?

Piste wrote: »

That's the argument they're making, that consultant already have these skills but the HPAT results dont reflect this, meaning that many people who have the potential to become great surgeons will be screened out by this test, which is inherently unfair.

The point they're making is that they don't like HPAT. The science is incidental. There is nothing to suggest it would have ruled out great surgeons, since they scored equally well as others and we don't know how great these surgeons are in the first place, or how much greater their peers would have been! And we're not even getting to the point that the HPAT is not the sole instrument of selection. Oh, and the fact that it is not unfair, the same can not be so easily said about the LC.

As I said before, I don't particularly like our current system of selection, HPAT or LC. But the above is garbage: opinion dressed as science; enough to fool journalists and the scientifically illiterate; enough to validate the preconceived prejudice of anti-HPATers. Just not science though.

I would seriously like to see some real science on the issue. This isn't it.