Statistical significance difference advice anyone?

Tomc77

I've carried out a study where I observed people using a certain type of device they had been given. I wanted to see what percentage were able to use the device properly. THe overall stat is one thing but do these results demonstate significant differences between the devices.

Am I over simplifying the study by saying that if I show one, device A: % correct use as 30% +/- 2 CI 95%, and another B, at 23% +/- 4 CI 95% that I have shown (everything else being equal that A is easier to use).

Any advice would be gratefully received
Tom
(results table attached)

Ostrom

Did you repeat the test with the same people on each? Did you use a scale or just a yes/no for your outcome? I'm remembering an inference test for proportions in one of my stats books, i'll check tomorrow and post. If you used a scale then you have a few parametric test choices.

Time Magazine

How are you calculating your variance?

Tomc77

Efla,

No I didn't repeat the test on the same peopleas each person was actually using the device already. Although patients were marked out of 10 for each device a score of 9 meant the person wasn't using properly it so in essence it was all or nothing yes/no. (a score of 9 is in effect the same as 0 as the device won't work properly unless all steps are followed- and users are often unaware the device hasn't worked).

I haven't seen anything os stats in over 10 years so I've forgotton everything. I realised a bigger problem which will make comparisons impossible and so in essence answers my original question. As certain devices are percieved as easier to use the are often given to older or less dextrous (if that's a word) people. This would schew any comparison completely.

Time Magazine- I was just using the above figures as an example. They aren't calculated. I'm ashamed to say I used an on-line calculator based on sample size for my actual variace (thankfully my former Maths teacher will never see this!!)

I know I won't be able to compare the results but is my question correct for a yes/no survey with no factors that could be schewing it? "Am I over simplifying the study by saying that if I show one, device A: % correct use as 30% +/- 2 CI 95%, and another B, at 23% +/- 4 CI 95% that I have shown (everything else being equal that A is easier to use)."

Thanks guys

Time Magazine

Tomc77 wrote: »

Time Magazine- I was just using the above figures as an example. They aren't calculated. I'm ashamed to say I used an on-line calculator based on sample size for my actual variace (thankfully my former Maths teacher will never see this!!)

I'd be wary of this as it might be using a formula that's inappropriate.

It depends on your scale. Whether you use the 1-10 scale or the "Yes" or "No" idea is obviously something only you can decide upon depending on the details of the devices etc. I'm an economist so rarely use statistics related to whether things "work" or not (my scales are almost always continuous) but grabbing an undergrad textbook here it looks to me like you're looking at a binomial scenario, the variance of which is:
[latex]\displaystyle \sigma^2 = n \pi (1 - \pi)[/latex]
where n is population size and pi is the probability of a success on each trial. (I think when you're just testing people once, you treat n = 1.) However if pi is unknown, which I imagine it is, it seemed to me to be impossible to calculate the variance. That's why I asked how you calculated yours.

I know I won't be able to compare the results but is my question correct for a yes/no survey with no factors that could be schewing it? "Am I over simplifying the study by saying that if I show one, device A: % correct use as 30% +/- 2 CI 95%, and another B, at 23% +/- 4 CI 95% that I have shown (everything else being equal that A is easier to use)."

Conditional on your CI's being correctly calculated, I am tempted to say that yes you can say A is easier to use. To me it's simple: You're confident that the lower bound of A is 28% and you're confident that the upper-bound for B is 27%, so you're confident that A > B.

However, if you want to be precise, handy undergraduate book says you need these boys:

[latex]\displaystyle t = \frac{ \overline{X_A} - \overline{X_B} }{s_p^2 \left( \frac{1}{n_A} + \frac{1}{n_B} \right)}[/latex]

where
[latex]\overline{X_i}[/latex] is the mean of sample i
[latex]n_i[/latex] is the number of obs in sample i
[latex]s_p^2[/latex] is the pooled population variance, given by

[latex]\displaystyle s_p^2 = \frac{ (n_A-1)s_A^2 + (n_B-1)s_B^2 }{n_A + n_B - 2}[/latex]