Why does significance change when I switch level or add waves of data in Askia Analyse?
This post explains which count is considered for the ‘Count threshold’ setting in Col sig advanced options. There is detail on how to calculate ‘Counts when independent’ and why this can change sig testing when the total base grows.
From time to time clients will raise a query regarding sig testing (Column significativity on closed) for a particular cell changing when the total base of their table grows, even though the count and % for cells in the table remains the same. E.g. this can happen when new waves of data are added or the level has been changed to show the response counts rather than respondent counts.
The reason for this is almost certainly the Count threshold setting in the col sig advanced options; it has been left as the default value, 5.
The Count threshold does not use the counts you would normally see in your tables but, instead, it uses the counts from the calculation Counts when independent.
You need to use the count when independent rather than the actual count because the usual count in the cell (crossed count) can still carry importance at low or zero counts.
Below is an example of when the the sig testing changes even though the count and % have remained the same (see column G)
5 is the default because this doesn’t bring to your attention differences which are not statistically important. The side effect is that you can have changes as the total base grows over time (and hence changes the count when independent). If this behaviour is not in line with requirements, the Count threshold can be changed to 0.
This applies to Column significativity on closed but you cannot perform the same calculation for Column significativity on numeric. The Count threshold setting exists in its advanced settings but it is not applicable and has no effect.
