Pruebas A / B: prueba z vs prueba t vs chi cuadrado vs prueba exacta de fisher

Estoy tratando de entender el razonamiento eligiendo un enfoque de prueba específico al tratar con una prueba A / B simple (es decir, dos variaciones / grupos con una respuesta binaria (convertida o no). Como ejemplo, usaré los datos a continuación

Version  Visits  Conversions
A        2069     188
B        1826     220

La respuesta principal aquí es excelente y habla sobre algunos de los supuestos subyacentes para las pruebas de z, t y chi cuadrado. Pero lo que me parece confuso es que los diferentes recursos en línea citarán diferentes enfoques, y ¿usted pensaría que las suposiciones para una prueba básica A / B deberían ser más o menos las mismas?

1. Por ejemplo, este artículo usa puntuación z :
2. Este artículo usa la siguiente fórmula (que no estoy seguro si es diferente del cálculo de zscore):

3. Este artículo hace referencia a la prueba t (p 152):

Entonces, ¿qué argumentos se pueden hacer a favor de estos enfoques diferentes? ¿Por qué uno tendría una preferencia?

Para incluir a un candidato más, la tabla anterior se puede reescribir como una tabla de contingencia 2x2, donde se puede usar la prueba exacta de Fisher (p5)

              Non converters  Converters  Row Total
Version A     1881            188         2069  
Versions B    1606            220         1826
Column Total  3487            408         3895

Pero de acuerdo con este hilo, la prueba exacta de Fisher solo debe usarse con tamaños de muestra más pequeños (¿cuál es el límite?)

Y luego están las pruebas t y z pareadas, la prueba f (y la regresión logística, pero quiero dejar eso fuera por ahora) ... Siento que me estoy ahogando en diferentes enfoques de prueba, y solo quiero poder Haga algún tipo de argumento para los diferentes métodos en este simple caso de prueba A / B.

Usando los datos de ejemplo obtengo los siguientes valores p

1. https://vwo.com/ab-split-test-significance-calculator/ da un valor p de 0.001 (puntaje z)

2. http://www.evanmiller.org/ab-testing/chi-squared.html (usando la prueba de chi cuadrado) da un valor p de 0.00259

3. Y en R fisher.test(rbind(c(1881,188),c(1606,220)))$p.valueda un valor p de 0.002785305

Que supongo que están muy cerca ...

De todos modos, solo espero una discusión saludable sobre qué enfoques usar en las pruebas en línea donde los tamaños de muestra generalmente son miles y las relaciones de respuesta a menudo son del 10% o menos. Mi instinto me dice que use chi-cuadrado, pero quiero poder responder exactamente por qué lo estoy eligiendo entre las otras muchas formas de hacerlo.

We use these tests for different reasons and under different circumstances.

1. $z$-test. A $z$-test assumes that our observations are independently drawn from a Normal distribution with unknown mean and known variance. A $z$-test is used primarily when we have quantitative data. (i.e. weights of rodents, ages of individuals, systolic blood pressure, etc.) However, $z$-tests can also be used when interested in proportions. (i.e. the proportion of people who get at least eight hours of sleep, etc.)

2. $t$-test. A $t$-test assumes that our observations are independently drawn from a Normal distribution with unknown mean and unknown variance. Note that with a $t$-test, we do not know the population variance. This is far more common than knowing the population variance, so a $t$-test is generally more appropriate than a $z$-test, but practically there will be little difference between the two if sample sizes are large.

With $z$- and $t$-tests, your alternative hypothesis will be that your population mean (or population proportion) of one group is either not equal, less than, or greater than the population mean (or proportion) or the other group. This will depend on the type of analysis you seek to do, but your null and alternative hypotheses directly compare the means/proportions from the two groups.

3. Chi-squared test. Whereas $z$- and $t$-tests concern quantitative data (or proportions in the case of $z$), chi-squared tests are appropriate for qualitative data. Again, the assumption is that observations are independent of one another. In this case, you aren't seeking a particular relationship. Your null hypothesis is that no relationship exists between variable one and variable two. Your alternative hypothesis is that a relationship does exist. This doesn't give you specifics as to how this relationship exists (i.e. In which direction does the relationship go) but it will provide evidence that a relationship does (or does not) exist between your independent variable and your groups.

4. Fisher's exact test. One drawback to the chi-squared test is that it is asymptotic. This means that the $p$-value is accurate for very large sample sizes. However, if your sample sizes are small, then the $p$-value may not be quite accurate. As such, Fisher's exact test allows you to exactly calculate the $p$-value of your data and not rely on approximations that will be poor if your sample sizes are small.

I keep discussing sample sizes - different references will give you different metrics as to when your samples are large enough. I would just find a reputable source, look at their rule, and apply their rule to find the test you want. I would not "shop around," so to speak, until you find a rule that you "like."

Ultimately, the test you choose should be based on a) your sample size and b) what form you want your hypotheses to take. If you are looking for a specific effect from your A/B test (for example, my B group has higher test scores), then I would opt for a $z$-test or $t$-test, pending sample size and the knowledge of the population variance. If you want to show that a relationship merely exists (for example, my A group and B group are different based on the independent variable but I don't care which group has higher scores), then the chi-squared or Fisher's exact test is appropriate, depending on sample size.

Does this make sense? Hope this helps!

For a 3 way test you usually use an ANOVA rather than 3 separate tests. Please also check on the Bonferroni correction before multiple testing. Please use this https://www.google.com/search?q=testing+multiple+means&rlz=1C1CHBD_enIN817IN817&oq=testing+multiple+means+&aqs=chrome..69i57j69i60l3j69i61j0.3564j0j7&sourceid=chrome&ie=UTF-8