mightymetrika-ccvfo

Who Said It Best Episode 1: Daniel McNeish

Mon, 19 Aug 2024 14:25:25 GMT

Mighty Metrika focuses on statistical methods and mathematics for the analysis of small sample size data. As such, the project runs the risk of people with small sample sizes using tools and methods from mightymetrika.com and becoming over confident in their results because they used "small sample size methods."

The long term rigorous goal to combat this disservice is to host citizen science projects, include simulation function in R packages, and share simulation results from the literature and from mightymetrika.com tools through blogs.

A short and quick way to combat misuse is through the Who Said It Best series. The series will share some of the best warnings from the small sample size statistical literature.

In the Conclusion section of Daniel McNeish's paper Challenging Conventional Wisdom for Multivariate Statistical Models With Small Samples he shares a clear and wonderfully worded warning:

Resources for Building R Shiny x PostgreSQL x AWS EC2 Apps

Tue, 25 Jun 2024 15:27:51 GMT

This is a quick blog post to list some of the essential resources that I needed to get a citizen science app up and running. The app uses:

R
Shiny
PostgreSQL
Pool
AWS EC2

The post is basically a way for me to bookmark resources that I found useful and also as a way to say thank you to the folks that put these resources up online.

Main R Workflow

Persistent Data Storage by Dean Attali: https://deanattali.com/blog/shiny-persistent-data-storage/

The main R workflow uses the MySQL section of Dean Attali's Persistent Data Storage. This workflow allows you to get a Shiny app up-and-running where the shiny app collects data , sends results to a database, and displays data from database.

Swapping Out MySQL for PostgreSQL

Instead of using MySQL like in the Persistent Data Storage blog, I used PostgreSQL with the RPostgres R package.

Best Practices in Working with Databases > Databases > PostgreSQL : https://solutions.posit.co/connections/db/databases/postgresql/

Posit's Best Practices in Working with Databases has a short section that covers connecting to a PostgreSQL database using RPostgres.

Emily Rieder Using Databases with Shiny: https://www.emilyriederer.com/post/shiny-db/

This blog post shows how to use the RPostgres R package with the DBI R package. Shows useful syntax and discusses best practices and pitfalls.

Both resources above use the RPostgres x DBI R package combination.

Swapping Out DBI for pool

Posit's Why Pool: https://rstudio.github.io/pool/articles/why-pool.html

To swap DBI with pool, I used the 'Why Pool?' blog post by Posit. The other articles on the website have good examples as well.

Open for Communication

Helena Alexander's How to Configure PostgreSQL for Remote Connections: A Beginner’s Guide: https://blog.devart.com/configure-postgresql-to-allow-remote-connection.html

I used the instructions for Windows to make sure the .conf files and the Windows Firewall were set up correctly.

The Firewalls section of Posit's Accessing Databases with ODBC chapter of the shinyapps.io User Guide: https://docs.posit.co/shinyapps.io/guide/applications/#firewalls

Since I deployed with shinyapps.io, I used these instructions to whitelist the shinyapps.io IP addresses. I also used this same worfklow for whitelisting IP addresses when I needed to test the app by running the Shiny app on one EC2 instance while the PostgreSQL database was on another instance.

.Renviron File

Posit forum's How to set a variable in .Renviron: https://forum.posit.co/t/how-to-set-a-variable-in-renviron/5029

I used cderv's response to the OPs question to set up a .Renviron file.

StackOverflow's shinyapps.io won't read environment variables from .Renviron: https://stackoverflow.com/questions/77579704/shinyapps-io-wont-read-environment-variables-from-renviron

I used the StackOverflow post to make sure the .Renviron file was stored in the correct place for deployment on shinyapps.io. I'm not too familiar with security so I don't know if this is good practice or not.

I combined the Posit x StackOverflow post when deploying to shinyapps.io by using the:

usethis::edit_r_environ("project") will open the one in your project

bullet point from crderv's Posit forum response since my app.R file was in the main project folder for deployment.

Next Steps

The apps I made are currently on https://www.mightymetrika.com/citsci . The source code for these apps are on GitHub:

https://github.com/mightymetrika/npboottprm/blob/master/R/replext_pgsql.R
https://github.com/mightymetrika/mmirestriktor/blob/master/R/replext_pgsql.R

I hope to improve on these apps in the future by:

Including user-logins
Add some sort of graphical exploration of the databse or better filtering
Learning more about best practices with security
Learning more about the pool package to see if I can use it in a better way
Add tool tips and information about data columns to serve as a type of interactive codebook to make the inputs and database columns easier to understand

In the coming weeks I'll have a few blog posts about the individual apps in order to document how the data is supposed to map onto the papers from which the replext (replication & extension) simulation tables are based.

BFfe: Test Update

Mon, 10 Jun 2024 13:51:59 GMT

In 'mmibain' v0.2.0, the unit tests are passing at the moment, but on r-devel-linux-x86_64-debian-clang it really seems to be hit or miss. I believe that when the test fails it is do to the new BFfe function which is a case-by-case type implementation of ' bain ' for linear models; however, I used a unit test which relies on a synthetic data set where I generated random numbers and then just used the rep() function to group observations by participants. As such, the data generating process does fit the statistical model and sometimes the random data set that is generated does not make it through bain::bain() without error. I have already changed the unit test and corresponding Roxygen2 documentation example on the Mighty Metrika GitHub and this blog post will walk through the new data and model. But just for further context, here is the original code that sometimes runs through and sometimes throws and error.

# Create data

ex_dat <- data.frame(

participant = rep(1:10, each = 10),

x = rnorm(100),

y = rnorm(100)

)

# Run analysis

res <- BF_for_everyone(.df = ex_dat, .participant = "participant",

formula = "y ~ x", hypothesis = "x > 0")

The BF_for_everyone() function from the 'mmibain' package implements the methods discussed in this paper:

Klaassen, F. (2020). Combining Evidence Over Multiple Individual Analyses. In R. van de Schoot & M. Miočević (Eds.), Small Sample Size Solutions: A Guide for Applied Researchers and Practitioners (1st ed., pp. 13). Routledge. <doi:10.4324/9780429273872-11>

The paper authors did not actually name the method BF for everyone. I used this name for the function because it sounds sufficiently close to best-friends-forever. I had to name it something.

The Loblolly Dataset

The new unit test and example will be based on the Loblolly dataset which is one of the builtin datasets. I chose this data set because it has repeated measurements so it fits the data structure that the BF_for_everyone() function was designed to work with. The dataset is described on rdocumentation.org as follows:

Let's pull a quick Search Labs | AI Overview to get a bit more context.

And let's also build a quick ggplot2 visualization of the data:

The new unit test and example for BF_for_everyone() will use the the Loblolly dataset to fit a height ~ age linear model with the hypothesis age > 2.5. You can find this example and unit test on the Mighty Metrika GitHub. In what follows, we will run this statistical model using the BFfe shiny app on mightymetrika.com. We'll structure it as a BFfe tutorial.

BFfe Tutorial

In this tutorial we will use the Loblolly dataset which is built into R to conduct an analysis using the BFfe Shiny application.

Step 1: Open the BFfe App

You can access the app by:

Going to mightymetrika.com
Clicking on Tools
Clicking on BFfe

This will open the Shiny application in your browser. Alternatively, you can open the app through R as follows:

Open RStudio
Run install.packages("mmibain") in your console
Run mmibain::BFfe() in your console

Once you have the app up and running, you should see the following screen:

Step 2: Upload Loblolly Data

Before uploading the Loblolly data to the app we need to make sure we have a copy of the data set in csv format on our hardrive. In R we can accomplish this with the code:

write.csv(Loblolly, file = "Loblolly.csv", row.names = FALSE)

I ran this code on my computer to produce the data set that you can download using the button below.

Now you can upload this data to the BFfe app as follows:

Click the BROWSE... button
Navigate to Loblolly.csv on your hard drive
Select Loblolly.csv
Click Open

If everything worked, you should see the following screen.

Step 3: Set-up the Run

Now to set-up and the run the analysis we need to:

Enter the Formula: height ~ age
Enter the Hypothesis: age > 2.5
Enter the Participant Variable: Seed
Click Run Analysis

Our results will not match exactly since we are not using a seed. But after completing these steps, you should see results that look like the following:

Step 4: Interpret Results

For a full introduction to interpreting results from 'bain', I recommend A Tutorial on Testing Hypotheses Using the Bayes Factor .

In our example we are testing the hypothesis H1: age > 2.5 for each seed. Notice that the note in the 'bain' output that states:

"BF.u denotes the Bayes factor of the hypothesis at hand versus the unconstrained hypothesis Hu. BF.c denotes the Bayes factor of the hypothesis at hand versus its complement. PMPa contains the posterior model probabilities of the hypotheses specified. PMPb adds Hu, the unconstrained hypothesis. PMPc adds Hc, the complement of the union of the hypotheses specified."

Our main focus in this analysis is BF.c for H1 which is labeled BF1c. In A Tutorial on Testing Hypotheses Using the Bayes Factor BF.c is described as follows:

The first section of the results, Distribution of Bayes Factors, just returns descriptive statistics and a boxplot of the BF1c for each seed. This section shows us that the median BF1c is over 2.5.

The Geometric Mean of the Product of Bayes Factors presents metrics introduced in the Klassen's paper :

The gPBF in our example suggests that the average seed's evidence is 2.61 times stronger in favor of our hypothesis compared to the complement. The ER of 0.79 suggests that 79% of the seeds have evidence in favor of our hypothesis compared to the complement.

The last section, BAIN Results for Seed: 301, shows the 'bain' output for a particular seed. You can use the Select Participant drop down underneath the Run Analysis button to select a different seed.

mmibain 0.2.0 Release Party

Mon, 27 May 2024 14:00:01 GMT

mmibain 0.2.0 is now available on CRAN: https://CRAN.R-project.org/package=mmibain . The updated package has a new function and a corresponding app.

BF_for_everyone()

The Bayes Factors for Everyone, BF_for_everyone(), function is a function which based on the work of Fayette Klaassen's Combining Evidence Over Multiple Individual Analyses :

"This chapter presents a Bayesian method to evaluate hypotheses for each person in a sample and aggregate this result to answer the question whether a hypothesis holds for everyone in the sample, rather than on average."

The main workhorse of the method is Ba yesian In formative Hypothesis Testing which you can learn more about on the bain website as well as in Mighty Metrika's Quick mmibain Tutorial blog post .

The method is special because it combines two major methods that often show up in the small sample size statistics literature and on mightymetrika.com:

Informative Hypothesis Testing
Informative Hypothesis Testing blog post
Quick mmirestriktor Tutorial blog post
Quick mmibain Tutorial blog post

Case-by-case methods
Ordinary least squares trajectories and case-by-case linear models as presented in the OLStrajr R package
mmiCATs Release Part blog post

Informative hypothesis testing is a valuable method in small sample size statistics because when a researcher specifies hypotheses based on their expectations gleaned through experience in their field, "power can be gained and inherently a smaller sample size is needed."

Case-by-case methods are are important in situation where the data generating method suggests that standard errors might be off due to lack of independent observations but there is not enough data to trust traditional methods (such as mixed effects models and cluster robust standard errors). Also, as stated in Klaassen's book chapter , case-by-case methods can also be important when you want to know if the hypothesis holds for each person in the sample.

The basic workflow for BF_for_everyone() is the following:

Fit a stats::lm linear model for each participant
Specify a hypothesis for bain
Use the fitted linear model and bain hypothesis to fit a bain model for each participant
Present Bayes Factors for each participant
Aggregate Bayes Factors by presenting the
gPBF
Evidence Rate
Stability Rate

These aggregate measures are described as follows in Klaassen's chapter :

The version of BF_for_everyone() function that's available in mmibain 0.2.0 only allows using stats::lm as the underlying statistical model. Future versions of mmibain might strive to include a richer set of statistical models given the flexibility of the 'bain' R package and the flexibility of the Bayesian informative hypothesis testing framework.

BFfe()

As is standard in Mighty Metrika software, the exported functions which represent the central statistical methods get their own Shiny application. The BFfe() function is the Shiny application which can be used to run the BF_for_everyone() function on csv data. To access the application online, go to https://www.mightymetrika.com/tools and click the BFfe button.

A quick tutorial on the BFfe application is coming up in the next few weeks.

scdtb Raw Data Plot Part 2

Mon, 20 May 2024 14:00:03 GMT

Building More Raw Data Plots with scdtb

In scd Raw Data Plot Part 1 we discussed how to use the scdtb shiny app to build some of the single case design plots presented in WWC 5.0 . This blog post will build upon part 1 by going over two plots from Small Sample Size Solutions .

Fictional Single Case Design Efficacy of CBT Example

This example is take from Marija Maric and Vera van der Werff's Single-Case Experimental Designs in Clinical Intervention Research . The data is available as the efficacy_of_CBT dataset in the scdtb R package. View data in R below and then download the csv if you wish to follow along.

To plot this data, open the scdtb app by going to mightymetrika.com > Tools > scdtb. With the app open:

click BROWSE...
Navigate to find efficacy_of_CBT.csv on your computer
Click on the file
Click Open

After completing these steps you should see the following screen:

For this example, we will use Anxious as the outcome of interest. As such, we can plot this data by:

Enter Anxious as the Outcome Variable
Enter time as the Time Variable
Enter phase as the Phase Variable
Click PLOT DATA

After completing these steps you should see the following screen:

Sleeping Pills and Dizziness Example

The next example is from Onghena's One by one: The design and analysis of replicated randomized single-case experiments . You can view this data as the sleeping_pills dataset in scdtb. Download the data below if you wish to follow-along.

As before, you can begin to plot this data by opening the scdtb app and uploading the data.

Open the scdtb app:

Go to mightymetrika.com
Click Tools
Click scdtb

Once the app is open, upload the data by:

clicking BROWSE...
Navigating to find sleeping_pills.csv on your computer
Click on the file
Click Open

After completing these steps you should see the following screen:

Instead of breaking observations up into successive phases as in the previous examples, this data set breaks the observations up into a randomly assigned experimental or control condition. As such, this plot will specify treatment as a Condition Variable.

To plot this data:

Enter sever_compl as the Outcome Variable
Enter day as the Time Variable
Enter treatment as the Condition Variable
Click PLOT DATA

scdtb Raw Data Plot Part 1

Mon, 13 May 2024 14:00:01 GMT

Building Raw Data Plots with scdtb

This blog will show you how to produce single case design raw data plots using the scdtb app. The blog post will focus on single case design plots that can be found in the What Works Clearinghouse Procedures and Standards Handbook, Version 5.0 .

Basic Single-Case Design

The first example is for the single-case design data presented in Figure 14 of the WWC 5.0 :

An approximation of this data is stored in the scdtb R package as the 'basic_scd' dataset.

The write.csv R function was used to create the csv file used in this example. You can download this dataset using the button below.

To get started, got to mightymetrika.com click on Tools then click on scdtb to open the Single Case Design Tool Box app. Next, upload the basic_scd.csv dataset by clicking BROWSE..., navigating to the basic_scd.csv file, clicking on the file, then clicking Open. Once the data finishes uploading, you should see the screen below.

Note that, 'X' is the "row number" variable which is in the dataset due to my using write.csv default settings. We will ignore this variable.

To produce a plot similar to WWC 5.0 Figure 14, simply:

Enter socbehavs as the Outcome Variable
Enter time as the Time Variable
Enter phase as the Phase Variable
Click PLOT DATA

After following these steps, you should see the screen below.

While this plot should allow one to draw the same conclusions as would be drawn from the WWC 5.0 Figure 14, it shows a few improvements which should be made in the next release of the scdtb R package. First, the values of time displayed on the x-axis should be selected more thoughtfully so that users can quickly see which time point each data point corresponds to. Another major difference is that in the scdtb app, a line connects the data points across the phase line. I'm not sure if this presents any problems, but if you would like to see this (or any other aspect of the app) changed in future editions, please submit an issue to the package's github .

Reversal/withdrawal Design

The next example is for the reversal/withdrawal design. This example is based on Figure 17 of WWC 5.0 :

An approximation of this data is stored in the "reversal_withdrawal" dataset of the scdtb R package. View and download this data below.

As in the previous example, we start by opening the scdtb app:

mightymetrika.com
Tools
scdtb.

Once the app is open, upload the reversal_withdrawal.csv:

BROWSE...
navigating to the reversal_withdrawal.csv file
Click on file
Click Open.

If all went well, you should see the screen below.

Plotting this data is very similar to the previous example. You can produce a plot like WWC 5.0 Figure 17 by:

Enter extbehavs as the Outcome Variable
Enter time as the Time Variable
Enter phase as the Phase Variable
Click PLOT DATA

scdtb Release Party

Mon, 06 May 2024 14:09:12 GMT

Single Case Design Toolbox

When I Google "randomized controlled trial gold standard", one of the first resources that pops up is Randomised controlled trials—the gold standard for effectiveness research :

In the world of education research, RCTs are one of the research designs that can attain a rating of Meets WWC Standards Without Reservations . Another research design that can attain this rating is the single case design. Single case designs are " experiments in which one unit is observed repeatedly during a certain period of time under different levels of at least one manipulated variable. " When it is not possible to get enough participants to demonstrate causal effects with a well-powered RCT, researchers might try to see if it is possible to demonstrate causal effects using one (or a few) participant(s) in a single case design.

The 'scdtb' (Single Case Design Toolbox) R package has a collection of functions which can be used to visualize and analyze data from single case designs. The main functions include:

raw_plot(), for visualizing raw data from different types of single case designs
nap(), for computing the nonoverlap of all pairs
mixed_model_analysis(), for running a type of mixed effects model that is useful for single case designs
cross_lagged(), to analyze cross lagged correlations
randomization_test, to compare observed outcome differences to a distribution of differences obtained through permutation

All of these tools in 'scdtb' v0.1.0 were obtained or modified from the What Works Clearing House Procedures and Standards Handbook Version 5.0 or from the n=1 section of Small Sample Size Solutions . The 'scdtb' package also contains datasets that were used as examples in these two resources.

The 'scdtb' package also contains two 'shiny' applications. The first can be accessed through the scdtb() function. It can also be accessed through the mightymetrika website. This function allows the user to upload a csv file with single case design data. Once data is uploaded to the application, the user can run an analysis using the five functions mentioned above (although for a specific design, only a subset of the functions might be applicable). In the coming weeks, there will be additional blog posts outlining how to use the scdtb() 'shiny' app to analyze single case design data using the datasets contained in the 'scdtb' package.

The napjack() function is a card game to help users get a better feel for looking at analysis results from single case designs. The Roxygen2 documentation for the game describes the game as follows:

The game consists of four phases: baseline 1, treatment 1, baseline 2, and treatment 2. In each phase, the player deals six cards and has the option to swap cards within a row once per phase. After all four phases are completed, the game is scored based on the analysis of the dealt cards using non-overlap of all pairs (NAP) and mixed effects modeling.

An upcoming blog post play and explain a few hands of napjack.

Release Party

Fri, 23 Feb 2024 02:47:06 GMT

Mighty Metrika Interface to Cluster Adjusted t Statistics

The goal of the 'mmiCATs' R package is to introduce researchers to cluster adjusted t statistics (CATs), help them better understand when to use them, and to provide a tool which makes the method easy to get started using.

My Introduction to CATs

As you may already know, mightymetrika's mission is centered around advancing methods for statistical methods which improve the analysis of small sample size data. A lot of studies that I work on have a double complication:

Small number of participants
A clustering structure such as repeated measurements within participant

When looking for papers that discuss methods for handling this double complication, I came across Small Samples in Multilevel Modeling by Joop Hox & Daniel McNeish (2020) . I often reference this paper in my work as an applied statistician. In particular, I often find two key parts of the paper extremely helpful. The first part that I often return to is Table 15.1 (see image below) and the discussion surrounding.

The other part of the paper which I find very useful is the discussion of what results from Bayesian models might look like in comparison to the results presented in Table 15.1:

"Table 15.1 does not mention Bayesian estimation because suggestions are highly dependent on specification of the prior. With uninformative priors, Bayesian estimation should work with the sample sizes indicated for ML. Bayesian estimation with weakly informative priors roughly corresponds to the REML column, and Bayesian estimation with strongly informative priors is typically appropriate with lower samples than suggested for REML with the Kenward–Roger correction ( McNeish, 2016b ). "

Originally, I read this paper as an applied statistician looking for more efficient ways to implement mixed effects modeling with smaller sample sizes. However, when I started working on mightymetrika, I came to a point where I wanted to work with a method for smaller sample sizes when the data generating process suggests that the "independent observations" assumption of a statistical model is violated; however, I was not ready to start working on a project focused on prior distributions. So I came back and read the Hox & McNeish (2020) paper again, and this time, here is the passage that stood out for me:

To learn more about this fascinating wild bootstrap method, I turned to the Esarey & Menger (2018) paper which studies four methods side-by-side:

Cluster-robust standard error
Which might be considered the reference method which inspires the search for a replacement
Pairs cluster bootstrapped t-statistics
Wild cluster bootstrapped t-statistics
Which I came to learn about
Cluster-adjusted t-statistics
Which turned out to be the star

One section of the paper provides a relatively simple overview of the CATs method:

When to Use CATs

The conclusion section of Esarey & Menger (2018) had a paragraph which provided a nice outline of when to select which method:

Often, researchers select a random effects model to handle clustered data. However, when the sample size is small, a statistician might try to avoid specifying a model that is too complex for the data by specifying a simpler random effects model than the model considered to be the correct specification. The 'mmiCATs' R package comes with a card game called CloseCATs(). The 'mmiCATs' GitHub README describes the game as follows:

You can also play CloseCATs on the mightymetrika website .

How to Start Using CATs

To start using CATs, I recommend three resources:

Esarey & Menger (2018) for a better understanding of the method
The clusterSEs R package which you can find on CRAN
The MetaCran GitHub repository for clusterSEs where you can browse the code for the method

If you have a csv with clustered data and you would like to get started experimenting with CATs, you can also use the mmiCATs::mmiCATs() shiny application on the mightymetrika website . This app implements the basic functionality of the clusterSEs::clusterIM.glm() function.

Coming Soon

In the coming weeks, mightymetrika.com is planning to release a few blog posts which can help you delve deeper into CATs including:

A tutorial which will use screenshots to walk through a CATs analysis using the mmiCATs::mmiCATs() application
A blog post that will walk through a hand or two of the mmiCATs::CloseCATs() game

In addition to the two main 'mmiCATs' functions (mmiCATs() & CloseCATs()), the 'mmiCATs' package also includes two functions which experiment with the possibility of using CATs with robust regression models in place of stats::glm():

mmiCATs::cluster_im_lmRob()
mmiCATs::cluster_im_glmRob()

These functions can be used with robust models from both ' robust ' and ' robustbase '.

The mmiCATs::pwr_func_lmer() function can be used to streamline the workflow of conducting simulation studies by:

Simulating clustered datasets
Fitting the data sets to the following statistical models:
Linear mixed effects model
Random intercept model
CATs
CATs with truncation
mmiCATs::cluster_im_lmRob() with 'robust'
mmiCATs::cluster_im_lmRob() with 'robustbase'

Then results can be compared to give users a better understanding of each models performance under various data generation schemes.

Handling Factors in Formulas pt 2

Fri, 29 Dec 2023 21:24:35 GMT

In a recent blog post we discussed the process for reading in variables correctly. The gist was this:

If you want your variable treated as a factor (i.e., a categorical variable) then ensure that the values have letters.

This is still good advice. But an ongoing (note: CRAN is closed for the holidays so the updates are taking a while) update to Mighty Metrika tools will have another way to make sure your variables are being handled correctly. This blog will give a basic overview on using this new method. Other blogs posts which will drop within the next few weeks will also feature this new method.

First, let's use mmirestriktor to read in the data_f_grpnum.csv file which gave us issues in the Handling Factors in Formulas .

As in the previous blog post, notice that the grp variable has the type integer when we know that we want type factor. Before, this meant that we would need to refresh the app, ensure that all the values have a letter (i.e., g1, g2, g3 instead of 1, 2, 3), and re-import the data. Now you can fix the issue by:

Double click 'integer' for the grp row
Replace 'integer' by 'factor'
Click in the white space anywhere beneath the table

Now you should see the following:

Now you can continue the analysis with the correct variable types.

I am hoping that this will make the app easier to use. However, the process still needs a few major fixes. For example, look what happens when you accidently enter the type wrong. Instead of "factor" we accidently enter "fctor":

Notice that we get an error in the bottom right hand corner alerting us that we made a mistake. This should help us understand that we did something wrong and that we should not continue without correcting the mistake. However, if we are rushing or absent-minded, then we might accidently continue and get unfortunate results.

Notice that we cannot set up constraints based on the groups we expect to see since our data was not processed as a factor. We can fix this by:

Going back up to the table and editing the type
Click Fit Model again

Now just delete the error flag (I left it there for didactic purposes but remove it whenever you like) and finish the analysis.

The process is not so pretty, but I believe it is better than before. I will make the same update to mmibain as soon as CRAN gets back from vacation. Blog posts with examples of the process will be posted soon too.

Categories of Small

Mon, 18 Dec 2023 21:44:39 GMT

There are a few categories of "small" that come to my mind when I hear the term small sample size study. All of these categories can lead to misleading inferences if they are not handled correctly. Here are the categories that are on my radar.

1) Small Number of Observations

This might mean collecting data on individuals when you are only capable of collecting data on very few individuals. This is the most common situation that comes to mind when I think of a small sample size study. Often times, this situation arises due to budget constraints such as data collection being expensive. It can also arise due to studying a population that is hard to reach or a rare event. In less savory situations, the situation may arise due to poor study planning such as collecting data without conducting a sample size estimation or power analysis first.

When there is a small number of observations, one of the best ways to understand the data is through an indepth description of the data via descriptive statistics and visualizations. Visualizations which show all of the data points for a variable can be particularly helpful.

In order to get adequate statistical power when there is a small number of observations, it may be necessary to spend a lot of time and effort during the planning phase to consider what is known and how this knowledge can be ingested into the statistical modeling process. Statistical methods such as Bayesian statistics with informative priors and informative hypothesis testing (see restriktor & bain for introductions to informative hypothesis testing) can be used to ingest prior knowledge into the model and will result in models with greater statistical power than standard methods.

The obvious downside to "ingesting knowledge" into statistical models is that it can be seen as highly subjective and it can potentially be seen as a form of p-hacking. For example consider the following fictional analysis:

"I ran an ANOVA via anova(stats::lm(formula, data)) and my p-value was 0.09, so I plugged the fitted model into restriktor::iht with a constraint that I obtained by reading my fitted model's summary, and now I've obtained a Type B p-value of 0.90 and a Type A p-value of 0.03, rendering statistically significant support for my hypothesis test." Ficticious P-hacker

In order to conduct a trustworthy study which ingests knowledge in to the model, I would recommend taking the advice given in a bain tutorial and pre-registering the study. This will give you the opportunity to formulate your knowledge in the form of a statistical analysis plan which discusses prior distributions and constraints before data is collected.

2) Small Number of Clusters

It is common to have a nested data generating processs such as polling citizens within district. In this situation, you might have a large amount of citizens but a smaller amount of districts. If the data shows that the nested structure of the data is likely to bias the standard errors, for example when the intraclass correlation is large, then it will be necessary to use a method that adjusts for the lack of independence between observations within cluster.

When the number of clusters is large, cluster-robust standard errors provides a solution that is easy to implement and does not suffer from sensitivity to misspecification. However, when the number of clusters is small, this method tends to have confidence intervals that are too narrow and false positive rates that are too high .

Mixed effects models are a very good option when the number of clusters is small; however, they are sensitive to model specification so this method takes more knowledge and care to implement successfully. In addition, with smaller samples, it may not be possible to specify the correct model without overfitting.

The Cluster adjusted t-statistics (CATs) approach often performs well with a small number of clusters. When it is possible to specify a mixed effects model correctly, the mixed effects model will be more efficient and powerful than CATs, but CATs is a safer option when correctly specifying a mixed effects model presents difficulties.

As of know, I am not sure what methods should be recommended in the case of a small number of clusters and a small number of observations within cluster. For some effect sizes, I have obtained sufficient statistical power using Bayesian random intercept models; however, in many situations, a simple random intercept model may not be the correct specification of the random effects structure.

3) Small Number of Observations Relative to Predictors

Another important category is when the statistical model has a lot of predictors relative to the number of observations. This situation could lead to overfitting if the statistical model is not specified in a way that can handle the complexity. Mighty Metrika may have more apps focused on this issue in 2024; but for now, one method worth learning about in this regard is Bayesian Penalized Regression with Shrinkage Priors .

Approaching CATs

Fri, 15 Dec 2023 21:15:47 GMT

Lately, I've been reading up on statistical methods for small sample sizes when observations are not independent. There seems to be this dilema where:

mixed effects models: efficient but sensitive to violations of assumptions
cluster robust standard errors: robust but do not perform well in smaller samples

In Small Samples in Multilevel Modeling by Hox & McNeish , the authors mention that:

A study by Cameron, Gelbach, and Miller (2008) showed that the “wild bootstrap”, which is similar to the residual bootstrap, was effective with as few as five clusters, which is even lower that the minimal sample size reported in Yung and Chan (1999). Unfortunately, the residuals bootstrap is not implemented in all software (but is available in MLwiN (Rasbash, Steele, Browne, & Goldstein, 2019) and Mplus (Muthén & Muthén, 2017); the wild bootstrap can be carried out in the R package clusterSEs (Esarey & Menger, 2018).

I was intrigued by this WILD BOOTSTRAP and I immediately went to read the Esarey & Menger paper . On first read (I'm still digesting and rereading), I came away with two major takeaways:

The Cluster adjusted t-statistics (CATs) method performs better than the wild bootstrap method
CATs "often produce a small number of outlying cluster coefficient estimates"

Given item 1, and that an easy to use implementation of CATs is available on CRAN , I am extremely exited to start experimenting and learning more about CATs. Given item 2, I wonder how well CATs would perform if stats::glm is swapped out for robust::glmRob/robust::lmRob or the like? I don't know but I am excited to find out.

As I learn more, I will be building some simple 'shiny' applications, games, and simulation functions focused on CATs in a github repo which is titled mmiCATs.

I should mention that the Esarey & Menger paper is written with a political science bent in that the clusters have a lot of observations within.

npboottprmFBar Release Party

Wed, 13 Dec 2023 21:09:35 GMT

The ' npboottprmFBar ' package is now available on CRAN. The name, stands for the nonparametric bootstrap test with pooled resampling method (npbp) via the FBar distribution. The goal is to combine two methods:

The nonparametric bootstrap test with pooled resampling method
Method: Dwivedi et al 2017
R package: npboottprm
Informative hypothesis testing for linear models
Method: Vanbrabant & Rosseel 2020
R package: restriktor

In Dwivedi et al 2007, the npbp is impemented for the independent sample t-test, the paired t-test, and the F-test; on the other hand, the npboottprmFBar package adapts the npbp to work with the Fbar statistic (as implemented in 'restriktor') rather than the usual F statistic used in ANOVA.

Unlike most Mighty Metrika packages released on CRAN, npboottprmFBar does not (yet) have an associated 'shiny' application. This is because I want to get a better understanding of how the method functions. As such, the npboottprmFBar::persimon() function can be used to run simulations which fill out table shells similar to Dwivedi et al 2017 tables S2 and S3 (see snippets below) which focus on Type I error and statistical power. These simulations will be able to compare the performance of the npboottprmFBar method to the methods already present in the aforementioned tables S2 and S3 as well as to the restriktor::iht() default method and restriktor::iht() with boot = "parametric".

Snippets of Table S2 and Table S3 from Dwivedi et al 2017

In the npboottprmFBar "persimon" simulations ( per formance sim ulation on type I error or statistical power), all of the methods associated with informative hypothesis testing will be run under three different constraint scenarios:

0 inequality constraints (grp 1 = grp2 = grp3)
1 inequality constraints (grp1 < grp3)
2 inequality constraints (grp1 < grp2 < grp3)

Another caveat is that for each model in the simulation, "the usual handling of the intercept" is employed. This means that for the methods which do not take constraints, the intercept is included in the model while for the methods which do take constraints, the intercept is not included in the model. While this distinction will make it harder to understand why a particular method is performing better or worse than a method which handles the intercept differently, the results may be more useful in that the results reflect how one would actually employ each of the models in practice.

Handling Factors in Formulas

Mon, 11 Dec 2023 21:02:02 GMT

Currently, two Tools on mightymetrika.com handle factors (we can also call factors categorical variables) in a way which is likely to cause confusion

Keep in mind that this post pertains to using formulas within the 'shiny' web-applications. When using 'mmirestriktor', 'mmibain', 'restriktor', or 'bain', users can handle factors in the regular R fashion before fitting statistical models.

The Factor Issue

Consider these two csv files:

When processing the csv on the left (data_f_grpnum) the grp variable will typically get treated as a numeric variable. On the other hand, when working with the csv on the right (data_f_grpfac) the grp variable will get treated as a factor.

When uploading a csv file for mightymetrika.com 'shiny' applications, you should make sure that variables that you want treated as factors cannot be automatically parsed as numeric. Put another way, make sure your factor variables include letters in their values.

For the csv files above, we force our grp variable to behave as a factor by prefixing the values with "Group". Any letter would achieve the same goal.

What Happens When We Read a Factor as Numeric?

Let's see what happens when we accidently try to run the mmirestriktor app on the data_f_grpnum.csv file when we want the grp variable to function as a categorical variable (i.e., a factor).

In the example below, we start by:

Read in the data_f_grpnum.csv file
Notice that the available variables are 'x' & 'grp'
Set up the formula x ~ -1 + grp

Up to this point, everything looks as expected. However, after we fit the model we see that the Available Terms for Constraint only shows the variable 'grp'. However, since we wanted the grp variable to at as a categorical variable, we wanted the available terms to show us something that corresponds to the groups. Since the "terms" (i.e., grp) are not consistent with the goal (comparing groups or another goal that would inspire use to use a categorical variable), we know that something has gone horribly wrong.

What Happens When We Read a Factor as Factor?

Now since we know we want the grp variable to function as a categorical variable, we run the mmirestriktor app on the data_f_grpfac.csv and we get the following Available Terms for Constraint.

Now we can use the information in the Available Terms for Constraint to specify a constraint which compares the groups in our 'grp' variable and then we can run an Informative Hypothesis Test analysis.

Informative Hypothesis Testing

Fri, 08 Dec 2023 21:14:38 GMT

In its essence, informative hypothesis testing (IHT) is a familiar concept to most who are familiar with the t-test. A two-tailed t-test can be used to test:

Ho: sample mean = 10

Ha: sample mean ≠ 10

If the test is conducted at the 0.05 alpha level, then we will reject the null hypothesis if:

The mean is in the lower 2.5% of the distribution
The mean is in the upper 2.5% of the distribution

The two-tailed test is simultaneously testing if if the mean is significantly greater the 10 and if the mean is significantly less than 10. On the other hand, a one-tailed t-test can be used to test (we choose mean > 10 for this example but we could also work an example with mean < 10):

Ho: sample mean = 10

Ha: sample mean > 10

If the test is conducted at the 0.05 alpha level, then we will reject the null hypothesis if:

The mean is in the upper 5% of the distribution

The one-tailed test which focuses on the upper tail of the distribution will miss a significant effect associated with the lower tail of the distribution. The one-tailed test gains statistical power at the expense of potentially missing a significant effect in the untested tail of the distribution.

A one-tailed t-test is an example of an IHT.

When is IHT Appropriate?

What I fear is a research project that does not conduct a proper power analysis before data collection and then, upon realizing that the sample size is less than ideal, a decision is made to use a one-tailed t-test rather than a two tailed t-test. With this fear in mind, I like the stats.oarc.ucla advice:

"If you consider the consequences of missing an effect in the untested direction and conclude that they are negligible and in no way irresponsible or unethical, then you can proceed with a one-tailed test."

Let's also consider the issue from a more optimistic perspective. As a working statistician, I often work with researchers, clinicians, and professionals who have a lot of experience and intuition when it comes to reasoning about the qualitative properties of the variables' distributions, correlations between pairs of variables, and how variables change over time under a condition of interest. With this experience and intuition in mind, I like what Vanbrabant and Rosseel (2020) explain:

"In many psychological fields, researchers have specific expectations about the relation between the means of different groups or between (standardized) regression coefficients...In observational studies, researchers often have clear ideas about whether the direction of the effects is positive or negative (see, for example, Richardson, Abraham, & Bond, 2012), indicated by symbols like “<” and “>”. Testing such specific expectations directly is known under various names, such as one-sided testing, order-constrained hypothesis testing, constrained statistical inference, and informative hypothesis testing."

Personally, if I am to use IHT, I would want to pre-register the study so that I can specify that I am using IHT and specify the "constraints" (see below for more on constraints) data collection begins. This would allow the sample size estimation to be informed by the IHT.

Extending IHT

The concept of IHT can be extended to linear regression, ANOVA, generalized linear models, robust regression, mixed effects models, structural equation models and more. In the R programming language, software is available for implementing IHT in both the frequentist and the Bayesian frameworks of statistics. I personally use the following in each each domain:

Frequentist: restriktor
Bayesian: bain

Click on one of the above links to learn more about implementing IHT in R. Both links have examples, tutorials, and literature that will get you up and running with IHT. While there are important differences between the software packages, both of them use a similar workflow:

Fit a statistical model in the regular fashion
Specify a constraint (i.e., "Group1 > Group2")
Pass the fitted model and the constraints to a specialized IHT function
Interpret the results

FBAR Cards Insight

Wed, 06 Dec 2023 21:22:17 GMT

Fbar Cards is a card game designed to help researchers gain initial insights on informative hypothesis testing for ANOVA models as implemented in the 'restriktor' R package. The game goes like this:

Set the "difficulty level" n
Click Start/Restart Game to deal an n x n grid of cards from a standard 52 card deck
For each row you can either:
Leave the cards as is
Swap two cards within the row
ONLY 1 swap is allowed per row
Click Score Game

In this game, the columns represent groups for an ANOVA analysis using an informative hypothesis testing framework where:

'col1 < col2 < col3'

is the constraint. As such, the user is expected to use their moves in each row to try to order the cards in increasing order.

Here is a sample hand of the game on level n = 3. After I click Start/Restart Game, I have the following grid:

With this hand I made the following moves:

Row 1: I did nothing in row 1. This was a mistake on my part; I forgot that hearts is defined as a higher value than diamonds. See the standard_deck function of 'mmcards' to see how the deck was defined (clubs < diamonds < hearts < spades).
Row 2: Swapped 10 and 2
Row 3: Swapped Ace and 4

This was the final hand:

With this hand, I then clicked Score Game and obtained:

At this point, we can get into the insight.

Insight: Focus on the Edges

Since I'm only allowed one swap per row, when I play at higher difficulty levels, my main (and often only) focus is on making sure the card in column n has the largest value in the row. I've found that this strategy is surprisingly efficient.

In the paper Constrained statistical inference: sample-size tables for ANOVA and regression I came across a small passage which helps me understand what is going on here:

"In particular we would like to know about the power when the means are not perfectly in line with the ordered hypothesis

...

the power for Hypothesis test Type A (HA0 vs. HA1) is largely dominated by the extremes (here the first and last mean). This means that, irrespective of the deviations of the two middle means, the power is almost not affected."

Keep in mind that to "Win" we need to fail to reject the null hypothesis on the Type B test and reject the null hypothesis on the Type A test.

Quick mmibain Tutorial

Mon, 04 Dec 2023 20:59:35 GMT

bain is my primary tool for Ba yesian in formative hypothesis testing. The 'mmibain' R package contains web applications (along with some helper functions) built on top of the 'bain' software. This tutorial walks through a quick example where we'll use the mmibain() 'shiny' application to fit a linear model to the same data set used in the quick mmirestriktor tutorial . Once again, we will work with the data_f.csv file which you can download from the npboottprm GitHub .

Step 1) Open the Application

To open the mmibain() 'shiny' application, go to https://www.mightymetrika.com/mmibain and click on the mmibain button. This should open up the following application:

Step 2) Upload Data

The data_f.csv file looks like this:

Step 3) Set Up Analysis

In the Upload CSV File section of the application, click Browse to find this data set on your computer and read the data into the app.

Once your upload is complete, you should see the following on your screen:

In what follows we will use the 'lm' engine to fit a model using the stats::lm function from R. Specify the Formula as x ~ -1 + grp (using -1 to remove the intercept so that none of our grps are used as a reference level). The Additional Arguments field can be used to pass extra arguments to the model fitting engine (stats::lm in this example but t_test or lavaan if a different engine is selected). In this example, we will not pass extra arguments to stats::lm. After specifying model details, click the Fit Model button to get the following screen.

Step 4) Specify Constraint

Using the terms available for constraint, specify the constraint. A great non-technical overview of using the Bayes Factor in the context of 'bain' is A tutorial on testing hypotheses using the Bayes factor ; this tutorial will give you a foundation on how to specify constraints and how to use and interpret the Bayes factor. Another great resource is Introduction to bain vignette from the R package which includes tens of examples on how to construct constraints and how to use 'bain' in the context of the t-test, ANOVA, linear regression, and structural equation modeling.

For our example we will use the constraint: grpGroup1 < grpGroup2 < grpGroup3.

We will use the default Fraction of 1, we will not standardize our independent variables since the values for each group use the same units of measurements, we will leave the confidence level at 0.95, and we'll set the seed for random number generation to 100. After entering these settings click Run Analysis to obtain the Bayesian informative hypotheses evaluation results.

In this example, H1 is the hypothesis, "grpGroup1 < grpGroup2 < grpGroup3". The "Analysis of the Monin Data Using Informative Hypotheses paper, " section of the paper A Tutorial on Testing Hypotheses Using the Bayes Factor mentions that:

If a hypothesis is specified only using inequality constraints (that is, smaller than and larger than), the column labeled BF.c contains the Bayes factor of the hypothesis at hand versus its complement Hc, that is, not the inequality constrained hypothesis at hand.

In our example, the only hypothesis, H1, is specified only using inequality constraints. As such, BF.c gives us the Bayes factor of our H1 versus its complement. Here, we have evidence a Bayes factor of 2475.8 which is evidence of overwhelming support for H1.

To learn more about 'bain', you are encouraged to read the paper 'A tutorial on testing hypotheses using the Bayes factor'. Two seperate links to the abstract have been provided above; however, as of the writing of this post, you can also find the paper on ResearchGate: https://www.researchgate.net/publication/331029513_A_Tutorial_on_Testing_Hypotheses_Using_the_Bayes_Factor

The paper delves into important issues that were only glossed over in this blog post including:

Defining constraints
Setting the priors (i.e., Fraction)
Interpreting Bayes factors
Interpreting Bayesian error probabilities
Interpreting results when using both equality and inequality constraints
Using multiple hypothesis in one analysis
Bayesian updating
Analysis of replication studies

The paper is a rich tutorial that opens the doors to what is, for most of us, a whole new paradigm for statistical inference.

Quick mmirestriktor Tutorial

Sun, 15 Oct 2023 16:33:41 GMT

The stats::lm engine.

This tutorial should help you get started with the mmirestriktor Shiny application. mmirestriktor is the Mighty Metrika Interface to restriktor. restriktor is an R package which has tools for working with informative hypothesis testing. To learn more about restriktor and informative hypothesis testing, visit https://restriktor.org/.

The mmirestriktor app can be used to import your csv data and then implement the iht() and restriktor() functions from the restriktor package. Please note that the current version of mmirestriktor only uses these functions with default settings and mmirestriktor does not currently support the lavaan SEM functionality; as such, users will need to access iht() and restriktor() using the restriktor package in an R session to gain the full power and functionality of informative hypothesis testing.

Here, we will work with the data_f.csv file which you can download from the npboottprm GitHub .

Step 1) Open the Application

To open the mmirestriktor Shiny application, go to https://www.mightymetrika.com/mmirestriktor and click on the mmirestriktor button. This should open up the following application:

Step 2) Upload Data

The data for this tutorial looks like this:

Please note the row names in column A and pay attention to how this shows up in the application. In the Upload CSV File section of the application, click Browse to find this data set on your computer and read the data into the app.

Once your upload is complete, you should see the following on your screen:

Notice that the row names show up as the column "X"; this is due to the way that I originally exported the R dataframe to csv. The other two variables are the "x" and "grp" variables from the data_f.csv file; these two variables can be used in our model formula.

Step 3) Model Specification

In what follows we will use the 'lm' engine to fit a model using the stats::lm function from R. Specify the Formula as x ~ -1 + grp (using -1 to remove the intercept so that none of our grps are used as a reference level). The Extra arguments field can be used to pass extra arguments to the model fitting engine (stats::lm in this example but stats::glm or MASS::rlm if a different engine is selected). In this example, we will not pass extra arguments to stats::lm. After specifying model details, click the Fit Model button to get the following screen.

Step 4) Specify Constraint

Using the terms available for constraint, specify the constraint. Visit the https://restriktor.org/index.html homepage for a basic overview on constraints and then proceed to https://restriktor.org/tutorial/syntax.html for further discussion.

Please notice that when defining constraints in R, we wrap the constraint in '' (i.e., single quotes) whereas in the mmirestriktor Shiny app, we do not wrap the constraint in single quotes.

For our example we will use the constraint: grpGroup1 < grpGroup2 < grpGroup3.

Step 5) Run Analysis

We will use the default 0.05 alpha level and select both analysis types. After specifying the constraint, selecting an alpha level and analysis type, click Run Analysis to get results:

This blog post covered the basics of using the mmirestriktor app with the lm engine. A future blog post will cover the glm engine which will incorporate the Extra arguments field to set the stats::glm family parameter. There should also be a future blog post with a continuous predictor in order to discuss how the mmirestriktor app standardizes numeric predictors before fitting the model.

Boot War

Fri, 25 Aug 2023 14:09:43 GMT

Intuitive Learning Through Card Play

Boot War is a card game designed to enhance one's understanding of the nonparametric bootstrap test with pooled resampling methods . By diving deep into various game settings, players can derive rich insights. Let's break down the basic gameplay:

1. Select a Mode

Choose 't' for the independent t-test or 'pt' for the paired t-test. In the independent t-test, the effect size is derived from the mean difference between the player's cards and the computer's cards. For the paired t-test, the effect size is calculated as the mean of round-by-round differences between the player and the computer.

2. Define the Deck

Default: A standard 52 card deck with ranked suits. For a twist, employ the R anonymous function to craft an " anonymous deck ". The game even supports a bring-your-own-deck feature, known as an " interleaved deck " in Boot War, where different decks are set for the player and the computer.

3. Select a Confidence Level

Set your desired confidence level for the test statistic and effect size. While the default is set at 0.95 (for 95% confidence intervals), it's a crucial component to experiment with.

4. Choose the Number of Bootstrap Resamples

Decide on the count of bootstraps for analysis, integral to the nonparametric bootstrap test with pooled resampling.

5. Select the Number of Rounds

Opt for any round count between 1 to 26. However, a word of caution: selecting 1 to 3 rounds may lead to crashes. My personal recommendation lies between 5 to 12 rounds, especially if you're interested in observing outcomes in smaller sample sizes.

6. Set a Seed

Consistency is key. By setting a similar random number seed, expect consistent outcomes for identical inputs.

7. Play out the Rounds

Simply hit the 'Deal Card' button to progress through the rounds.

8. Score the Game

When you finish playing out the rounds, the game will employ the nonparametric bootstrap test with pooled resampling to compute the final score, which includes effect size and confidence interval, and to determine the winner.

Final Thoughts

For an enriching experience, I advise players to experiment with an anonymous deck (e.g., function(x) { rpois(20, 15) }) or an interleaved deck (e.g., function(x) { list(rpois(10, 15), rpois(10,10)) }). Once you've set your deck, play around with the mode, confidence level, number of bootstrap rounds, and total rounds to glean insights into the chosen distribution's performance. You might also find it enlightening to think about the side-by-side comparison of the bootstrap p-value and Welch's p-value as you play with the settings.

A tip for enthusiasts: Repetitively playing the game with consistent settings (except for the seed) mimics a manual simulation study, which can offer profound insights if you set the deck up to sample from a distribution that is meaningful to you.

Release Party: npboottprm 0.1.0

Wed, 16 Aug 2023 15:36:41 GMT

Nonparametric Bootstrap Test with Pooled Resampling

Beta Release

The npboottprm 0.1.0 package implements nonparametric bootstrap tests with pooled resampling methods (NBPR), as detailed in Dwivedi, Mallawaarachchi, and Alvarado (2017). This release is a beta version of the software. The Shiny web app, its instruction manual, the open-source code base, and the CRAN package are accessible at https://www.mightymetrika.com/tools/npboottprm

When is npboottprm 0.1.0 Applicable in Research?

This package includes NBPR implementations for the independent t-test, paired t-test, and ANOVA F-test. Therefore, if your study aligns with any of these tests (or their alternatives like the Welch t-test, permutation t-test, etc.), you might find the npboottprm package useful.

A guiding principle is extracted from the conclusion of the Dwivedi et al. (2017) abstract: "We suggest using the nonparametric bootstrap test with pooled resampling method for comparing paired or unpaired means and for validating the one-way analysis of variance test results for non-normal data in small sample size studies."

Simulation Studies and Results

For a more in-depth assessment, researchers can turn to the simulation results highlighted in Dwivedi et al. (2017) or researchers can initiate a simulation comparing NBPR to standard methods on their own. While npboottprm can be combined with base R or the tidyverse to conduct simulation studies, this beta software version does not currently offer functions to simplify the process. Nevertheless, the simulation findings from Dwivedi et al. (2017) are illuminating. Tables II (focusing on type I error) and III (statistical power) present comparisons between the NBPR for the independent t-test and its traditional counterparts (i.e., Student’s t-test, Welch t-test, exact Wilcoxon rank sum test, permutation t-test) for sample sizes ranging from 3 to 15 across varying conditions where the distribution of the data and the variance between the two groups are: normal and equal variance; normal and unequal variance; same skewed and equal variance; same skewed and unequal variance; different skewed and equal variance; different skewed and unequal variance; unequal sample size, same skewed, and equal variance; Unequal sample size, same skewed, and unequal variance. In these tables, one of the NBPR's traditional counterparts tends to outperm the NBPR method.

Table IV delves into the comparison between NBPR and the traditional counterparts in terms of statistical power when dealing with non-normal data distributions such as log-normal, Poisson, Chi-square, and Cauchy. In this table, the advantages of using NBPR become increasingly clear.

The subsequent sections of the paper elaborate on similar simulation findings for the paired t-test.

The simulations presented in these tables emphasize the conclusion previously drawn from the abstract which endorses the NBPR for comparing means (paired or unpaired) and for validating ANOVA results in small sample studies with non-normal data.

Reference

Dwivedi AK, Mallawaarachchi I, Alvarado LA (2017). “Analysis of small sample size studies using nonparametric bootstrap test with pooled resampling method.” Statistics in Medicine, 36 (14), 2187-2205.

mightymetrika-ccvfo

Who Said It Best Episode 1: Daniel McNeish

Resources for Building R Shiny x PostgreSQL x AWS EC2 Apps

Main R Workflow

Swapping Out MySQL for PostgreSQL

Swapping Out DBI for pool

Open for Communication

.Renviron File

Next Steps

BFfe: Test Update

The Loblolly Dataset

BFfe Tutorial

Step 1: Open the BFfe App

Step 2: Upload Loblolly Data

Step 3: Set-up the Run

Step 4: Interpret Results

mmibain 0.2.0 Release Party

BF_for_everyone()

BFfe()

scdtb Raw Data Plot Part 2

Building More Raw Data Plots with scdtb

Fictional Single Case Design Efficacy of CBT Example

Sleeping Pills and Dizziness Example

scdtb Raw Data Plot Part 1

Building Raw Data Plots with scdtb

Basic Single-Case Design

Reversal/withdrawal Design

scdtb Release Party

Single Case Design Toolbox

Release Party

Mighty Metrika Interface to Cluster Adjusted t Statistics

My Introduction to CATs

When to Use CATs ﻿

How to Start Using CATs

Coming Soon

Handling Factors in Formulas pt 2

Categories of Small

1) Small Number of Observations

2) Small Number of Clusters

3) Small Number of Observations Relative to Predictors

Approaching CATs

npboottprmFBar Release Party

Snippets of Table S2 and Table S3 from Dwivedi et al 2017

Handling Factors in Formulas

The Factor Issue

What Happens When We Read a Factor as Numeric?

What Happens When We Read a Factor as Factor?

Informative Hypothesis Testing

When is IHT Appropriate?

Extending IHT

FBAR Cards Insight

Insight: Focus on the Edges

Quick mmibain Tutorial

Step 1) Open the Application

Step 2) Upload Data

Step 3) Set Up Analysis

Step 4) Specify Constraint

Quick mmirestriktor Tutorial

The stats::lm engine.

Step 1) Open the Application

Step 2) Upload Data

Step 3) Model Specification

Step 4) Specify Constraint

Step 5) Run Analysis

Boot War

Intuitive Learning Through Card Play

1. Select a Mode

2. Define the Deck

3. Select a Confidence Level

4. Choose the Number of Bootstrap Resamples

5. Select the Number of Rounds

6. Set a Seed

7. Play out the Rounds

8. Score the Game

Final Thoughts

Release Party: npboottprm 0.1.0

Nonparametric Bootstrap Test with Pooled Resampling

Beta Release

When is npboottprm 0.1.0 Applicable in Research?

Simulation Studies and Results

When to Use CATs