seidelj / mht Goto Github PK

Multiple Hypothesis Testing Procedure - List, Shaikh, Xu 2015

Stata 37.82% MATLAB 62.18%

mht's Introduction

Multiple Hypothesis Testing

Stata code for the procedure detailed in List, Shaikh, and Xu (2015) link

The software is made available as is, and no warranty - about the software, its performance or its conformity to any specification - is given or implied.

Stata 14 Users

You can install mhtexp using Stata's ssc command.

ssc install mhtexp

Once the command finishes, you should be able to use the command. A data set, data.csv, and an example do file, mhtexp_examples.do are available in this repository for download if you want to run an example.

Manual Stata Install

Download or clone this repository and make sure your current directory contains

mhtexp.ado -- this initializes the stata comand "mhtexp" for usage from the command line or .do file
lmhtexp.mlib -- the required mata functions that perform the computation
mhtexp.sthlp -- OPTIONAL but recommended. Usage: from stata command line: help mhtexp

If it is your first time running the code, ensure that Stata knows to look in lmhtexp.mlib. From the Stata command line enter

mata: mata mlib index

See mhtexp_examples.do for usage example OR from stata terminal type help mhtexp

Summary of contents

mhtexp_examples.do do file with examples using the included data set.
mhtexp.sthlp Stata14 help file
mhtexp.ado Contains the Stata command definition of mhtexp
data.csv contains the data set used in the List, Shaikh, and Xu (2015)
lmhtexp.mlib contains mata function required for the command; compiled using Stata14
stata11/lmhtexp11.mlib contains the mata functions required for the command; compiled using Stata11.

For older versions of Stata (<Stata14 and =>Stata11)

Stata versions that are atleast Stata11 can still use this command. However, the bootstrap option is currently unavailable for older versions of Stata. To use this command with an older version of Stata, first replace lmhtexp.mlib with lmhtexp11.mlib and remove or comment out line 2 in mhtexp.ado.

### Remove ###
version 14

The key difference in these two files (outside of how they are compiled), is the way in which ids are selected for the bootstrap sample.

In both cases the same random number generater is used to select random variates over [a, b]. While Stata14 has a built in method, Stata11 does not.

floor( (b-a+1) * runiform() + a)  // in lmhtexp11.mlib (Stata11)
runiformint(r, c, a, b) // in lmhtexp.mlib (Stata14)

The two methods both produce the desired result, but the matrix of IDs is slightly different accross these two methods. Therefore, the bootstrapped statistics used to generate the outputted p-values will not be identical to the results presented in List, Shaikh, Xu 2015.

contact: [email protected]

mht's People

Contributors

Stargazers

Watchers

Forkers

dkillian anastasia1302 luoxiaofan666 ikeokoli tommasoreggiani nosequeldeebee blainewong0321

mht's Issues

add option to change size of bootstrap sample

Currently, its 3,000. However, with more hypothesises being testing one might need to increase the size of the bootstrapped sample-- seems to be conjecture at this point.

I expect hardware limitations to reveal themselves with increasing the sample size, though still something I'm looking into.

Stay tuned.

User reported problems with old version

I have Stata 13.1. However, even if I use the _v11.mlib mata, when I try to run the command I get a message saying that my version of Stata cannot run version 14 programs.
I tried that in 2 different computers.

store results to variables

Currently, the command only outputs to display. Update function to return values.

Versioning

The .mlib file is compiled using Stata14 and therefore does not work on older version of Stata

Missing Error Message

The mhtexp command does not recognize conditional statements. It interprets "mhtexp x if y== z" as the same command as "mhtexp x". It does not throw an error if a conditional is included which makes it very easy for unfamiliar users to make a mistake.

Specifying baseline controls/block fixed effects

I am not sure there is a way to specify baseline controls or block fixed effects right now? Maybe this could be added in in a future update.

comment examples

Specify the scenarios that each example addresses in the .do file

Stata vs. Matlab Code

I have a question regarding the Stata vs. the Matlab code. The two different programs seem to give different results for me. I am using your example code and example data and the same variables for both commands (the difference in means is the same). Discrepancies also seem to persist when using a larger number of bootstrap iterations in both programs (e.g. 10,000).

To generate reproducible code, please see below for the code I ran using your first example:

Here is the code I ran in Stata (from mhtexp_examples.do):

clear all
insheet using data.csv, comma names
//Creating outcome variable
gen amountmat = amount * (1+ratio)
gen groupid = (redcty==1 & red0 == 1) + (redcty==0 & red0 == 1)*2 + (redcty==0 & red0 == 0)*3 + (redcty==1 & red0 == 0)*4
replace groupid = . if groupid == 0

mata: mata mlib index

// help mhtexp

//Example 1: Hypothesis testing with multiple outcomes: 
//  We consider four outcome variables: response rate, dollars given not including
//  match, dollars given including match, and amount change.
mhtexp gave amount amountmat amountchange, treatment(treatment) 
// results are the same when explicitly using bootstrap(3000) as it is the default
mhtexp gave amount amountmat amountchange, treatment(treatment) bootstrap(3000)

This is the code I ran in Matlab (from mhtexp_examples.m):

data = importdata('data.csv');
data = data.data;               % read the dataset
B=3000;                         % the number of simulated samples


%% Hypothesis testing with multiple outcomes: 
% We consider four outcome variables: response rate, dollars given not including
% match, dollars given including match, and amount change.
amountmat = data(:,1).*(data(:,10)+ones(size(data,1),1));             % dollars raised per letter including match
Y = [data(:,[12,1]) amountmat data(:,35)];     % the matrix of outcomes
D = data(:,8);                                 % the vector of treatment statuses
sub = ones(size(D,1),1);                       % the subgroup ID's
numoc = size(Y,2);                             % the number of outcomes
numsub = size(unique(sub),1);                  % the number of subgroups
numg = size(unique(D),1)-1;                    % the number of treatment groups (not including the control group)
combo = [zeros(numg,1) (1:numg)'];             % We compare each treatment to the control.
numpc =size(combo,1);                          % the number of pairs of treatment (control) groups of interest
select = ones(numoc,numsub,numpc);             % We are interested in all the numoc*numsub*numpc hypotheses.
[example1] = mhtexp(B,Y,sub,D,combo,select)

I have also attached screenshots of the results I am getting. Could you please clarify?

Thank you!

Stata Output:

Matlab Output: