after modification it works well upto page 3 but I had another bug on prediction/forecasts due to depreciation of ix and I modified .ix to .loc but the plot still looks no lower and upper bound.

Third bug happened when I tried Metropolis-Hastings-ARIMA (1,1) and for some reason I had problem with installation with pymc nonetheless of my successful records on installation with pip.

However, Gibbs codes run well.

My last bug was found at the last box titled "Construct the predictions / forecasts" This is same as my second bug and I had no lower and upper bound line on my plot and followings are error message for your information.

Happy New Year and thanks for your excellent implementation on MCMC and TSA.

KeyError Traceback (most recent call last)
C:\Anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
3062 try:
-> 3063 return self._engine.get_loc(key)
3064 except KeyError:

pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'lower CPIAUCNS'

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last)
in ()
15 alpha=0.15)
16 lines, = ax.plot(forecast.index, forecast)
---> 17 ax.fill_between(forecast.index, ci['lower CPIAUCNS'], ci['upper CPIAUCNS'],
18 alpha=0.2)
19

C:\Anaconda3\lib\site-packages\pandas\core\frame.py in getitem(self, key)
2683 return self._getitem_multilevel(key)
2684 else:
-> 2685 return self._getitem_column(key)
2686
2687 def _getitem_column(self, key):

C:\Anaconda3\lib\site-packages\pandas\core\frame.py in _getitem_column(self, key)
2690 # get column
2691 if self.columns.is_unique:
-> 2692 return self._get_item_cache(key)
2693
2694 # duplicate columns & possible reduce dimensionality

C:\Anaconda3\lib\site-packages\pandas\core\generic.py in _get_item_cache(self, item)
2484 res = cache.get(item)
2485 if res is None:
-> 2486 values = self._data.get(item)
2487 res = self._box_item_values(item, values)
2488 cache[item] = res

C:\Anaconda3\lib\site-packages\pandas\core\internals.py in get(self, item, fastpath)
4113
4114 if not isna(item):
-> 4115 loc = self.items.get_loc(item)
4116 else:
4117 indexer = np.arange(len(self.items))[isna(self.items)]

C:\Anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
3063 return self._engine.get_loc(key)
3064 except KeyError:
-> 3065 return self._engine.get_loc(self._maybe_cast_indexer(key))
3066
3067 indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'lower CPIAUCNS'

so useful work ! Is this any way that I can use yearly data in our tool?

For this notebook: https://github.com/ChadFulton/tsa-notebooks/blob/master/estimating_rbc.ipynb

Can you provide any details on how the values
disutility_labor=3, depreciation_rate=0.025, capital_share=0.36
were calculated?

I got this error:
_http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)

HTTPError: Forbidden_

And if I try to get direct acces from google chrome:

This XML file does not appear to have any style information associated with it. The document tree is shown below.

AccessDenied
Access Denied
MF157NTQTG6RVEC4
yc9cxlUb/cf8pWldoeN0En85G2+IYGQPRv2vx4RSicMSnYp3QwM8g1ox1XI+1N6eMDyPmcQB+eA=

I suggest saving the xrates.xls file to a CSV file xrates.csv so that it is easier for a Python program to read. When I tried to run the stochastic volatility notebook stochastic_volatility_mcmc.ipynb I got an error

ValueError: Excel file format cannot be determined, you must specify an engine manually.

and resolved it by replacing

ex = pd.read_excel(r'data\xrates.xls')

with

ex = pd.read_csv(r"data\xrates.csv")

Thanks for the project.

Hi Chad,
Thank you so much for your work on statsmodels and DynamicFactorMQ, it's an extremely useful tool.
I only have one question: when I try to replicate your results for nowcasting US GDP, whenever there is a new monthly observation set occurring the same month of the quarterly GDP release the nowcast does not update.
For example: new set of monthly information for June 2020, the GDP nowcast for 2020 Q2 is the same as the one with data up to May 2020. Can I ask you why is that?
Many thanks.

From @ckorkpoe (migrated to issues from a commit comment):

Chad,
I am trying to run your example on the stochastic volatility for univariate models using MCMC but I have problems with the import src as sv. I have been roaming the internet but so far have drawn a blank. It seems the src is a folder. Unfortunately, I can't really resolve the issue. I am doing pip install src and the response I get is
Running setup.py clean for src
Failed to build src
Installing collected packages: src
Running setup.py install for src ... error

What am I missing here? Thanks.

Dear Chad, thanks a lot for the excellent work of implementing these codes in Python. Regarding your example given in this notebook, you compute the quarterly forecast / nowcast from its unobserved monthly counterpart using

# Resample to quarterly frequency by taking the value in the last
# month of each quarter
point_forecasts_q = point_forecasts_m.resample('Q').last()

However, this does not correspond to the internal modelling a la Mariano and Murwasa (2003) or Banbura and Modugno (2014), where the quarterly series growth rate is a weighted average of the unobserved monthly series. I summarized this in this note here:

So, if you follow Banbura and Modugno (2014) as I presume, should not the quarterly forecast be obtained as:

def wma_BM(x):
    return x + 2*x.shift(1) + 3*x.shift(2) + 2*x.shift(3) + x.shift(4)

wma_BM(gdp_forecast)

Final comment on the notebook: the FRED databases are no longer on amazon AWS, the base URL should be changed to:

base_url = 'https://files.stlouisfed.org/files/htdocs/fred-md'

Dear Chad,

Thank you for implementing setar_model and tsa_setar_sunspots_full.ipynb. There are some modules and methods start with "_" show missing.
Line 260 of setar_model.py: beta = self._fit() shows cannot find "_fit()"
Line 12 of tsa\statespace\tools.py: from . import _statespace shows cannot find "_statespace"
Is there anything I missed? Thank you.

Hi @ChadFulton,

Can you kindly help me identify how .predict() results are calculated for sm.tsa.SARIMAX with an MA term?

Suppose the model is of order (1,0,1) with two exogenous variables. I have tried replicating the .predict() results via following equation:
y(t) = coef1 * X1 + coef2 * X2 + ar.L1coef * y(t-1) + ma.L1coef * residual(t-1)+ e(t)
but looks like this is not the right perspective.

I was able to replicate the results for an order with AR terms only via following equation:
y(t) = ar.L1coef( [y(t-1)- (coef1X1(t-1) + coef2X2(t-1)) ] ) + [y(t) - (coef1X1(t) + coef2X2(t)) ]

but I am stuck when it comes to MA.

Can you help me, please? Would really appreciate your help.

Thank you very much in advance. :)

Dear Chad
Thank you for the excellent tutorials and as a novice on python I am learning Python with MCMC now. On your script titled "Implementing and estimating an ARMA(1, 1) state space model", I found some bugs and first one is yahoo data feed and I fixed the one with st. louis's own api, using my persoanl api key given by the Fed.

from fredapi import Fred
fred = Fred(api_key='...')
cpi = fred.get_series('CPIAUCNS',"1971-01","2016-12")
cpi.tail()

It works well now and let me send you other bugs one by one with separate letter

Hi Dr.Fulton,

I have been working on how to do simulation of RBC model in python and your notebook (simple_rbc.ipynb) is greatly helpful. But when I tried to run the codes in IPython, some errors came out.

First one is, AttributeError: 'odict_keys' object has no attribute 'index' in
self.idx_cap_share = parameters.index('capital_share').
I changed that to self.idx_cap_share = list(parameters).index('capital_share').
And it's fixed.

However, another error emerged: IndexError: too many indices for array in
structural_params = np.array(self.parameters.values())[self.idx_estimated] ,
start_params = self.start_params
and
calibrated_res = calibrated_mod.fit(method='nm', maxiter=1000, disp=0).
I have no clue how to fix this one.

Sorry if I don't make myself clear. I just start to learning python and plan to use it in my academic research. I googled a lot about this error but nothing works. It would be a huge help if you could help me with this problem. If there's any other information you may need, please let me know.

Thanks,
Tengyu

Hi ChadFulton,

This notebook is very useful. Using the Hamilton data and your code I could have the same result as you showed. However when I used current available real GNP or GDP data and took their log difference (quarterly) as input, the model doesn't give satisfactory results. I plotted the example used rgnp data and the new rgnp calculated from current available real GNP. They are quite close but slightly different.

Can you enlighten me why it is the case? I believe you know about the code much better then I do.

Thanks very much.

Irene

Hi Chad,

I recently came across your "Large dynamic factor models, forecasting, and nowcasting" posting and I found it very informative. When trying to replicate the code I received this HTTP error (in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden) when trying to implement the following code:

Load the vintages of data from FRED

dta = {date: load_fredmd_data(date)
for date in ['2020-02', '2020-03', '2020-04', '2020-05', '2020-06']}
I initially thought that the base URL needed to be updated from the time of your posting base_url = 'https://files.fred.stlouisfed.org/files/htdocs/fred-md' to base_url = 'https://s3.amazonaws.com/files.fred.stlouisfed.org/fred-md' but this returns an SSL certificate error. I'm relatively new to python and I would greatly appreciate some help on solving this issue.

Thank you again.

Kind regards,

Ryan

Dear Chad,
Thanks for your work on statsmodels! it's a great and flexible tool.
But there's some calc bug or mistake in dynamic factor model large example? pls see below
In your example anti-transform of GDP are done like it's described here: https://www.bea.gov/help/faq/122
But it doesn't correspond to the function 'transform' that actually transforms gdp variable (with code '5') to stationary according to FRED's defs.
Regards

It appears that FRED MD and FRED QD are no longer hosted on AWS, but instead directly on FRED's website (e.g. https://files.stlouisfed.org/files/htdocs/fred-md/monthly/current.csv).

Following the code described in Chad Fulton "Large dynamic factor models, forecasting and nowcasting", and using the method get_prediction for forecasting and nowcasting of quarterly variables (GDP), should I resample the forecast output (mean of the quarter)? Or this method already resamples the data (i.e. Mariano Murasawa) and the estimated GDP of the quarter is simply the value of the third month of each quarter?

Dear Chad,

This is with regards to :

https://github.com/ChadFulton/tsa-notebooks/blob/master/dfm_coincident.ipynb

When I try to recreate your work ,cell number 10 gives me the following error:

ValueError Traceback (most recent call last)
in ()
9 rec = DataReader('USREC', 'fred', start=start, end=end)
10 ylim = ax.get_ylim()
---> 11 ax.fill_between(dates[:len(rec)], ylim[0], ylim[1], rec.values[:,0], facecolor='k', alpha=0.1);

/home/ashim/anaconda2/lib/python2.7/site-packages/matplotlib/init.pyc in inner(ax, *args, **kwargs)
1890 warnings.warn(msg % (label_namer, func.name),
1891 RuntimeWarning, stacklevel=2)
-> 1892 return func(ax, *args, **kwargs)
1893 pre_doc = inner.doc
1894 if pre_doc is None:

/home/ashim/anaconda2/lib/python2.7/site-packages/matplotlib/axes/_axes.pyc in fill_between(self, x, y1, y2, where, interpolate, step, **kwargs)
4785
4786 if not (x.shape == y1.shape == y2.shape == where.shape):
-> 4787 raise ValueError("Argument dimensions are incompatible")
4788
4789 mask = reduce(ma.mask_or, [ma.getmask(a) for a in (x, y1, y2)])

ValueError: Argument dimensions are incompatible

How can I fix this?

First of all, thank you very much for your extremely helpful notes and examples.

I am having an issue with the DFM nowcasting model, specifically the line:
for date in dta.keys():
dta[date].dta_m = dta[date].dta_m.reindex(columns, axis=1)
I get this error: ValueError: cannot reindex from a duplicate axis

Omitting this line then also gives the same error when I try to construct the model:
model = sm.tsa.DynamicFactorMQ(
endog_m, endog_quarterly=endog_q,
factors=factors, factor_orders=factor_orders,
factor_multiplicities=factor_multiplicities)

I wonder whether you can advise. Everything else until that point is fine. I constructed the description files as you did.

Thanks so much.

Hi Chad,

Thanks very much for sharing these useful notebooks. I went through markov_autoregression.ipynb and explored the MarkovAutogression class within the python statsmodels.tsa package. I have seen that the model is capable of performing in-sample predictions, but I couldn't figure out how to perform out-of-sample prediction. It will be a great help if you please direct me to the method which can perform out-of-sample predictions.

Thanks,
Tozammel

I am trying to reproduce your code and I keep getting the same error when running the EstimateRBC on simulated data. Specifically, when trying to fit the model, in the following lines:

sim_mod=EstimateRBC1(output=sim_observed[:,0],labor=sim_observed[:,1],consumption=sim_observed[:,2],measurement_errors=True)

sim_res = sim_mod.fit(maxiter=1000, information_matrix_type='oim')

The problem only arises when I try running more than 50 iterations, below that, everything runs but I keep getting the same warning:
Warnings: [1] Covariance matrix calculated using the outer product of gradients (complex-step)
I wonder if it is a problem with the fit() method? This has been a recurring problem with many Kalman Filter implementations in Python.

Thank you for your help.

Hi Chad,

I was trying to reproduce the SimpleRBC notebook and got the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-5e9b9ae9a7e9> in <module>()
     34 }
     35 model = SimpleRBC(rbc_data, calibrated=calibrated)
---> 36 sim_smoother = model.simulation_smoother()
     37 
     38 # Specify priors

AttributeError: 'SimpleRBC' object has no attribute 'simulation_smoother'

Is there a replacement function that I should be calling? Thanks!

Thank you for this notebook.
Could you share, pls 'fredmd_definitions.csv' and 'fredqd_definitions.csv'
Guess you've made it manually.

Hi,

Thank you for the examples. But I have hard time replicating your results using your notebook. I have no access to:

 base_url = 'https://s3.amazonaws.com/files.fred.stlouisfed.org/fred-md'
    
    # - FRED-MD --------------------------------------------------------------
    # 1. Download data
    orig_m = (pd.read_csv(f'{base_url}/monthly/{vintage}.csv')
                .dropna(how='all'))
    

Is possible to make the dataset available ?

Hi Chad,

Thank you very much for your model and walk through notes. I am running into the following error whenever I attempt to run:

Construct the dynamic factor model

model = sm.tsa.DynamicFactorMQ(
endog_m, endog_quarterly=endog_q,
factors=factors, factor_orders=factor_orders,
factor_multiplicities=factor_multiplicities)

"If a factors dictionary is provided, then it must include entries for each observed variable. Missing variables are: {nan}."

This is despite having the factors dictionary established with each variable having a corresponding group as per the following image.

Anything you might be able to share in solving this issue would be greatly appreciated.

Thank you,

Ryan

setar_model.py
in init(self, endog, order, ar_order, delay, thresholds, min_regime_frac, max_delay, threshold_grid_size, dates, freq, missing)

---> 55 if delay < 1 or delay >= len(endog):
56 raise ValueError('Delay parameter must be greater than zero'
57 ' and less than nobs. Got %d.' % delay

TypeError: '<' not supported between instances of 'NoneType' and 'int'

I suggest adding the topics time-series-analysis, econometrics, stochastic-volatility, mcmc, finance in the About section at https://github.com/ChadFulton/tsa-notebooks

I am running this example

In the past I had no problems replicating tis example. Now it seems has something changed and I get the following errors when I try to create the DFM object:

# Construct the dynamic factor model
model = sm.tsa.DynamicFactorMQ(
    endog_m, endog_quarterly=endog_q,
    factors=factors, factor_orders=factor_orders,
    factor_multiplicities=factor_multiplicities)

Traceback (most recent call last):
  File "/Users/safishajjouz/opt/anaconda3/envs/myport_management_env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-55-40928c00cd95>", line 2, in <module>
    model = sm.tsa.DynamicFactorMQ(
  File "/Users/safishajjouz/opt/anaconda3/envs/myport_management_env/lib/python3.8/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py", line 1320, in __init__
    s = self._s = DynamicFactorMQStates(
  File "/Users/safishajjouz/opt/anaconda3/envs/myport_management_env/lib/python3.8/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py", line 443, in __init__
    self.factor_block_orders.append(default_block_orders))
  File "/Users/safishajjouz/opt/anaconda3/envs/myport_management_env/lib/python3.8/site-packages/pandas/core/generic.py", line 5989, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'Series' object has no attribute 'append'

Dear All,

I had a look at the SV model using the MLE method for the univariate case, does anyone know how to implement the extended SV model with an additional dependent varaible, say SPY data? It is also known as SV-X model

And is it possible to change the distribution from a normal to a student-t distribution @ChadFulton

Hi Chad,

Thanks for the very interesting work. I was studying one of your projects and was confused by the following: in section "Details of impacts: April vintage compared to March vintage", Industrial Production (Final Products) is revised down (from -0.12 to -6.52), but this has a positive impact on the GDP growth forecast for Q2. Weird from a economic perspective.

What might be causing this?

Thanks,
Luís

Hi Chad, thank you very much for your amazing implementation of the DFM. I saw the monthly and quarterly endog variable but was wondering if the normal endog variable can also handle mixed frequency data which is not monthly and quarterly. I.e. data which is only daily and monthly to predict a monthly indicator? Any guidance on how to specify other mixed frequencies in your framework, while still maintaining the weighted average calculation, would be much appreciated. Many thanks, Tom

Hi,

I am getting some errors when running the notebook you provided. Isn't any problem with the default initialization ?

Thanks!

.ix indexer is deprecated, in favor of the more strict .iloc and .loc indexers

Hi,
I am new to python programming. I tried to replicate your code for setar model but it seems not to work. I receive the following error:

ModuleNotFoundError: No module named 'statsmodels.tsa.setar_model'

I am using python 3.

Also, I could not find the setar functionality in the official website of statsmodels. Thanks in advance.

Hi Chad
First of all i want to thank you for your amazing work!
I would like to use a 3 state Markov switching model for Treasury rates volatility. Am using (High - Low) from ^TNX from Yahoo as input (code snippet below).
am getting the following error:
'RuntimeError: Steady-state probabilities could not be constructed.'|
When i try to fit the model. Would be greatly appreciated if you could help me to resolve this.
Kind regards,
Steffen

data = pdr.get_data_yahoo('^TNX')
data = (data.drop(['Open','Close','Adj Close','Volume'], axis=1)).dropna()
data['dly_range'] = np.absolute(np.diff(data.values, axis=1))
dta_kns = data['dly_range']-data['dly_range'].mean()
# Kim, Nelson, and Startz (1998) Three-state Variance Switching - Fit the model
mod_kns = sm.tsa.MarkovRegression(dta_kns, k_regimes=3, trend='nc', switching_variance=True)
res_kns = mod_kns.fit()

The number and the order of lag of factors are directly given in the notes, but in fact how do we determine them?

Hi Chad,
Thank you so much for your work on statsmodels and DynamicFactorMQ. I have one question. 2020-02 - 2020-06 forecast for real GDP growth in 2020Q2 are 2.68%, 2.52%, -8.23%,-37.17%, -22.33%, respectively (In [32]). Total impact on 2020Q2 forecast from 2020-02 - 2020-06 are 0.16, -10.70, -29.30, 17.18. I found some difference between the total impact value and the change in GDP growth rate. For example, -22.33%-37.17% = 14.84% , not equal to 17.18. So what is the reason for the difference between the two?

Thank you so much.

• And estimation doesnt work as of now. I hope to update it soon.
  Thumbs up for the codes.

Good morning,
hope you are well.

i'm interested in your fully implemented DFM model with mixed frequency observations.
I would like to submit you to your attention:

1. in colab example, i get and error in defn_m = pd.read_csv('data/fredmd_definitions.csv'). FileNotFoundError: [Errno 2] No such file or directory: 'data/fredmd_definitions.csv'

2. It's not clear for me how does it works and the syntax, could you explain me better?

Thank you very much for your attention and passion, i wish you the best.

Paolo

Hello Mr. Fulton,

I have a problem when running kalman filter smoothing.

I build a model using jupyter notebook. If I run the same cell (model built and fit() ) a few times, then the model starts failing in the sense that I got huge AIC and BIC. But the first few runs of the cell are all good.

After that even if I restart my notebook, I still have the problem that I direct got huge huge AIC/BIC and the model stopped running.

Have you ever ran into this issue? And how do you solve it?

Many thanks and happy 2021