maroux / firefly-ynab4-importer Goto Github PK

View Code? Open in Web Editor NEW

6.0 3.0 0.0 90 KB

Importer for moving from YNAB4 to Firefly-iii

License: MIT License

Python 100.00%

firefly-iii firefly personal-finance ynab ynab4 ynab-converter budgeting

firefly-ynab4-importer's Introduction

Firefly Importer Importer

Simple importer for moving from YNAB4 (You Need A Budget) to Firefly-iii.

What?

This tool lets you migrate your financial history from YNAB 4 (not nYNAB) to Firefly-iii with minimal manual actions. It's written in Python and has the following features:

Create asset accounts, budgets, budget history, categories, revenue accounts, expense accounts.
Import splits accurately!
Automatically verifies the integrity of import by comparing Running Balance in YNAB 4 to Firefly iii
Foreign currency support - convert to foreign amounts in Firefly-iii seamlessly. Also gets real amounts from memo if that's part of your YNAB workflow.
Idempotent imports! Ran into an error mid-import? Just run it again after correcting the problem.
Handle inactive budgets
Caches firefly data so re-runs are fast

These cases are unsupported / not on the roadmap:

Importing budget limits - YNAB 4's secret sauce is the rules. One of those rules - live on last month's income doesn't match how Firefly budgets things. So this will require some more thought / investigation on how to set up sanely.
Multiple foreign currencies (rare, but possible)

Why?

YNAB 4 is (was?) a desktop software with support for sharing using external mechanisms like Dropbox or just local file sharing like Airdrop. After version 4, YNAB went online with monthly subscription, all data on the cloud and stopped supporting YNAB 4. If you don't like your financial data on the cloud, very few alternatives exist, and Firefly is one of the solid ones.

Firefly iii gives you control over where you want to host the application - local / in your own cloud etc, provides most of the features that YNAB 4 did and then some. Reports in particular are 💯.

However, if you have historical data in YNAB 4 like I do, starting fresh means throwing away all that history. Firefly iii does support migration from nYNAB (the cloud version) natively, but not from YNAB 4. There's also a csv importer but it misses a lot of things that this tool fixes, such as:

Doesn't import budget history.
Need manual management of accounts (otherwise it gets confused about "Citi" the asset account with "Citi" the expense account).
Transfers get duplicated because YNAB stores them as two transactions.
Splits are a whole can of worms.
No support for foreign transactions
Flaky import - doesn't really work for imports of more than 500 transactions at a time and fails intermittently.

How?

Pre-requisites:

Python 3.8+

Setup:

Install: pip install Firefly-YNAB4-Importer
Export YNAB 4 data to local disk
Setup config (see config.example.toml for documentation)
Backup Firefly iii database! (IMPORTANT!)
- Either using Firefly iii export, or just backing up your SQL database.
Run import:
```
export FIREFLY_III_URL=<firefly url>
export FIREFLY_III_ACCESS_TOKEN=<firefly access token>
firefly-ynab4-importer import <config file> "<register path>" "<budget path>"
```
where:
- <firefly url> is the url for your firefly installation
- <firefly access token> is the personal access token for your user
- <config file> is the path to config file created earlier
- <register path> is the path to the YNAB export register file (the one named
  
  <budge name> as of <timestamp>-Register.csv)
- <budget path> is the path to the YNAB export budget file (the one
  
  named <budge name> as of <timestamp>-Budget.csv).
- Remember to double quote since that path contains spaces.
Additional options:
- Limit imports to certain dates (this is useful in verifying that import works fine for your use case)
```
firefly-ynab4-importer import <config file> "<register path>" "<budget path>" "<start month>" "<end month>"
```

Development

Setup:

Install python 3.8+ using favorite tool e.g. Pyenv.
Optionally create virtualenv using your favorite method e.g. Pyenv virtualenv.
Install requirements: pip install -e .[dev]
Verify Firefly-YNAB4-Importer runs cleanly.

Publish new version:

python setup.py sdist bdist_wheel

twine upload dist/*

Bugs

We use GitHub issues for tracking bugs and feature requests. YNAB 4 and Firefly iii are both fairly complicated software. It's not only possible, but likely that you'll run into issues if your setup is moderate to high level of customization.

If you find a bug, please open an issue.

Contributing

If you find this useful and want to contribute, here's a list of feature I'd like to add -

Import Reconciliation transactions correctly
Investigate budget history support
Multiple foreign currencies (rare, but possible)
Command to clear cache, and move cache to appropriate directory depending on platform (e.g. ~/.cache on Unix)
Better error handling - guide user on how to correct problems
Testing
- Set up test fixtures for inputs and expected outputs
- Verify all the options in Config work correctly
Type checking - mypy checks.

firefly-ynab4-importer's People

Contributors

Stargazers

Watchers

firefly-ynab4-importer's Issues

Various errors and workarounds

Hi! I just wanted to start off by saying thank you for writing this! I had the fairly heavy task of importing almost six years of transactions from classic YNAB to Firefly, and despite the issues I'll be listing, this script simplified the task immensely!

That being said, I had some issues come up in the process. I was able to work through them all, but I wanted to post both how I solved them for other users, and also for the possibility of solutions in the code.

1. I was getting an error with the Pre-YNAB Debt Categories (possibly because I had them all hidden?)

AssertionError: Unable to process transaction with unknown budget: |Visa (Citibank) (hidden)|
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in <module>
----> 1 import_transactions(
      2     config_file="config.toml",
      3     register_file = "register.csv",
      4     budget_file = "budget.csv",
      5     dry_run = True,

~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in import_transactions(config_file, register_file, budget_file, dry_run, firefly_url, firefly_access_token, filter_min_date, filter_max_date)
     27         filter_max_date=filter_max_date,
     28     )
---> 29     importer.run(dry_run)
     30 

~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in run(self, dry_run)
    556         self._process_budgets()
    557         self._process_accounts()
--> 558         self._process_transactions()
    559         if not dry_run:
    560             self._load_cache()

~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in _process_transactions(self)
    808                 budget = self._budget(tx)
    809                 if budget:
--> 810                     assert budget in self.data.budgets, f"Unable to process transaction with unknown budget: |{budget}|"
    811                     # ignore hidden categories
    812                     if self.data.budgets[budget].active:

AssertionError: Unable to process transaction with unknown budget: |Visa (Citibank) (hidden)|

I fixed this by a) manually removing initial balance items from the register csv, and b) changing the lines with other transactions that were assigned to the category (mainly interest that was being charged the to pre-ynab debt balance) to be a misc. transaction.

2. Creation of the budget categories didn't work (possibly because of a change to the API?)

{'message': 'The given data was invalid.', 'errors': {'auto_budget_amount': ['The amount is required.']}}
HTTPError: 422 Client Error: Unprocessable Entity for url: <server redacted>/api/v1/budgets
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in <module>
----> 1 import_transactions(
      2     config_file="config.toml",
      3     register_file="register.csv",
      4     budget_file="budget.csv",
      5     dry_run=False,

~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in import_transactions(config_file, register_file, budget_file, dry_run, firefly_url, firefly_access_token, filter_min_date, filter_max_date)
   1356         filter_max_date=filter_max_date,
   1357     )
-> 1358     importer.run(dry_run)
   1359 # if __name__ == "__main__":
   1360 # cli()

~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in run(self, dry_run)
    568             self._create_currencies()
    569             self._create_categories()
--> 570             self._create_budgets()
    571             self._create_budget_limits()
    572             self._create_available_budgets()

~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in _create_budgets(self)
    988                         self.firefly_data.budgets[budget.name] = response.json()["data"]
    989                 else:
--> 990                     response = self._session.post("/api/v1/budgets", json=data)
    991                     self.firefly_data.budgets[budget.name] = response.json()["data"]
    992             except requests.HTTPError as e:

~/.local/share/virtualenvs/Firefly-YNAB4-Importer-br1XYdXy/lib/python3.8/site-packages/requests/sessions.py in post(self, url, data, json, **kwargs)
    588         """
    589 
--> 590         return self.request('POST', url, data=data, json=json, **kwargs)
    591 
    592     def put(self, url, data=None, **kwargs):

~/.repos/Firefly-YNAB4-Importer/firefly_ynab4_importer/main.py in request(self, method, url, **kwargs)
    511                 print(response.json())
    512                 print(f"{kwargs=}")
--> 513             response.raise_for_status()
    514         return response
    515     def get_all_pages(self, url, params=None, **kwargs) -> dict:

~/.local/share/virtualenvs/Firefly-YNAB4-Importer-br1XYdXy/lib/python3.8/site-packages/requests/models.py in raise_for_status(self)
    951 
    952         if http_error_msg:
--> 953             raise HTTPError(http_error_msg, response=self)
    954 
    955     def close(self):

HTTPError: 422 Client Error: Unprocessable Entity for url: <server redacted>/api/v1/budgets

I manually created all the budget categories in firefly to get around this error.

3. The function to create transactions is commented out by default (line 673 in main.py). Is this on purpose for some reason?

Fixed by uncommenting that line.

4. I had the same error reported in #2

This seems to occur on a subsequent execution when the previous attempt crashed. I fixed it by deleting the cache json before running (as mentioned in the issue)

5. I had some issues with special characters in the Payee column (e.g. é in café)

I fixed this by changing them to their ascii equivilent in both the csv and and the expense accounts page in Firefly.

Other Notes

Just a few things I jotted down while working through this that I think would be nice-to-haves:

The config template has some values that are different from the default. I found this challenging because I wondered which would be the "best" setting for most users. I think the default should be something that a new user can always try first, either with no config file or with the template.
Some sort of verbose mode that lists transactions and other info as its processing them would be very helpful to figure out where its breaking (I ended up needing to drop print statements into spots in the code to figure out the issues I had).

Import Errors

Hi,

Any ideas what could be causing this error mid import?

dacite.exceptions.WrongTypeError: wrong value type for field "revenue_accounts" - should be "typing.Dict[str, dict]" instead of value "{Redacted list of YNAB payees} of type "dict"

A few issues and a question

Hey,
Thanks for building the tool! I am trying to import my YNAB4 info for the last six years into Firefly and have been running into a few issues and am currently stuck. Python isn't my forte so have been fixing things as best I could.

Not sure if you prefer separate issues for things that I was able to fix, so listing them out here:

Had to install Colorama manually:
It's a dependency in code but is not part of the pip package. So in my venv I had to install it manually.
Accounts with no starting balance break import:
For some of my account I don't have a starting balance so the import script crashes on with

Traceback (most recent call last):
  File "firefly_ynab4_importer/main.py", line 1446, in <module>
    cli()
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "firefly_ynab4_importer/main.py", line 1442, in import_transactions
    importer.run(dry_run)
  File "firefly_ynab4_importer/main.py", line 638, in run
    self._process_accounts()
  File "firefly_ynab4_importer/main.py", line 795, in _process_accounts
    start_date, balance = starting_balances[acc]
KeyError: 'Tangerine MasterCard'

I got around this by changing line 792 to:

            if acc in starting_balances:
                start_date, balance = starting_balances[acc]

But now I'm stuck with this error:

Traceback (most recent call last):
  File "firefly_ynab4_importer/main.py", line 1445, in <module>
    cli()
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/raveesh/Projects/playground/ff-import/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "firefly_ynab4_importer/main.py", line 1441, in import_transactions
    importer.run(dry_run)
  File "firefly_ynab4_importer/main.py", line 639, in run
    self._process_transactions()
  File "firefly_ynab4_importer/main.py", line 880, in _process_transactions
    assert (
AssertionError: Unable to process transaction with unknown category: |Income:Available this month|

I can't figure out how that is supposed to be handled. Is this something in the config I need to change?

Thanks