Git Product home page Git Product logo

dickwolff / export-to-ghostfolio Goto Github PK

View Code? Open in Web Editor NEW
35.0 1.0 8.0 890 KB

Convert transaction history export from your favorite broker to a format that can be imported in Ghostfolio.

Home Page: https://hub.docker.com/r/dickwolff/export-to-ghostfolio

License: Apache License 2.0

TypeScript 99.81% Dockerfile 0.17% JavaScript 0.02%
ghostfolio ghostfolio-plugin degiro trading212 etoro rabobank schwab swissquote finpension ibkr

export-to-ghostfolio's Introduction

Export to Ghostfolio

Github-sponsors   BuyMeACoffee

Code Coverage

This tool allows you to convert a multiple transaction exports (CSV) to an import file that can be read by Ghostfolio. Currently there is support for:

Is your broker not in the list? Feel free to create an issue or, even better, build it yourself and create a pull request! For more info, see contributing.

Download transaction export

See the transaction export instructions for each of the supported brokers below.

View transaction export instructions

Bitvavo

Login to your Bitvavo account and click on your name at the top-right. Next, click "Transaction history". Click the blue "Export" button at the to to create an export. Select one of the CSv options and again click "Export".

Export instructions for Bitvavo

BUX

Open the app and go to "Account Value", and then "View History". Click the download icon in the top right corner to download your transaction history. The export will be sent to your email address.

Due to limitations by BUX, you can request up to 3 CSV exports per day!.

DEGIRO

Login to your DEGIRO account and create an export file (via Inbox > Account Overview, see image below). Choose the period from which you wish to export your history and click download.

Export instructions for DEGIRO

eToro

Login to your eToro account and navigate to "Portfolio". Then select "History" in the top menu. Next, click on the icon on the far right and select "Account statement". Choose the dates of interest and click "Create". On the next page, click on the Excel icon on the top right to download the file. After downloading, open the file in Excel and delete all the tabs except the "Account Activity" tab. Then use Excel to convert the file to CSV.

Finpension

Login to your Finpension account. Select your portfolio from the landing page. Then to the right of the screen select “Transactions”, on the following page to the right notice “transaction report (CSV-file)” and click to email or click to download locally.

Freetrade

Open the Freetrade app. Select your portfolio from the option in the top-left under the "Portfolio" heading. Select "Activity" from the list of icons along the bottom of the screen. Select the share icon in the top-right corner and then follow the on-screen instructions.

Interactive Brokers (IBKR)

Login to your Interactive Brokers account. Navigate to Account Management and click "Reporting" in the sidebar. Next, click on the "Flex Queries"-tab in the "Reporting" section. From the Flex "Queries section", Click the plus (+) icon on the right side to create a new Flex Query. Create a new Flex Query for Trades, and another one for Dividends. Set the export format to "CSV". See the required columns below the image.

Export instructions for IBKR

Trades

For trades, select "Trades". Then select the following properties: Buy/Sell, TradeDate, ISIN, Quantity, TradePrice, TradeMoney, CurrencyPrimary, IBCommission, IBCommissionCurrency.

Dividends

For dividends, select "Cash Transactions". Then select the following properties: Type, SettleDate, ISIN, Description, Amount, CurrencyPrimary.

Investimental

Login to your Investimental account and click on the "Orders Daily Log". Select account and desired time period then click refresh button. Transactions should appear and then click on the download button.

Export instructions for Investimental

Rabobank

Login to Rabobank and navigate to your investments. Navigate to "Transactions & Contract Notes" (Mutaties & Nota's). Select the range you wish to export at the top. Then scroll to the bottom of the page and click "Export as .csv"

Schwab

Login to your Schwab account. Go to “Accounts” then “History”. Select the account you want to download details from. Select the “Date Range” and select “Export” (csv). Save the file.

Export instructions for Schwab

Swissquote

Login to your Swissquote account. From the bar menu click on “Transactions”. Select the desired time period as well as types and then select the “export CSV” button to the right.

Trading 212

Login to your Trading 212 account and create an export file (via History > Download icon). Choose the period from which you wish to export your history and click download.

XTB

Login to your XTB account and from the top bar click on "Account history", then "Cash operations". Click the "Export button". Choose the period from which you wish to export your history, select report type "Cash Operations" choose file format "csv" then click "Export Report" button.

How to use

You can run the tool on your local machine by cloning this repository. You can also run the tool inside a Docker container. See the runtime specific instructions below.

Docker

Docker Pulls

View instructions

System requirements

To run the Docker container you need to have Docker installed on your machine. The image is published to Docker Hub.

How to use with the Dockerhub image

Contrary to the locally run version of the tool, the containerized version tries to determine which file type to process by looking to the header line inside the file. So there is no need to specify which converter to use.

You can then run the image like:

docker run --rm -v {local_in-folder}:/var/tmp/e2g-input -v {local_out_folder}:/var/tmp/e2g-output --env GHOSTFOLIO_ACCOUNT_ID=xxxxxxx dickwolff/export-to-ghostfolio

The following parameters can be given to the Docker run command.

Command Optional Description
-v {local_in-folder}:/var/tmp/e2g-input N The input folder where you put the files to be processed
-v {local_out_folder}:/var/tmp/e2g-output N The output folder where the Ghostfolio import JSON will be placed. Also the input file will be moved here when an error ocurred while processing the file.
-v {local_cache_folder}:/var/tmp/e2g-cache Y The folder where Yahoo Finance symbols will be cached
--env GHOSTFOLIO_ACCOUNT_ID=xxxxxxx N Your Ghostolio account ID 1
--env USE_POLLING=true Y When set to true, the container will continously look for new files to process and the container will not stop.
--env DEBUG_LOGGING=true Y When set to true, the container will show logs in more detail, useful for error tracing.
--env PURGE_CACHE=true Y When set to true, the file cache will be purged on start.
--env GHOSTFOLIO_VALIDATE=true Y When set to true, the tool with automatically validate the generated file against Ghostfolio.
--env GHOSTFOLIO_IMPORT=true Y When set to true, the tool will try to automatically import the generated file into Ghostfolio.
--env GHOSTFOLIO_URL=http://xxxxxxx Y The endpoint of your local Ghostfolio instance. E.g. http://192.168.1.15:3333. Use ONLY with a local Ghostfolio instance!
--env GHOSTFOLIO_SECRET=xxxxxxx Y The credentials of your Ghostfolio user. Used to authenticate with the import API endpoint. Use ONLY with a local Ghostfolio instance!

image

How to use by generating your own image

Use this option if you wish to run using an isolated docker environment where you have full control over the image and thus can trust it to contain only what is expected.

Clone this repository to your system and then run:

docker build -t export-to-ghostfolio .
docker run --rm -v {local_in-folder}:/var/tmp/e2g-input -v {local_out_folder}:/var/tmp/e2g-output --env GHOSTFOLIO_ACCOUNT_ID=xxxxxxx -t export-to-ghostfolio

You can use the same options as above for using the image on Dockerhub

Caching

The tool uses cacache to store data retrieved from Yahoo Finance inside the container. This way the load on Yahoo Finance is reduced and the tool should run faster. The cached data is stored inside the container in /var/tmp/e2g-cache. If you feel you need to invalidate your cache, you can do so by adding --env PURGE_CACHE=true to your run command. This will clear the cache on container start, and the tool will recreate the cache the next time it has to retrieve data from Yahoo Finance.

Run locally

View instructions

System requirements

The tool requires you to install the latest LTS version of Node, which you can download here. The tool can run on any OS on which you can install Node.

Use the tool

Next, clone the repo to your local machine and open with your editor of choice (e.g. Visual Studio Code).

Run npm install to install all required packages.

The repository contains a sample .env file. Rename this from .env.sample.

  • Put your export file path in the INPUT_FILE variable. This has to be relative to the root of the project.
  • Put the Ghostfolio account name where you want your transactions to end up at in GHOSTFOLIO_ACCOUNT_ID
    • This can be retrieved by going to Accounts > select your account and copying the ID from the URL

      image

  • Optionally you can set the GHOSTFOLIO_UPDATE_CASH variable to TRUE to automatically update your Ghostfolio account cash balance after processing the activities.
  • Optionally you can enable debug logging by setting the DEBUG_LOGGING variable to TRUE.

You can now run npm run start [exporttype]. See the table with run commands below. The tool will open your export and will convert this. It retrieves the symbols that are supported with YAHOO Finance (e.g. for European stocks like ASML, it will retrieve ASML.AS by the corresponding ISIN).

Exporter Run command
Bitvavo run start bitvavo (or bv)
BUX run start bux
DEGIRO run start degiro
eToro run start etoro
Finpension run start finpension (or fp)
Freetrade run start freetrade (or ft)
IBKR run start ibkr
Investimental run start investimental
Rabobank run start rabobank
Schwab run start schwab
Swissquote run start swissquote (or sq)
Trading 212 run start trading212 (or t212)
XTB run start xtb

Caching

The tool uses cacache to store data retrieved from Yahoo Finance on disk. This way the load on Yahoo Finance is reduced and the tool should run faster. The cached data is stored in /var/tmp/e2g-cache. If you feel you need to invalidate your cache, you can do so by removing the folder and the tool will recreate the cache when you run it the next time.

Import to Ghostfolio

Manually

The export file can now be imported in Ghostfolio by going to Portfolio > Activities and pressing the 3 dots at the top right of the table. Since Ghostfolio 1.221.0, you can now preview the import and validate the data has been converted correctly. When it is to your satisfaction, press import to add the activities to your portfolio.

image

Automatically (experimental)

There is an experimental feature (since 0.12.0) with which you can automatically validate and import the generated file into Ghostfolio! To use this, set the corresponding environment variables:

Variable Description
--env GHOSTFOLIO_VALIDATE=true When set to true, the tool with automatically validate the generated file against Ghostfolio.
--env GHOSTFOLIO_IMPORT=true When set to true, the tool will try to automatically import the generated file into Ghostfolio.
--env GHOSTFOLIO_URL=http://xxxxxxx The endpoint of your local Ghostfolio instance. E.g. http://192.168.1.15:3333. Use ONLY with a local Ghostfolio instance!
--env GHOSTFOLIO_SECRET=xxxxxxx The credentials of your Ghostfolio user. Used to authenticate with the import API endpoint. Use ONLY with a local Ghostfolio instance!

Contributing

We welcome any contribution to the repository. Feel free to create an issue or, even better, build it yourself and create a pull request!

The tool can be run two ways, manually and via Docker. Both entrypoints of the tool can be found in the ‘src/‘ folder. The tool uses a mock in the tests, which allow the tests to be run in a consistent and repeatable manner. This way there is no necessity for a live Yahoo Finance service. The mock was added because of inconsistencies in between test runs and rate-limiting issues with Yahoo Finance (with multiple consequetive runs, especially when running locally).

Whenever you add a new converter or create a fix for an existing one, please refer to the Wiki for instructions on how to extend the mock with testdata.

Footnotes

  1. You can retrieve your Ghostfolio account ID by going to Accounts > Edit for your account and copying the Account ID field

export-to-ghostfolio's People

Contributors

deej81 avatar dickwolff avatar gabrielgradinaru avatar lildude avatar sonlis avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

export-to-ghostfolio's Issues

No error indication when debugging errors raised when processing lines.

When running the conversion, I had an error that would display as such:

[e] An error ocurred while trying to retrieve symbol US7561091049 (line 8)! 

After 1 hour of debugging, and printing the correct error, it was because of this:

[Error: EACCES: permission denied, mkdir '/var/e2g-cache'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: '/var/e2g-cache'
} 

It would be an improvement to display the user why did it fail, instead of just letting them know an error has risen.
I would propose to add the error to the console log message here:

let message = `\n[e] An error ocurred while trying to retrieve {query} (line ${index + 2})!\n`;

Such as:

    protected logQueryError(query: string, index: number, err: string) {

        let message = `\n[e] An error ocurred while trying to retrieve {query} (line ${index + 2}): ${err}.\n`;

And add the error when calling the logQueryError function for each converter, but for example here:

this.logQueryError(record.isin || record.product, idx);
.

Or modify the ErrorCallbackFunction from just returning process.exit(99) to log the error before exiting.

all stocks give an error only the fee's gets ready to upload

when i run the npm run start degiro with my file in it it says at all stocks "x with currency USD! please add this manually with currency EUR"
this is the same for EUR stocks.

is there a way to fix this ?

i tried changing these lines:
XTB_ACCOUNT_CURRENCY = "EUR"
XTB_ACCOUNT_CURRENCY = "USD"
but that didn't help

[Degiro exporter v2] Taxes on dividend are added as dividends instead of substracting it from the dividend

Taxes on dividends are counted as dividends instead of being substracted from said dividend.

Bug

Given this CSV, and Impots sur dividende meaning tax on dividend:

18-03-2024,08:38,15-03-2024,REALTY INCOME CORP,US7561091049,Dividende,,USD,0.51,USD,0.36,
18-03-2024,08:38,15-03-2024,REALTY INCOME CORP,US7561091049,Impôts sur dividende,,USD,-0.15,USD,-0.15,

Outputs the following:

    {
      "accountId": "ed7a85f8-1c5f-4d84-b700-c0081c6c6ff2",
      "comment": "",
      "fee": 0,
      "quantity": 1,
      "type": "DIVIDEND",
      "unitPrice": 0.51,
      "currency": "USD",
      "dataSource": "YAHOO",
      "date": "2024-03-18T08:38:00+02:00",
      "symbol": "RY6.F"
    },
    {
      "accountId": "ed7a85f8-1c5f-4d84-b700-c0081c6c6ff2",
      "comment": "",
      "fee": 0,
      "quantity": 1,
      "type": "DIVIDEND",
      "unitPrice": 0.15,
      "currency": "USD",
      "dataSource": "YAHOO",
      "date": "2024-03-18T08:38:00+02:00",
      "symbol": "RY6.F"
    },

There should be only one dividend of 0.36.

new install fails on dayjs import

i tried running the degiro exporter both by using dockerfile and by running it without dockerfile, both gave same error.

i used release 0.12.0 and the latest main branch, both failed
OS: Windows 11
Nodejs version used when running without dockerfile: 20.11.1

node:internal/process/esm_loader:34
internalBinding('errors').triggerUncaughtException(
^
Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'C:\Users\jovangoe\Downloads\Export-To-Ghostfolio\node_modules\dayjs\plugin\customParseFormat' imported from C:\Users\jovangoe\Downloads\Export-To-Ghostfolio\src\converters\degiroConverter.ts
Did you mean to import dayjs/plugin/customParseFormat.js?
at finalizeResolution (node:internal/modules/esm/resolve:264:11)
at moduleResolve (node:internal/modules/esm/resolve:917:10)
at defaultResolve (node:internal/modules/esm/resolve:1130:11)
at nextResolve (node:internal/modules/esm/hooks:865:28)
at S (file:///C:/Users/jovangoe/Downloads/Export-To-Ghostfolio/node_modules/tsx/dist/esm/index.mjs?1714897062814:2:2951)
at nextResolve (node:internal/modules/esm/hooks:865:28)
at Hooks.resolve (node:internal/modules/esm/hooks:303:30)
at handleMessage (node:internal/modules/esm/worker:196:24)
at Immediate.checkForMessages [as _onImmediate] (node:internal/modules/esm/worker:138:28)
at process.processImmediate (node:internal/timers:478:21) {
code: 'ERR_MODULE_NOT_FOUND',
url: 'file:///C:/Users/jovangoe/Downloads/Export-To-Ghostfolio/node_modules/dayjs/plugin/customParseFormat'
}

Failure to deal with empty response from Yahoo Finance in combination with no symbol being present.

Hi,

Yesterday I've setup Ghostfolio, found your project and I tried to use this project to import my DeGiro transactions.

However, in my DeGiro export contains the ISIN, US00165C1045. US00165C1045 is AMC, but the Yahoo Finance response is an empty array on the quotes element. So the converter it tries to rely on the symbol but the DeGiro export doesn't contain symbols for that line and thus subsequently fails.

The CSV line on which the convert fails:
01-02-2021,15:39,01-02-2021,AMC ENTERTAINMENT HOLD,US00165C1045,Valuta Debitering,1.2108,USD,-16.70,USD,0.00,43a88697-7358-4389-9440-d8aa6db536ce

Yahoo finance response

{
  explains: [],
  count: 0,
  quotes: [],
  news: [],
  nav: [],
  lists: [],
  researchReports: [],
  screenerFieldResults: [],
  totalTime: 23,
  timeTakenForQuotes: 415,
  timeTakenForNews: 0,
  timeTakenForAlgowatchlist: 400,
  timeTakenForPredefinedScreener: 400,
  timeTakenForCrunchbase: 400,
  timeTakenForNav: 400,
  timeTakenForResearchReports: 0,
  timeTakenForScreenerField: 0,
  timeTakenForCulturalAssets: 0
}

Stacktrace

TypeError: Cannot read properties of undefined (reading 'symbol')
    at YahooFinanceService.<anonymous> (/home/sander/tmp/Export-To-Ghostfolio/src/yahooFinanceService.ts:77:33)
    at Generator.next (<anonymous>)
    at fulfilled (/home/sander/tmp/Export-To-Ghostfolio/src/yahooFinanceService.ts:5:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
/home/sander/tmp/Export-To-Ghostfolio/src/yahooFinanceService.ts:77
            symbol = symbols[0].symbol.split(".")[0];
                                ^
TypeError: Cannot read properties of undefined (reading 'symbol')
    at YahooFinanceService.<anonymous> (/home/sander/tmp/Export-To-Ghostfolio/src/yahooFinanceService.ts:77:33)
    at Generator.next (<anonymous>)
    at fulfilled (/home/sander/tmp/Export-To-Ghostfolio/src/yahooFinanceService.ts:5:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Swchab conversion error

Tried with multiple accounts and I always get the following output:

docker run --rm -v $PWD/exports/:/var/tmp/e2g-input -v .:/var/tmp/e2g-output --env GHOSTFOLIO_ACCOUNT_ID=<some-acct-id>  --env DEBUG_LOGGING=true dickwolff/export-to-ghostfolio

> [email protected] watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Personal-Positions-2024-05-19.csv!
[i] Determined the file type to be of kind 'ibkr'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using IBKR converter
[e] An error ocurred while processing.
[e] Error details: Error: An error ocurred while parsing!
[e] Moving file to output..
[i] Stop container as usePolling is set to false..

Then when trying to import to Ghostfolio I get this error:

activities.0.currency is not valid
{
  "Positions for account Personal ...436 as of 04:48 PM ET, 2024/05/19": null,
  "": null
}

Degiro import fails with "[e] An error ocurred while trying to retrieve an empty symbol (line 38)!"

Hello.

I'm using the latest 0.16.0 version and when I try to import the Degiro transactions (from Accounts.csv), I get the following error: [e] An error ocurred while trying to retrieve an empty symbol (line 38)!

My Degiro data is in Portuguese (not sure if that is the problem, but Degiro doesn't change the transaction data even after changing the interface language).

The line in question reads: 18-06-2024,07:16,17-06-2024,,,Crédito de divisa,,EUR,2.69,EUR,3292.76,

Add test coverage to `SecurityService.ts`

After #58 (#60) is completed, add test coverage to the Security Service class. This is now tested because all the converters call it, but not all paths have been covered. Add a test file for this.

XTB import fails with `TypeError: Cannot read properties of null (reading '1')`

Hello.

I'm using the latest 0.16.0 version and when I try to import the XTB transactions, I get the following error: TypeError: Cannot read properties of null (reading '1')

This is the last block of the import plus the error:

{
  iD: '519422100',
  type: 'Swap',
  time: '21.03.2024 13:30:19',
  symbol: 'DIS.US',
  comment: 'Swap of position #994025001',
  amount: -142.12
}
/app/src/converters/xtbConverter.ts:114
                const quantity = parseFloat(match[1]);
                                            ^


TypeError: Cannot read properties of null (reading '1')
    at <anonymous> (/app/src/converters/xtbConverter.ts:114:45)

Node.js v20.15.1

And on the csv, the corresponding line of the message and the next one (not sure if the error is about the block already presented or when picking the next one) is:

519422100;Swap;21.03.2024 13:30:19;DIS.US;Swap of position #994025001;-142.12 
519422099;Profit/Loss (FX/CFD);21.03.2024 13:30:19;DIS.US;Profit of position #994025001;500.79

Include TradeRepublic

is there some way to include TradeRepublic? There is a Private API, where i get all my Transactions (in PDF)

UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'length' of undefined at DeGiroConverter

I just installed Ghostfolio the docker compose way and thats up and running. Now im trying to import my deGiro transactions from the last couple year, but after running npm run start degiro I get the following error:

> [email protected] start
> nodemon "degiro"

[nodemon] 2.0.22
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): src/**/*
[nodemon] watching extensions: ts,js
[nodemon] starting `ts-node ./src/index.ts degiro`
Processing file using DeGiro converter
Read CSV file sample-trading212-export.csv. Start processing..
(node:2894272) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'length' of undefined
    at DeGiroConverter.<anonymous> (/home/ingz/docker/compose/apps/Ghostfolio/Export-To-Ghostfolio/src/converters/degiroConverter.ts:52:49)
    at Generator.next (<anonymous>)
    at /home/ingz/docker/compose/apps/Ghostfolio/Export-To-Ghostfolio/src/converters/degiroConverter.ts:31:71
    at new Promise (<anonymous>)
    at __awaiter (/home/ingz/docker/compose/apps/Ghostfolio/Export-To-Ghostfolio/src/converters/degiroConverter.ts:27:12)
    at /home/ingz/docker/compose/apps/Ghostfolio/Export-To-Ghostfolio/src/converters/degiroConverter.ts:40:45
    at Parser.<anonymous> (/home/ingz/docker/compose/apps/Ghostfolio/Export-To-Ghostfolio/node_modules/csv-parse/dist/cjs/index.cjs:1392:7)
    at Parser.emit (events.js:314:20)
    at Parser.EventEmitter.emit (domain.js:483:12)
    at errorOrDestroy (internal/streams/destroy.js:108:12)
(node:2894272) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:2894272) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
[nodemon] clean exit - waiting for changes before restart

[Degiro exporter v2] Error parsing Dutch transaction export from degiro.nl

claesmathias@rpifive:~/export-to-ghostfolio$ docker logs -f export-to-ghostfolio

[email protected] watch
ts-node ./src/watcher.ts

[i] Watching /var/e2g-input (using polling)..
[i] Found Transactions.csv!
[i] Determined the file type to be of kind 'degiro-v2' (overidden by environment variable).
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using DeGiro converter (V2 Beta)
[i] NOTE: You are running a converter that is currently in beta.
[i] If you have any issues, please report them on GitHub. Many thanks!
[e] An error ocurred while processing.
[e] Error details: Error: An error ocurred while parsing!
[e] Moving file to output..
Transactions.csv

Support for Interactive Brokers (IBKR)

It would be awesome if you could support IBKR.
I've created a custom flex query for the trades and dividends + taxes.
I'll create a small explanation on how to create the query, if you decide to implement this broker.

Trades

  • The trades report also contains money conversions (the last lines, which are missing an ISIN).
  • You do not need to subtract the IBCommission from TradeMoney, this is already done. I could also add the Money without the fees subtracted, if you prefer.

Trades.csv:

"Buy/Sell","TradeDate","ISIN","Quantity","TradePrice","TradeMoney","CurrencyPrimary","IBCommission","IBCommissionCurrency"
"BUY","20230522","CH0111762537","7","282.7","1978.9","CHF","-5","CHF"
"BUY","20230522","US9220427424","95","93.78","8909.1","USD","-1","USD"
"BUY","20230609","US9220427424","105","95.42","10019.1","USD","-1","USD"
"BUY","20230609","US9220427424","11","95.46","1050.06","USD","-1","USD"
"BUY","20230803","US9220427424","1","97.84","97.84","USD","-0.979","USD"
"BUY","20231228","US9220427424","2","103.18","206.36","USD","-1","USD"
"BUY","20240126","US9220427424","97","103.61","10050.17","USD","-1","USD"
"BUY","20240129","US9220427424","12","103.66","1243.92","USD","-1","USD"
"SELL","20230522","","-8000","1.1173","-8938.4","USD","-1.79924","CHF"
"SELL","20230609","","-10012","1.10725","-11085.787","USD","-1.79802","CHF"
"SELL","20240126","","-10020","1.1537","-11560.074","USD","-1.73416","CHF"

Dividends and Taxes

Dividends_Taxes.csv

  • Sadly, there is no option to directly output the amount of the shares you're holding. You'll need to divide the Amount by Price per share (extracted from Description string).
  • Regex for price per share: (\d+(\.\d+)?)(?= PER SHARE)
"Type","SettleDate","ISIN","Description","Amount","CurrencyPrimary"
"Withholding Tax","20230913","CH0111762537","SMMCHA(80486947) CASH DIVIDEND CHF 5.99 PER SHARE - CH TAX","-14.68","CHF"
"Withholding Tax","20230623","US9220427424","VT(US9220427424) CASH DIVIDEND USD 0.6504 PER SHARE - US TAX","-20.58","USD"
"Withholding Tax","20230921","US9220427424","VT(US9220427424) CASH DIVIDEND USD 0.4055 PER SHARE - US TAX","-12.9","USD"
"Withholding Tax","20231221","US9220427424","VT(US9220427424) CASH DIVIDEND USD 0.8008 PER SHARE - US TAX","-25.47","USD"
"Dividends","20230913","CH0111762537","SMMCHA(CH0111762537) CASH DIVIDEND CHF 0.69 PER SHARE (Return of Capital)","4.83","CHF"
"Dividends","20230913","CH0111762537","SMMCHA(80486947) CASH DIVIDEND CHF 5.99 PER SHARE (Ordinary Dividend)","41.93","CHF"
"Dividends","20230623","US9220427424","VT(US9220427424) CASH DIVIDEND USD 0.6504 PER SHARE (Ordinary Dividend)","137.23","USD"
"Dividends","20230921","US9220427424","VT(US9220427424) CASH DIVIDEND USD 0.4055 PER SHARE (Ordinary Dividend)","85.97","USD"
"Dividends","20231221","US9220427424","VT(US9220427424) CASH DIVIDEND USD 0.8008 PER SHARE (Ordinary Dividend)","169.77","USD"

Docker image?

Hello,
Any plans of containerizing this application yet? I am trying to do it myself but it appears that some dependencies wouldn`t work in production environment (and crash in development as well)
Thanks!

DeGiro conversion failing

When using the docker instance I get the following error when trying to convert a freshly downloaded DeGiro transaction list;

[email protected] watch
tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found ._Transactions.csv!
[i] Determined the file type to be of kind 'freetrade'.
[i] Found Transactions.csv!
[i] Determined the file type to be of kind 'degiro'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using Freetrade converter
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using DeGiro converter
[e] An error ocurred while processing.
[e] Error details: Error: An error ocurred while parsing!
=> undefined
[e] Moving file to output..
[i] Stop container as usePolling is set to false..
npm notice
npm notice New minor version of npm available! 10.5.2 -> 10.8.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.8.0
npm notice Run npm install -g [email protected] to update!
npm notice

file format:
Transactions.csv

Could you support understanding this issue?

Improve predictability and decrease manual testing time with automatic testing

To prevent manual testing every time I do a change, which is a chore on it’s own, the library should be tested automatically using a testing framework and run automatically with GitHub actions.

Especially the DEGIRO converter could profit massively from this.

Start with an easy one, eg. Finpension and then ramp up to all of them.

Only Fees being processed in Docker

Hi Dick,

I'm back after a long time! great work on the converter in the mean time! Congrats.

I've been playing with the docker instance. Using both docker and docker compose.
When trying to process Finpension and Swissquote, the only data converted is fees and interest but no equities.

let me know what you may need to track this down.
Mikael

eToro converter - transactions with same `Details` not counted?

Hi!
I am trying to convert my January transactions and there are multiple transactions (with the same name) which are happening. It looks like the converter only took in consideration the first of the list and then it jumps to the next one which has a different value in the Details field.

In the example below it will only count the first NXPI/USD dividend and the first OLED/USD position closed.

Date,Type,Details,Amount,Units,Realized Equity Change,Realized Equity,Balance,Position ID,Asset type,NWA
07/01/2024 13:21:57,Dividend,NXPI/USD,0.02,-,0.02,"45,3.00",0.00,1520124620,Stocks,0.00
07/01/2024 13:21:57,Dividend,NXPI/USD,0.02,-,0.02,"45,3.02",0.00,1539539526,Stocks,0.00
07/01/2024 13:21:57,Dividend,NXPI/USD,0.03,-,0.03,"45,3.05",0.00,2472065850,Stocks,0.00
09/01/2024 15:30:40,Position closed,OLED/USD,18.43,0.102626,7.37,"45,0.42",0.00,2355395242,Stocks,0.00
09/01/2024 15:30:40,Position closed,OLED/USD,8.87,0.049420,1.80,"45,2.22",0.00,2472033271,Stocks,0.00
09/01/2024 15:30:41,Position closed,OLED/USD,89.92,0.500702,4.31,"45,6.53",0.00,1446690287,Stocks,0.00

Thanks! for checking this out!

Fix workflow runs on PR from other contributors

When a contributor opens a PR, the GitHub workflows do not run. I have to approve each run manually. When I run a workflow (eg. testing), the run fails on adding the test results on the PR:
image

The testing workflow needs to be able to run on any PR, and publish the results on the PR.

I don't want the Docker pipeline to run though, so not any image will be pushed to Docker hub.

[Degiro exporter v1] undefined 'comment'

ran using docker:
docker run -d -v ./input:/var/e2g-input -v ./output:/var/e2g-output --env GHOSTFOLIO_ACCOUNT_ID=<id here> dickwolff/export-to-ghostfolio

[i] Watching /var/e2g-input..
[i] Found degiro_export.csv!
[i] Determined the file type to be of kind 'degiro'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using DeGiro converter
[i] NOTE: There is a new version available of the DeGiro converter
[i] The new converter has multiple record parsing improvements and also supports platform fees.
[i] The new converter is currently in beta and we're looking for your feedback!
[i] You can run the beta converter with the command 'npm run start degiro-v2'.
[i] Read CSV file. Start processing..
/app/src/converters/degiroConverter.ts:155
            if (result.activities[result.activities.length - 1].comment === "txfees") {
                                                                ^
TypeError: Cannot read properties of undefined (reading 'comment')
    at DeGiroConverter.<anonymous> (/app/src/converters/degiroConverter.ts:155:65)
    at Generator.next (<anonymous>)
    at fulfilled (/app/src/converters/degiroConverter.ts:6:58)
npm notice
npm notice New minor version of npm available! 10.2.4 -> 10.5.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.5.0>
npm notice Run `npm install -g [email protected]` to update!
npm notice

running version 2 worked : --env FORCE_DEGIRO_V2=true in terms of no errors.

However, after checking the data there are parsing issues.

Symbol mapping

I think currently you take the first symbol we can find in Yahoo ?

Would it be possible to somehow specify the symbol to be used?

Eg in my case it used SPY5.L but I would have preferred SPY5.DE.

Many thanks
T

Degiro conversion error - module not found

Hi everyone.
I'm trying to convert my order history exported from Degiro but I'm getting the following error:

➜  Export-To-Ghostfolio git:(main) npm run start degiro

> [email protected] start
> tsx ./src/manual.ts "degiro"


node:internal/process/esm_loader:34
      internalBinding('errors').triggerUncaughtException(
                                ^
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/home/username/Scaricati/Export-To-Ghostfolio/node_modules/dayjs/plugin/customParseFormat' imported from /home/username/Scaricati/Export-To-Ghostfolio/src/converters/degiroConverter.ts
Did you mean to import "dayjs/plugin/customParseFormat.js"?
    at finalizeResolution (node:internal/modules/esm/resolve:265:11)
    at moduleResolve (node:internal/modules/esm/resolve:933:10)
    at defaultResolve (node:internal/modules/esm/resolve:1157:11)
    at nextResolve (node:internal/modules/esm/hooks:866:28)
    at S (file:///home/username/Scaricati/Export-To-Ghostfolio/node_modules/tsx/dist/esm/index.mjs?1714900896692:2:2951)
    at nextResolve (node:internal/modules/esm/hooks:866:28)
    at Hooks.resolve (node:internal/modules/esm/hooks:304:30)
    at handleMessage (node:internal/modules/esm/worker:196:24)
    at Immediate.checkForMessages [as _onImmediate] (node:internal/modules/esm/worker:138:28)
    at process.processImmediate (node:internal/timers:478:21) {
  code: 'ERR_MODULE_NOT_FOUND',
  url: 'file:///home/username/Scaricati/Export-To-Ghostfolio/node_modules/dayjs/plugin/customParseFormat'
}
Node.js v20.12.2

Same issue when using Docker image or local build.
I'm on Fedora 40, Node version v20.12.2

Make DEGIRO V2 GA

V2 works good. Make V2 the default and mark V1 as obsolete.

  • Replace degiro-v2 with degiro
  • Replace old degiro with degiro-v1
  • Remove beta notice and other relevant messages from degiro-v2
  • Add console.log notice to degiro-v1 that it won't receive updates anymore

Add functionality to automatically validate JSON and upload to Ghostfolio

Ghostfolio offers endpoints to automatically validate and upload activities.

There should be two settings that allows users to automatically upload the converted JSON to Ghostfolio. This can be done in two steps. Firstly, validate the JSON. And then also import automatically.

This will be an experimental feature, because there is no human oversight for checking duplicate transactions.

Got a parsing error

[email protected] watch
tsx ./src/watcher.ts

[i] Purging cache (PURGE_CACHE set to true)..
[i] Watching /var/tmp/e2g-input (using polling)..
[i] Cache purged!
[i] Found Transactions(1).csv!
[i] Determined the file type to be of kind 'degiro'.
[i] Found xStation5_cashOperations_2770548_31-12-2006--29-05-2024.csv!
[i] Determined the file type to be of kind 'xtb'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using DeGiro converter
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using XTB converter
[e] An error ocurred while processing.
[e] Error details: Error: An error ocurred while parsing!
[e] Moving file to output..
[i] Read CSV file. Start processing..
/app/src/converters/xtbConverter.ts:112
const quantity = parseFloat(match[1]);
^

TypeError: Cannot read properties of null (reading '1')
at (/app/src/converters/xtbConverter.ts:112:45)
at Parser. (file:///app/node_modules/csv-parse/lib/index.js:102:7)
at Parser.emit (node:events:519:28)
at Parser.emit (node:domain:488:12)
at endReadableNT (node:internal/streams/readable:1696:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Node.js v20.13.1

Example File
xStation5_cashOperations_2770548_31-12-2006--29-05-2024.csv

Error when processing DEGIRO file due to Currency Exchange operation

When operating in DEGIRO in multiple currencies (In my case EUR and USD), there is an entry on the export file that refers to registering the exchange rate used for the transaction (more information here in Spanish 🙏🏼 ).

This causes an error on the export.

Screenshot 2024-07-02 at 20 08 10

How to reproduce

Using this file:
Account-2.csv

Possible fix

Skip this line? As it is apparently only a registry of the type of currency exchange? 🤔

[Degiro exporter v2] Fees are not properly matched to their transaction

The fees paid during a transaction are not linked to said transaction.

Error

In my example, using this CSV:

Date,Heure,Date de,Produit,Code ISIN,Description,FX,Mouvements,,Solde,,ID Ordre
11-03-2024,10:39,11-03-2024,QT GROUP OYJ,FI4000198031,Frais DEGIRO de courtage et/ou de parties tierces,,EUR,-4.90,EUR,33.53,cce1bd4c-9404-49b0-b69a-43a5c307d3c5
11-03-2024,10:39,11-03-2024,QT GROUP OYJ,FI4000198031,"Achat 6 QT GROUP OYJ@79,96 EUR (FI4000198031)",,EUR,-479.76,EUR,38.43,cce1bd4c-9404-49b0-b69a-43a5c307d3c5

The fees are not added to the transaction, even though they are from the same order.

Using the degiro sample export csv, the same error arises:

15-05-2019,09:05,15-05-2019,ISHARES MSCI WOR A,IE00B4L5Y983,"Compra 6 ISHARES MSCI WOR A@49,785 EUR (IE00B4L5Y983)",,EUR,-298.71,EUR,0.64,a47e2746-bfbd-4654-bd6c-5e58e470d32f
02-01-2024,14:42,02-01-2024,ISHARES MSCI WOR A,IE00B4L5Y983,Comissões de transação DEGIRO e/ou taxas de terceiros,,EUR,-1.00,EUR,2.54,7b377a93-5695-4131-8954-5c78996fbed4
02-01-2024,14:42,02-01-2024,ISHARES MSCI WOR A,IE00B4L5Y983,"Compra 1 ISHARES MSCI WOR A@82,055 EUR (IE00B4L5Y983)",,EUR,-82.06,EUR,3.54,7b377a93-5695-4131-8954-5c78996fbed4

The commision is added to the first transaction instead of the second, while it belongs to the second as the order number suggests.

    {
      "accountId": "ed7a85f8-1c5f-4d84-b700-c0081c6c6ff2",
      "comment": "",
      "fee": 1, <- 1 fee here even though it should be 0
      "quantity": 6,
      "type": "BUY",
      "unitPrice": 49.785,
      "currency": "EUR",
      "dataSource": "YAHOO",
      "date": "2019-05-15T09:05:00+03:00",
      "symbol": "IWDA.AS"
    },
    {
      "accountId": "ed7a85f8-1c5f-4d84-b700-c0081c6c6ff2",
      "comment": "",
      "fee": 0, <- 0 fee here even though there should be 1
      "quantity": 1,
      "type": "BUY",
      "unitPrice": 82.06,
      "currency": "EUR",
      "dataSource": "YAHOO",
      "date": "2024-01-02T14:42:00+02:00",
      "symbol": "IWDA.AS"
    },

Problem within the code

The code always tries to match a transaction and fee with transaction being n, and fee n+1. However, it is not always the case, as my example export shows, and even the sample one.

if (this.lookaheadIsSameProduct(records, record, idx) || this.isBuyOrSellRecord(record) || this.isDividendRecord(record)) {

if (this.lookaheadIsSameProduct(records, record, idx) || this.isBuyOrSellRecord(record) || this.isDividendRecord(record)) {
  const combinedRecord = this.combineRecords(record, records[idx + 1], security);

Possible solution

Add also a check to n-1 or/and add a check if the order number is the same between the order and the fee.

I could work on it if needed 👍

JSON not working if language does not match - date must be a valid ISO 8601 date string

Hi

Since I can not respond to the old issue (dickwolff/DEGIRO-to-Ghostfolio#1) I opened a new one here. It looks better now, but now something with the date doesn`t work.

Image:
image

Example JSON output
{ "accountId": "9b6128be-97ff-4d53-a7b0-ef58413e307f", "comment": "", "fee": 0.29, "quantity": 1, "type": "DIVIDEND", "unitPrice": 1.93, "currency": "USD", "dataSource": "YAHOO", "date": "Invalid Date", "symbol": "A" }

degiro exporters only writes 12 transactions

When trying to run the degiro exporters it only exports 12 transactions and then stops:
[nodemon] 2.0.22 [nodemon] to restart at any time, enter rs[nodemon] watching path(s): src\**\* [nodemon] watching extensions: ts,js [nodemon] startingts-node ./src/index.ts degiro Processing file using DeGiro converter Read CSV file C:\Users\User\Downloads\Account.csv. Start processing.. ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 4% | ETA 225s | Duration: 18s | 12/273 Processing complete, writing to file.. Wrote data to 'ghostfolio-degiro.json'! [nodemon] clean exit - waiting for changes before restart

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.