eshaham / israeli-ynab-updater Goto Github PK

To encrypt the user credentials, we currently use a salt hash and check it into the repo.
We should either instruct users to generate their own salt, or generate a random one on first use.

Hey,

Awesome project!
I might be missing something obvious, but - is there a way to define a few different user accounts for the same scraper? E.g., if I want to scrape Leumi Card's site using two different user accounts?

Thanks,
Edan

On initial config and scraper run, the following result is yielded:

Bank Leumi: START_SCRAPING
Bank Leumi: INITIALIZING
(node:1113292) [DEP0106] DeprecationWarning: crypto.createDecipher is deprecated.
Error: Protocol error (Performance.enable): Target closed.
    at /home/yuval/dev/israeli-ynab-updater/node_modules/puppeteer/src/common/Connection.ts:270:57
    at new Promise (<anonymous>)
    at CDPSession.send (/home/yuval/dev/israeli-ynab-updater/node_modules/puppeteer/src/common/Connection.ts:269:12)
    at Page._initialize (/home/yuval/dev/israeli-ynab-updater/node_modules/puppeteer/src/common/Page.ts:492:20)
    at Function.create (/home/yuval/dev/israeli-ynab-updater/node_modules/puppeteer/src/common/Page.ts:361:16)
    at /home/yuval/dev/israeli-ynab-updater/node_modules/puppeteer/src/common/Target.ts:120:14
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 0)
    at BrowserContext.pages (/home/yuval/dev/israeli-ynab-updater/node_modules/puppeteer/src/common/Browser.ts:621:19)
    at async Promise.all (index 0)

Any ideas how to debug this?

Regarding the following code:

Even if the credentials are encrypted with AES with a preconfigured key, it's still decryptable by simply the file (since the key is on GitHub).

How about moving to using system's keychain management instead? With a cross-platform library such as keytar

(node:65127) UnhandledPromiseRejectionWarning: TypeError: (0 , _json2csv2.default) is not a function
at generateSingleReport (/bank-scrapers/israeli-ynab-updater/src/scrape/generate-reports.js:123:23)
at exports.default (/bank-scrapers/israeli-ynab-updater/src/scrape/scrape-task.js:92:13)
(node:65127) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:65127) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code

This repository aimed for YNAB format, but as far as I can see, he generally creates a csv with only some of the details.

This repository provides very convenient CLI, and it is a pity not to use it as the primary CLI.

So I suggest that there be an option, some flag, that allows exporting all transactions as they are, to csv.

If someone updates the israeli-bank-scrapers npm module then it doesn't automatically gets reflected in the ynab updater because the package.json points to a specific version.

Even if the package.json pointed to the latest version (e.g. israeli-bank-scrapers@latest) then if someone ran npm install a while ago on israeli-ynab-updater then he'll have to remember to run npm install to get the latest bank scrapers.

@ifeins @eshaham I'm closing eshaham/israeli-bank-scrapers#596 in favor of this one.

scraping Visa Cal
Visa Cal: START_SCRAPING
Visa Cal: INITIALIZING
Visa Cal: LOGGING_IN
Visa Cal: LOGIN_SUCCESS
Visa Cal: TERMINATING
Visa Cal: END_SCRAPING
success: false
error type: GENERIC
error: unknown transaction type 60
Error: unknown transaction type 60
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scrape-base.js:61:11)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scrape-task.js:82:33)
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scraping-main-menu.js:30:7)
(node:27638) UnhandledPromiseRejectionWarning: Error: unknown transaction type 60
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scrape-base.js:61:11)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scrape-task.js:82:33)
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scraping-main-menu.js:30:7)
(node:27638) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:27638) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I'm getting the following:
Run task scrapers
scraping Visa Cal
Visa Cal: START_SCRAPING
Visa Cal: INITIALIZING
Visa Cal: LOGGING_IN
Visa Cal: LOGIN_SUCCESS
Visa Cal: TERMINATING
Visa Cal: END_SCRAPING
success: false
error type: GENERIC
error: invalid json response body at https://cal4u.cal-online.co.il/Cal4U/CardsByAccounts reason: Unexpected end of JSON input
Error: invalid json response body at https://cal4u.cal-online.co.il/Cal4U/CardsByAccounts reason: Unexpected end of JSON input
at exports.default (/home/pi/Documents/exp/israeli-ynab-updater/src/scrape/scrape-base.js:61:11)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:4108) UnhandledPromiseRejectionWarning: Error: invalid json response body at https://cal4u.cal-online.co.il/Cal4U/CardsByAccounts reason: Unexpected end of JSON input
at exports.default (/home/pi/Documents/exp/israeli-ynab-updater/src/scrape/scrape-base.js:61:11)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:4108) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:4108) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Hi @eshaham!

A few weeks ago, while working with this lib to import my bank data to YNAB, I noticed that all the transactions that made with currencies other than ILS, are actually imported is if they were in ILS.

For example, this iTunes payment is 19.90$, not ₪19.90

I think having something like this will be nicer:

Notice how the outflow in ILS changes according to the date the transaction was made on.

I implemented support for currency conversion in my fork, It works like this:

• It lets you set up your app id for openexchangerates.org in npm run credentials
• It extracts the dates from all the transactions and fetches the historical rates from openexchangerates.org
• To keeps things free, it caches each request (we only got 1000 requests per month for free).
• It converts each foreign transaction to ILS and adds a memo with the original amount.

Is anyone aware of the problem of inquirer and windows?

For example:
303

Hi, after running npm install I getting found 186 vulnerabilities (1 low, 1 moderate, 184 high)

Let's say I created a Task, and I want to run it every morning.

I do not want to enter to the menu every time, but to run the Task directly, with one command.

scraping Bank Hapoalim - Be Online Account
Bank Hapoalim - Be Online Account: START_SCRAPING
Bank Hapoalim - Be Online Account: INITIALIZING
(node:25997) [DEP0106] DeprecationWarning: crypto.createDecipher is deprecated.
Bank Hapoalim - Be Online Account: TERMINATING
Bank Hapoalim - Be Online Account: END_SCRAPING
success: false
error type: GENERIC
error: waiting for selector "#inputSend" failed: timeout 30000ms exceeded
Error: waiting for selector "#inputSend" failed: timeout 30000ms exceeded
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scrape-base.js:61:11)
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scrape-individual.js:104:31)
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scraping-main-menu.js:27:7)

scraping Bank Hapoalim
Bank Hapoalim: START_SCRAPING
Bank Hapoalim: INITIALIZING
(node:25755) [DEP0106] DeprecationWarning: crypto.createDecipher is deprecated.
Bank Hapoalim: TERMINATING
Bank Hapoalim: END_SCRAPING
success: false
error type: GENERIC
error: waiting for selector "#inputSend" failed: timeout 30000ms exceeded
Error: waiting for selector "#inputSend" failed: timeout 30000ms exceeded
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scrape-base.js:61:11)
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scrape-individual.js:104:31)
at exports.default (/Users/dudut/Dudu/local code/israeli-ynab-updater/src/scrape/scraping-main-menu.js:27:7)

I'm using node 7.9.0 and running npm run credentials fails with the following error:

➜  israeli-ynab-updater git:(master) npm run credentials

> [email protected] credentials /Users/ifeins/Development/web/israeli-ynab-updater
> node_modules/babel-cli/bin/babel-node.js src -m credentials

/Users/ifeins/Development/web/israeli-ynab-updater/src/helpers/files.js:78
var writeFileAsync = _util2.default.promisify(_fs2.default.writeFile);
                                    ^

TypeError: _util2.default.promisify is not a function
    at Object.<anonymous> (/Users/ifeins/Development/web/israeli-ynab-updater/src/helpers/files.js:6:29)
    at Module._compile (module.js:571:32)
    at loader (/Users/ifeins/Development/web/israeli-ynab-updater/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/Users/ifeins/Development/web/israeli-ynab-updater/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/ifeins/Development/web/israeli-ynab-updater/src/scrape.js:6:1)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] credentials: `node_modules/babel-cli/bin/babel-node.js src -m credentials`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] credentials script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/ifeins/.npm/_logs/2018-01-23T14_44_50_256Z-debug.log

This doesn't happen on node 8.2.1.

So either it should be fixed or that the readme files should state that it needs node 8 to use.

Some credit card sites allow users to login with a code sent to the mobile phone.
Some users (not me :)), don't even have a "regular" password.

Was there any thought in adding support for this?
Basically the experience can be where you run npm start followed by selecting 2FA login option and if the scraper supports it then it will invoke the correct bank/credit API for 2FA login and once the user gets the SMS he can fill the code in the command line and from there the actual scraping will commence.

I tried running the visa cal scraper (site: https://www.cal-online.co.il/) and I get the following error:

scraping Visa Cal
Visa Cal: START_SCRAPING
Visa Cal: INITIALIZING
Visa Cal: LOGGING_IN
Visa Cal: TERMINATING
Visa Cal: END_SCRAPING
success: false
error type: generic
error: invalid json response body at https://restservices.cal-online.co.il/Cal4U/CalAuthenticator reason: Unexpected token < in JSON at position 0
Error: invalid json response body at https://restservices.cal-online.co.il/Cal4U/CalAuthenticator reason: Unexpected token < in JSON at position 0
at /Users/bsorek/Dev/git/personal/bankapp/israeli-ynab-updater/src/scrape/scrape-base.js:62:11
at Generator.next ()
at step (/Users/bsorek/Dev/git/personal/bankapp/israeli-ynab-updater/src/scrape/scrape-base.js:47:191)
at /Users/bsorek/Dev/git/personal/bankapp/israeli-ynab-updater/src/scrape/scrape-base.js:47:361
at process._tickCallback (internal/process/next_tick.js:68:7)

Hello @eshaham. As a nYnaber for a year and a half I was using Finanda import module to scrape all the sites. As they decided to end the life of their desktop product I was looking for alternatives and found your two awesome repositories.

After playing with this library I think adding the following will contribute to its' daily flow:

1. extend npm run credentials to manage multiple account together (allowing the user to view list of configured accounts, modify/delete existing, add new ones) and name them as you wish, ie 'home accounts' or 'work accounts'
2. scrape a group of accounts and export a joined csv file to be easily import them together to nYnab.
3. change the layout of the exported csv to match the fields recommended by nYnab: account, date, payee, inflow, outflow, memo.

I think having supporting multiple accounts will reduce the number of key strokes significantly.

I would appreciate your insights :)