valueraider / yfinance-cache Goto Github PK

bulk download failure occurred when setting "threads=True" default

see my code below:
def test_yfc_download():
tickers = ["INTC", "MSFT"]

  df = yfc.download(tickers, 
              interval='1h', 
              period=TIME_PERIOD_60M, 
              max_age='4h', 
              group_by='ticker', 
              # threads=False
              )


  print(df['MSFT'][['High', 'Low', 'Close', 'Volume']])

  return

if __name__ == '__main__':

  import yfinance_cache as yfc

  TIME_PERIOD_60M = '250d'
  test_yfc_download()`

The failure log is as below:

Process SpawnPoolWorker-3:
Traceback (most recent call last):
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
Traceback (most recent call last):
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\pool.py", line 114, in worker
task = get()
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\pool.py", line 114, in worker
task = get()
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\queues.py", line 367, in get
return ForkingPickler.loads(res)
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\queues.py", line 367, in get
return ForkingPickler.loads(res)
File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache_init.py", line 4, in
from .yfc_ticker import Ticker, verify_cached_tickers_prices
File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache_init.py", line 4, in
from .yfc_ticker import Ticker, verify_cached_tickers_prices
File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_ticker.py", line 7, in
from . import yfc_time as yfct
File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_ticker.py", line 7, in
from . import yfc_time as yfct
File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py", line 22, in
manager = Manager()
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\context.py", line 57, in Manager
m.start()
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\managers.py", line 554, in start
self._process.start()
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 118, in start assert not _current_process._config.get('daemon'),
File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py", line 22, in
manager = Manager()
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\context.py", line 57, in Manager
m.start()
AssertionError: daemonic processes are not allowed to have children
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\managers.py", line 554, in start
self._process.start()
File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 118, in start assert not _current_process._config.get('daemon'),
AssertionError: daemonic processes are not allowed to have children

I'm using Window 11 OS.

my Python version:
Python 3.9.18 (main, Sep 11 2023, 14:09:26) [MSC v.1916 64 bit (AMD64)] on win32

see dependences' version:
yfance_cache (0.4.7)
yfance (0.2.31)
pandas (2.1.1)
exchange-calendars (4.5)
scipy (1.10.1)
click (8.1.7)
numpy (1.26.0)
pyluach (2.2.0)
python-dateutil (2.8.2)
toolz (0.12.0)
tzdata (2023.3)
korean-lunar-calendar (0.3.1)
pytz (2023.3.post1)
requests (2.31.0)
multitaskg(0.0.11)
lxml (4.9.3)
appdirs (1.4.4)
frozendict(2.3.8)
peewee (3.16.3)
beautifulsoup4 (4.12.2)
html5lib (1.1)
colorama (0.4.4)
soupsieve>1.2 (2.5)
six>=1.9 (1.16.0)
webencodgs (0.5.1)
charset-normalizer (3.3.0)
idna(3.4)
urllib3 (1.26.17)
certifi (2023.7.22)

Tried to use yfc as plugin-replacement for yf. However, on one call I got a typeerror, apparently coming from yfc_prices_manager:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

** Stuff deleted due to personal information **
** Last call in my code:  parthist = ticker.history(period="7d", interval="1m") **

File [/usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py:232](https://untitled+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py:232), in Ticker.history(self, interval, max_age, period, start, end, prepost, actions, adjust_splits, adjust_divs, keepna, proxy, rounding, debug, quiet, trigger_at_market_close)
    [230](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py?line=229) hist = self._histories_manager.GetHistory(interval)
    [231](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py?line=230) if period is not None:
--> [232](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py?line=231)     h = hist.get(start=None, end=None, period=period, max_age=max_age, trigger_at_market_close=trigger_at_market_close, quiet=quiet)
    [233](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py?line=232) elif interday:
    [234](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py?line=233)     h = hist.get(start_d, end_d, period=None, max_age=max_age, trigger_at_market_close=trigger_at_market_close, quiet=quiet)

File [/usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py:609](https://untitled+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py:609), in PriceHistory.get(self, start, end, period, max_age, trigger_at_market_close, repair, prepost, adjust_splits, adjust_divs, quiet)
    [605](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py?line=604) f_nfinal = ~f_final
    [606](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py?line=605) # - also treat repaired data as non-final, if fetched near to interval timepoint
    [607](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py?line=606) #   because Yahoo might now have correct data
    [608](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py?line=607) #   TODO: test!
--> [609](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py?line=608) f_repair = (self.h["Repaired?"].to_numpy() & (self.h["FetchDate"] < (self.h.index + self.itd + td_7d)).to_numpy())
    [610](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py?line=609) f_nfinal = f_nfinal | f_repair
    [611](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py?line=610) for idx in np.where(f_nfinal)[0]:
    [612](file:///usr/local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py?line=611)     # repaired = False

TypeError: unsupported operand type(s) for &: 'float' and 'bool'

same code with same arguments runs for yfinance. But I can not exclude the possibility of incorrect handling, hence, could also be on my side, but I suppose incorrect arguments to an "&" should be addressed within a library.

Would be great if it could support AWS S3 (including Googles implementation) to support distributed cloud-based applications.

_

How do you know when your cache is out-of-date?
For instance - yfinance-cache was run on day n and the data wasn't in cache so the cache was filled-in.

Scenario 1:

Now it is day n+2 and during day n+1 yahoo finance updated its records such that a quarterly element was updated. you must cache that in as your cache is out of date.

Scenario 2:

Now it is day n+2 and during day n+1 yahoo finance did not update its records such that you can use your cache - but how do you know that without checking - rendering the cache obsolete (unless same operations per day or unless you allow some time to not be updated). I use caching in my stock scanner and screener (https://github.com/asafravid/sss) using json, but thats just for crash and continue or reruns - is that your intention with this caching tool?

Scenario 3:

As we saw in a few issues (and as I saw as well) - sometimes two different calls to yfinance can yield different data as yahoo picks the data randomly from several websites which may or may not be updated, yielding different (sometimes less updated by one quarter) results

Bare that in mind while you develop your tool, maybe add usecases in the Readme, etc.

Would be great if it could support memcache to support distributed cloud based applications.

import yfinance_cache as yfc
dat = yfc.Ticker('RCI-B.TO')
dat.get_release_dates(check=True)

Exception: RCI-B.TO: interval = 31 doesn't fit standard intervals

Because yfc stops requesting candle updates after market close, it excludes any aftermarket trading/changes. I am working on a fix.

Example: CNE.L: a stock split today, and the prices should all be stock-split-adjusted down to 200-230, but most are not. So this data is corrupt. Currently YFC can't check for this.

A solution is technically possible, and probably means another data column e.g. "IsCorrupt?" that triggers a future refresh.

Implement download() function to batch download many ticker price histories.

Should just need to make GetCalendarViaCache and SetExchangeTzName thread-safe with per-exchange mutexes.

exchange_calendars package doesn't support parameter 'cache'

Error:
Traceback (most recent call last):
File "/Users/mike/Projects/eiten/test_yfc.py", line 3, in
hist = msft.history(period="max")
File "/Users/mike/Projects/eiten/venv/lib/python3.9/site-packages/yfinance_cache/yfc_ticker.py", line 178, in history
sched = yfct.GetExchangeSchedule(exchange, d_now-(7*td_1d), d_now+td_1d)
File "/Users/mike/Projects/eiten/venv/lib/python3.9/site-packages/yfinance_cache/yfc_time.py", line 180, in GetExchangeSchedule
cal = GetCalendar(exchange)
File "/Users/mike/Projects/eiten/venv/lib/python3.9/site-packages/yfinance_cache/yfc_time.py", line 126, in GetCalendar
cal = xcal.get_calendar(cal_name, start=start, cache=True)
TypeError: get_calendar() got an unexpected keyword argument 'cache'

Sample code:
import yfinance_cache as yf
msft = yf.Ticker("MSFT")
hist = msft.history(period="max")
print(hist)

my test code is like this #43 (when setting threads=False)

the warning logs:

C: \miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py:482: FutureWarning: Series.getitem treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use ser.iloc[pos]
last_open_day = sched["open"][-1].date()

C: \miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_prices_manager.py:647: FutureWarning: Series.getitem treating keys as positions is deprecated.
In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use ser.iloc[pos]
fetch_dt = yfct.ConvertToDatetime(self.h["FetchDate"][idx], tz=tz_exchange)

C: \miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py:816: FutureWarning: Series.getitem treating keys as positions is deprecated.
In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use ser.iloc[pos]
if ts < sched["open"][i]:

C: \miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py:818: FutureWarning: Series.getitem treating keys as positions is deprecated.
In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use ser.iloc[pos]
return {"market_open": sched["open"][i].to_pydatetime().astimezone(tz), "market_close": sched["close"][i].to_pydatetime().astimezone(tz)}

C: \miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py:1087: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas.
Value 'DatetimeIndex(['2022-10-14 09:30:00-04:00', '2022-10-14 10:30:00-04:00',
'2022-10-14 11:30:00-04:00', '2022-10-14 12:30:00-04:00',
'2022-10-14 13:30:00-04:00', '2022-10-14 14:30:00-04:00',
'2022-10-14 15:30:00-04:00', '2022-10-17 09:30:00-04:00',
'2022-10-17 10:30:00-04:00', '2022-10-17 11:30:00-04:00',
...
'2023-10-12 13:30:00-04:00', '2023-10-12 14:30:00-04:00',
'2023-10-12 15:30:00-04:00', '2023-10-13 09:30:00-04:00',
'2023-10-13 10:30:00-04:00', '2023-10-13 11:30:00-04:00',
'2023-10-13 12:30:00-04:00', '2023-10-13 13:30:00-04:00',
'2023-10-13 14:30:00-04:00', '2023-10-13 15:30:00-04:00'],
dtype='datetime64[ns, America/New_York]', length=1751, freq=None)' has dtype incompatible with datetime64[ns], please explicitly cast to a compatible dtype first.
intervals.loc[intervals.index[f], "interval_open"] = tis.left[idx[f]].tz_convert(tz)

C: \miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py:1088: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas.
Value 'DatetimeIndex(['2022-10-14 10:30:00-04:00', '2022-10-14 11:30:00-04:00',
'2022-10-14 12:30:00-04:00', '2022-10-14 13:30:00-04:00',
'2022-10-14 14:30:00-04:00', '2022-10-14 15:30:00-04:00',
'2022-10-14 16:00:00-04:00', '2022-10-17 10:30:00-04:00',
'2022-10-17 11:30:00-04:00', '2022-10-17 12:30:00-04:00',
...
'2023-10-12 14:30:00-04:00', '2023-10-12 15:30:00-04:00',
'2023-10-12 16:00:00-04:00', '2023-10-13 10:30:00-04:00',
'2023-10-13 11:30:00-04:00', '2023-10-13 12:30:00-04:00',
'2023-10-13 13:30:00-04:00', '2023-10-13 14:30:00-04:00',
'2023-10-13 15:30:00-04:00', '2023-10-13 16:00:00-04:00'],
dtype='datetime64[ns, America/New_York]', length=1751, freq=None)' has dtype incompatible with datetime64[ns], please explicitly cast to a compatible dtype first.
intervals.loc[intervals.index[f], "interval_close"] = tis.right[idx[f]].tz_convert(tz)

C: \miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_prices_manager.py:805: FutureWarning: Series.getitem treating keys as positions is deprecated.
In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use ser.iloc[pos]
if (not h_intervals.empty) and isinstance(h_intervals["interval_open"][0], datetime):

C: \miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py:482: FutureWarning: Series.getitem treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use ser.iloc[pos]
last_open_day = sched["open"][-1].date()

I'm using Window 11 OS.
my Python version:
Python 3.9.18 (main, Sep 11 2023, 14:09:26) [MSC v.1916 64 bit (AMD64)] on win32

see dependences' version:
yfance_cache (0.4.7)
yfance (0.2.31)
pandas (2.1.1)
exchange-calendars (4.5)
scipy (1.10.1)
click (8.1.7)
numpy (1.26.0)
pyluach (2.2.0)
python-dateutil (2.8.2)
toolz (0.12.0)
tzdata (2023.3)
korean-lunar-calendar (0.3.1)
pytz (2023.3.post1)
requests (2.31.0)
multitaskg(0.0.11)
lxml (4.9.3)
appdirs (1.4.4)
frozendict(2.3.8)
peewee (3.16.3)
beautifulsoup4 (4.12.2)
html5lib (1.1)
colorama (0.4.4)
soupsieve>1.2 (2.5)
six>=1.9 (1.16.0)
webencodgs (0.5.1)
charset-normalizer (3.3.0)
idna(3.4)
urllib3 (1.26.17)
certifi (2023.7.22)

I have a simple application with fastapi with single endpoint it fetches historical data and return it in webpack format. I run it few days ago and didn't used it. Now I have tried to fetch data and got error below. Restart has cleared the issue. Have used the latest library installed from git.

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/var/mail/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/var/mail/.local/lib/python3.11/site-packages/msgpack_asgi/_middleware.py", line 29, in __call__
    await responder(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/msgpack_asgi/_middleware.py", line 72, in __call__
    await self.app(scope, self.receive_with_msgpack, self.send_with_msgpack)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/ESP-TickerGraph.py", line 21, in get_history
    hist = getHist(ticker, points)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/ESP-TickerGraph.py", line 28, in getHist
    hist = t.history(period=period, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py", line 233, in history
    h = hist.get(start=None, end=None, period=period, max_age=max_age, trigger_at_market_close=trigger_at_market_close, quiet=quiet)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 897, in get
    self._fetchAndAddRanges_sparse(pstr, ranges_to_fetch, prepost, debug_yf, quiet=quiet)
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 1233, in _fetchAndAddRanges_sparse
    h2 = self._fetchYfHistory(pstr, fetch_start, fetch_end, prepost, debug)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 2311, in _fetchYfHistory
    accept = click.confirm(msg, default=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/click/termui.py", line 235, in confirm
    raise Abort() from None
click.exceptions.Abort
esp-tickergraph_server-esp-tickergraph-1  |
Failed to map these Yahoo intervals to xcal: (tkr=FIVN, exchange=NGM, xcal=NASDAQ). Normally happens when 'exchange_calendars' is wrong so inform developers.
                               Close  Volume  Dividends  Stock Splits
Datetime
2024-04-03 15:30:00-04:00  60.505001   21274        0.0           0.0
Accept into cache anyway? [y/N]: INFO:     192.168.1.2:51284 - "GET /hist?ticker=FIVN&points=200 HTTP/1.0" 500 Internal Server Error

in yfinance, parameter “period” means the only "trading" days,
while in yfinance_cache, it means all the ordinary days (including weekends and holidays)

so, please you can check.

By the way, this is a great module offering cache capabilities, really appreciate that, keep going!