fetching all stocks about degiro-connector HOT 11 CLOSED

Hello there,

Pagination
In the ProductSearch endpoint, the pagination is handled with these two parameters :

• offset
• limit

This is how most Databases (SQL or NoSQL) handle the pagination : with parameters for the offset and limit.

Indices
The index parameter seems to be the stock index.

Examples of stock index :

• CAC 40
• Dow Jones Industrial Average
• Nasdaq Composite
• S&P 500

You can get all the available indices for each country in the ProductsConfig table.

As mentioned in the documentation : This table contains useful parameters to filter products. .

Hope that helps

from degiro-connector.

hi @Chavithra thanks for your answer

I thought as much and a I already tried that. For example I pulled the NASDAQ id from the productsconfig:

products_info = message_to_dict(trading_api.get_products_config())

then pulled out nasdaq:
{'country': 846.0, 'id': 663.0, 'exchange': 663.0, 'postfix': 'NASDAQ'},
and have the index 663

and then tried this code:

index = 0
products = []
while True:
    print(index)
    r = ProductSearch.RequestStocks(
        stock_country_id=846, #USA
        index_id=663,
        offset=index*100,
        is_in_us_green_list=True,
        limit=100,
        require_total=True,
        sort_columns="name",
        sort_types="asc",
    )
    res = message_to_dict(trading_api.product_search(request=r, raw=False))
    if len(res['products']) == 0:
        break
    else:
        index += 1
        products += res['products']
        print(len(res['products']))

but then I always get zero products?

I just tried to remove the whole index_id and only use offset and then I seem to get loads of stocks so thats progress. looking back at your code after a day makes wonders haha. but I still would like to filter beforehand.

from degiro-connector.

extra remark, I see that the column 'productBitTypes' does not always contain US_RAS_GREEN_LIST even though I put "is_in_us_green_list=True" in there. does that mean that filter is broken? or does that column mean something else?

from degiro-connector.

Congrats
It's good that you understood for the offset : there is progress indeed.

More on indices
Looks like the issue you encounter with the index_id is not on you.

With your feedback and by doing some tests I found out that the ProductSearch endpoint uses two parameters to filter on indices (what is called indices on the website).

Here are this two parameters :

• index_id
• exchange_id

So according to what you are looking for, you need to set the right parameter in the ProductSearch.RequestStock.

New release
A new version of this library was released : you can now use exchange_id inside ProductSearch.RequestStock.

ProductsConfig
To find out which index_id and exchange_id are available according to a country you can use the ProductsConfig table.

Example for the US :

{
    ...
    "countries": [
        ...
        {
            "id": 846,
            "name": "US",
            "region": 2,
            "translation": "list.country.846"
        },
        ...
    ],
    ...
    "stockCountries": [
        ...
        {
            "country": 846,
            "exchanges": [
                650,
                1004,
                663
            ],
            "id": 846,
            "indices": [
                12,
                122001,
                13,
                14
            ]
        },
        ...
    ],
    ...
    "exchanges": [
        ...
        {
            "city": "New York",
            "code": "XNAS",
            "country": "US",
            "hiqAbbr": "NDQ",
            "id": 663,
            "micCode": "XNAS",
            "name": "NASDAQ"
        },
        ...
    ],
    ...
    "indices": [
        {
            "id": 122001,
            "name": "NASDAQ 100",
            "productId": 12153880
        },
        ...
    ],
    ...
}

from degiro-connector.

extra remark, I see that the column 'productBitTypes' does not always contain US_RAS_GREEN_LIST even though I put "is_in_us_green_list=True" in there. does that mean that filter is broken? or does that column mean something else?

Do you have a specific example ?

from degiro-connector.

hi @Chavithra Thank you for the quick new release. However the new version throws an error for me. I have looked up the ids in product config like so:

# find the stock exchange
[x for x in product_info['exchanges'] if x['name'] == 'NASDAQ'][0]
# this is for specific subsets of an index
[x for x in product_info['indices'] if x['name'] == 'NASDAQ 100'][0]
# find the country id
[x for x in product_info['countries'] if x['name'] == 'US'][0]

which returns the following three dictionaries:

{'country': 'US', 'city': 'New York', 'name': 'NASDAQ', 'id': 663.0, 'code': 'XNAS', 'micCode': 'XNAS', 'hiqAbbr': 'NDQ'}
{'productId': 12153880.0, 'name': 'NASDAQ 100', 'id': 122001.0}
{'name': 'US', 'region': 2.0, 'id': 846.0, 'translation': 'list.country.846'}

then I use the follow code to fetch the stocks:

r = ProductSearch.RequestStocks(
    stock_country_id=846, #USA
    index_id=663, #NASDAQ
    exchange_id=122001, #NASDAQ 100
    offset=0,
    is_in_us_green_list=True,
    limit=100,
    require_total=True,
    sort_columns="name",
    sort_types="asc",
)
res = message_to_dict(trading_api.product_search(request=r, raw=False))

but this throws the following error:

<Response [400]>
400 Client Error: Bad Request for url: https://trader.degiro.nl/product_search/secure/v5/stocks?isInUsGreenList=True&indexId=663&exchangeId=122001&stockCountryId=846&limit=100&requireTotal=True&sortColumns=name&sortTypes=asc&intAccount=<snip>&sessionId=<snip>BC3FF02A3E564EE06E31AFAD.prod_a_168_5

AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_12571/2263861331.py in <module>
     11     sort_types="asc",
     12 )
---> 13 res = message_to_dict(trading_api.product_search(request=r, raw=False))

~/pycharmprojects/finance/venv/lib/python3.9/site-packages/degiro_connector/core/helpers/pb_handler.py in message_to_dict(message)
     10 
     11 def message_to_dict(message: Message) -> dict:
---> 12     return json_format.MessageToDict(
     13         message=message,
     14         including_default_value_fields=True,

~/pycharmprojects/finance/venv/lib/python3.9/site-packages/google/protobuf/json_format.py in MessageToDict(message, including_default_value_fields, preserving_proto_field_name, use_integers_for_enums, descriptor_pool, float_precision)
    163       float_precision=float_precision)
    164   # pylint: disable=protected-access
--> 165   return printer._MessageToJsonObject(message)
    166 
    167 

~/pycharmprojects/finance/venv/lib/python3.9/site-packages/google/protobuf/json_format.py in _MessageToJsonObject(self, message)
    197   def _MessageToJsonObject(self, message):
    198     """Converts message to an object according to Proto3 JSON Specification."""
--> 199     message_descriptor = message.DESCRIPTOR
    200     full_name = message_descriptor.full_name
    201     if _IsWrapperMessage(message_descriptor):

AttributeError: 'NoneType' object has no attribute 'DESCRIPTOR'

If I go to the link itself then this is the specific error message thrown by the api:

{"errors":[{"text":"No corresponding model found"}]}

from degiro-connector.

extra remark, I see that the column 'productBitTypes' does not always contain US_RAS_GREEN_LIST even though I put "is_in_us_green_list=True" in there. does that mean that filter is broken? or does that column mean something else?

Do you have a specific example ?

the same code as above. but I think its better to first focus on the error that I get now with the new version

from degiro-connector.

Error
You haved used this code to get the exchange_id :

# find the stock exchange
[x for x in product_info['exchanges'] if x['name'] == 'NASDAQ'][0]

Result:

{'country': 'US', 'city': 'New York', 'name': 'NASDAQ', 'id': 663.0, 'code': 'XNAS', 'micCode': 'XNAS', 'hiqAbbr': 'NDQ'}

So :

exchange_id = 663

But in your ProductSearch.RequestStocks you have set :

exchange_id = 122001

Dict
By the way you can use the raw option to get the result as a dict.

trading_api.product_search(request=request, raw=True)

from degiro-connector.

hi @Chavithra thank you that was indeed my mistake I swapped the index and exchange ids, oops.

my code for only nasdaq100 is now:

r = ProductSearch.RequestStocks(
    stock_country_id=846, #USA
    exchange_id=663, #NASDAQ
    index_id=122001, #NASDAQ 100
    is_in_us_green_list=True,
    limit=100,
    require_total=True,
    sort_columns="name",
    sort_types="asc",
)
res = message_to_dict(trading_api.product_search(request=r, raw=False))

my code to fetch ALL nasdaq stocks is:

index = 0
products = []
while True:
    print(index)
    r = ProductSearch.RequestStocks(
        stock_country_id=846, #USA
        exchange_id=663, #NASDAQ
        offset=index*100,
        is_in_us_green_list=True,
        limit=100,
        require_total=True,
        sort_columns="name",
        sort_types="asc",
    )
    res = message_to_dict(trading_api.product_search(request=r, raw=False))
    if len(res['products']) == 0:
        break
    else:
        index += 1
        products += res['products']

some observations:

• using raw=True like you say results in a crash with message_to_dict, I guess that makes sense. I use that function since the raw dictionary has many superfluous keys.
• offset is ignored when getting a result that is less then 100 results. That makes kinda sense but if you dont know the number of results on beforehand and you just keep increasing the offset you will keep getting the same stocks over and over again. something to keep in mind

an error I still see:
when fetching all stocks with is_in_us_green_list=True I still see many stocks that have property productBitTypes as an empty list instead of [US_RAS_GREEN_LIST]. to me that seems this property is ignored?

from degiro-connector.

Total
If you look at the result of product_search method, you can see that the first parameter the API returns is the total :

{
    'total': 84,
    'offset': 0,
    'products': [{'id': '332109'...}]
}

productBitTypes
No idea what this productBitTypes attribute is, you will have to find it by yourself.

from degiro-connector.

okay thanks for the help!

from degiro-connector.