This has not been tested but we should have the ability to implement a selection of stocks universe selection strategy as well.
Most probably this should be defined by the PARAMETERS of the strategy by setting the rules of the selection.
Then in the security changed and removed we should handle the behaviour of removing and adding full securities.

I tried to set this on the security but the live algo does not work anymore if i do. Figure out a better way to handle this. The issue happened on a 1M paper account on IBKR while existing positions for 60x CCS and 70xPCS SPX were open. This should not require that much margin by any means as they are spreads.
Error is:

Order Error: ids: [213,214], Insufficient buying power to complete orders (Value:[-1025,275]), Reason: Id: 213, Initial Margin: -806977.5, Free Margin: 0.

the last merge, when i run the code twice, results in an error, because the load_positions tried to access the strategy Alpha.Order, you can replicate the error by running the main SPXic algorithm twice.
I ran it from the cloud

When you set the useSlice:False, then the provideroptioncontract is not able to handle greeks. Following error is an example:
Runtime Error: 'ProviderOptionContract' object has no attribute 'greeks'
at delta
return contract.greeks.delta
^^^^^^^^^^^^^^^
in ContractUtils.py: line 200
at
log_message += f"Delta: {[round(self.contractUtils.delta(c), 2) for c in contracts]}"
^^^^^^^^^^^^^^^^^^^^^^^^^^^
in LimitOrderHandlerWithCombo.py: line 278
at logOrderExecution
log_message += f"Delta: {[round(self.contractUtils.delta(c), 2) for c in contracts]}"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
in LimitOrderHandlerWithCombo.py: line 278
at makeLimitOrder
self.logOrderExecution(position, order, newLimitPrice)
in LimitOrderHandlerWithCombo.py: line 128
at call
self.makeLimitOrder(position, order)
in LimitOrderHandlerWithCombo.py: line 73
at Execute
self.limitOrderHandler.call(position, order)
in Base.py: line 99 (Open Stack Trace)

It seems like this PR QuantConnect/Lean#8266 will introduce a method onError that we can use to catch these errors and cancel orders.

We need to keep an eye on it.

This error showed a couple of times but did not break the overall flow it just restarted the algo.

Runtime Error: Trying to dynamically access a method that does not exist throws a TypeError exception. To prevent the exception, ensure each parameter type matches those required by the 'QuantConnect.Symbol'>) method. Please checkout the API documentation. at optionChainProviderFilter contract = OptionContract(symbol, symbol.Underlying) 

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
in DataHandler.py: line 113 at getOptionContracts contracts = self.optionChainProviderFilter(symbols, -self.strategy.nStrikesLeft, self.strategy.nStrikesRight, minDte, maxDte) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
in DataHandler.py: line 159 at Call chain = self.base.dataHandler.getOptionContracts(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
in Scanner.py: line 34 at update filteredChain, lastClosedOrderTag = Scanner(self.context, self).Call(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
in Base.py: line 227 Stack Trace: Trying to dynamically access a method that does not exist throws a TypeError exception. To prevent the exception, ensure each parameter type matches those required by the 'QuantConnect.Symbol'>) method. Please checkout the API documentation. at optionChainProviderFilter contract = OptionContract(symbol, symbol.Underlying) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
in DataHandler.py: line 113 at getOptionContracts contracts = self.optionChainProviderFilter(symbols, -self.strategy.nStrikesLeft, self.strategy.nStrikesRight, minDte, maxDte) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
in DataHandler.py: line 159 at Call chain = self.base.dataHandler.getOptionContracts(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
in Scanner.py: line 34 at update filteredChain, lastClosedOrderTag = Scanner(self.context, self).Call(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
in Base.py: line 227

Example of the new feature added by Lean/QC team and algo using it:

class OptionChainFullDataRegressionAlgorithm(QCAlgorithm):

    def initialize(self):
        self.set_start_date(2015, 12, 24)
        self.set_end_date(2015, 12, 24)
        self.set_cash(100000)

        goog = self.add_equity("GOOG").symbol

        # Get contracts expiring within 10 days, with an implied volatility greater than 0.5 and a delta less than 0.5
        contracts = [
            contract_data
            for contract_data in self.option_chain(goog)
            if contract_data.id.date - self.time <= timedelta(days=10) and contract_data.implied_volatility > 0.5 and contract_data.greeks.delta < 0.5
        ]
        # Get the contract with the latest expiration date
        self._option_contract = sorted(contracts, key=lambda x: x.id.date, reverse=True)[0]

        self.add_option_contract(self._option_contract)

    def on_data(self, data):
        # Do some trading with the selected contract for sample purposes
        if not self.portfolio.invested:
            self.market_order(self._option_contract, 1)
        else:
            self.liquidate()

This should help us fix this error:

Runtime Error:

During the algorithm initialization, the following exception has occurred: Error getting open orders from brokerage: An existing position or open order for an unsupported security type was found: 0 . Please manually close the position or cancel the order before restarting the algorithm.An existing position or open order for an unsupported security type was found: 0 . Please manually close the position or cancel the order before restarting the algorithm. in BrokerageSetupHandler.cs:line 476 An existing position or open order for an unsupported security type was found: 0 . Please manually close the position or cancel the order before restarting the algorithm.

The alpha base class builds out the working order from the Order class but we should just return a WorkingOrder instance from the Order class and just append it to the self.context.workingOrders array in the Alpha.Base class so we can have the flexibilty of adding extra data to it.

Example 28-34 dte returns the following expiry dates on 1st of August 2024:

DEBUG -> Scanner.Call: Expiry List: {datetime.date(2024, 8, 1): [datetime.datetime(2024, 9, 4, 0, 0), datetime.datetime(2024, 9, 3, 0, 0), datetime.datetime(2024, 8, 30, 0, 0), datetime.datetime(2024, 8, 29, 0, 0), datetime.datetime(2024, 8, 28, 0, 0)]}

I didn't investigate but the underlyingPriceAtOpen was returning underlyingprice and underlyingPriceAtClose wasn't so there must be a little bug, I noticed this when I was trying to get the price in the assignment function handleassignment through the assignedPosition

There is this increasing slowness when backtesting where the processor stays at 160% and not that much memory happening but this is well documented here: https://www.quantconnect.com/forum/discussion/10071/increasing-slowness-with-large-universe-option-backtests/p1

This issue should be closed when a solution is found in this repo or by fixing it in lean.

Add min_call_strike, max_call_strike, min_put_strike, max_put_strike to Alpha for filtering and update the DataHandler to use the optionChainProviderFilter method directly instead of going through the chain and universe as it does now using the getOptionContracts method.

Any approach to using this should be taken to Scanner.py as that is what is caling the getOptionContracts method.

Right now we have a smartpricing execution approach to limit orders. Based on the approach that option alpha has here: https://optionalpha.com/help/smartpricing

Add a new parameter to turn that off in case its not needed and make it the default.

Some of the logs have been added already that would result in a solution for this but so far this has not been solved.
The issue seems to happen later in the day on SPXic strat.

Here are the logs:

This is the runtime error message:

2024-06-12 20:05:00  DEBUG -> SPXExecutionModel.Execute: SPXExecutionModel -> Execute -> checkOpenPositions
2024-06-12 20:05:01 Runtime Error: (datetime.date(2024, 6, 12),)
2024-06-12 20:05:01 Runtime Error: (datetime.date(2024, 6, 12),): StackTrace: (datetime.date(2024, 6, 12),)

There is an issue with the ComboLimitOrder execution on the lean framework. It seems that it does not respect the limit price so i had to change the code in LimitOrderHandler.py to use ComboLegLimitOrder.
The problem with using this approach is that when i have to update the price of each individual legs and make adjusts based on that and also i have to cancel the order and submit it again to make sure that as i update one of the leg prices the execution does not happen at a bad price.

Here is what i found on the forum mentioning that the best option is to use ComboLegLimitOrder as the others are wrappers around MarketOrders: https://www.quantconnect.com/forum/discussion/14896/limit-orders-on-multi-leg-option-strategies-best-method/

It should not be that impossible to stub the QC lean framework and be able to add tests to the python code.
This would make sure any changes we introduce do not degrade the existing code.

it seems the structure is similar for the future option, we need to still use the canonical symbol, and see whether the slice is going to provide data for that.

https://www.quantconnect.com/docs/v2/writing-algorithms/securities/asset-classes/future-options/handling-data

example error

Runtime Error: 'ProviderOptionContract' object has no attribute 'implied_volatility'
at getSpread
iv_short_leg = sorted_contracts[i].implied_volatility
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
in OrderBuilder.py: line 387
at getSpreadOrder
legs = self.strategyBuilder.getSpread(contracts, type, strike = strike, delta = delta, wingSize = wingSize, fromPrice = fromPrice, toPrice = toPrice, premiumOrder = premiumOrder)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
in Order.py: line 688
at getOrder
call = self.order.getSpreadOrder(
^^^^^^^^^^^^^^^^^^^^^^^^^^
in SPXic.py: line 82
at CreateInsights
order = self.getOrder(chain, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^
in Base.py: line 317
at update
insights = self.CreateInsights(filteredChain, lastClosedOrderTag, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
in Base.py: line 284

Right now the code does not check for existing contracts/positions so you could open multiple same day positions with the same contract and break the whole strategy.

In alpha/base.py have a parameter defined that by default checks for existing positions and if any new order is tried for the same contract it should not be placed if it breaks existing positions.

Should be simple enough to add.

There is a posibility of running out of memory so we should save the self.context.allPositions into cache at the end of day. All the other arrays should clear out as the algo runs.

We should also just save the positions that are closed in there.

Same thing we should do for Performance logic data.