[INTEGRATION] Improve Endpoint configuration about superduperdb HOT 1 CLOSED

here is the job: https://github.com/SuperDuperDB/superduperdb/actions/runs/7906999810/job/21583069571

superduperdb.base.exceptions.ServiceRequestException: Server error at cdc with 400 :: {'error': 'Config is not match'}

The problem is that conftest.py effectively ignores the env variables, resulting into different configurations from CDC, Vector-search, ray-workers etc.
There are two reasons for that:

1. It reads non-standard env variables like SUPERDUPER_MONGO_URI.
2. Has hardcoded values.

Here are some examples from conftest.py

    data_backend = 'mongodb://superduper:superduper@mongodb:27017/test_db'
    data_backend = os.environ.get('SUPERDUPER_MONGO_URI', data_backend)
    db_name = data_backend.split('/')[-1]

    artifact_store = 'filesystem:///tmp/artifacts'
    monkeypatch.setattr(CFG, 'artifact_store', artifact_store)

Standard and Non-Standard Variables
The standard variables are only those defined in the Config (e.g SUPERDUPER_DATA_BACKEND ). The fact that a variable begins with SUPERDUPER does not make it standard. (e.g SUPERDUPER_MONGO_URI)

As Kartik said on a previous iteration, the reason for having non-standard variables is that we may want to test multiple databases (e.g, mongo, sql, etc).
So we cannot simply provide a SUPERDUPER_DATA_BACKEND .

I get the problem, but we cannot simply hardcode paths because we lose portability.

Possible Solution

Use SUPERDUPERDB_TEST_CONFIGS that will as input a JSON of different Datalayer configuration we want to test.

For example:

export SUPERDUPERDB_TEST_CONFIGS='[
  {
    "DATA_BACKEND": "mongodb://...",
    "COMPUTE_URI": "ray://localhost...",
    "ARTIFACT_STORES": "/artifacts"
  },
  {
    "DATA_BACKEND": "sql://...",
    "COMPUTE_URI": "ray://localhost...",
    "ARTIFACT_STORES": "/whatever"
  }
]'

from superduperdb.