Allow for registering feature that doesn't have warehouse store about feast HOT 15 CLOSED

So the question here is

• Do we allow toggling of the warehouse storage option at the feature level
• or at the system level (Feast deployment)
• or both?

I think we should have the option to both disable warehouse storage on specific features, as well as deploy a Feast instance without a warehouse (serving only)

from feast.

My pull request fixes a typo.. But basically ingestion already supports setting a "no operation" store, for either serving or ingestion. The core api would still need to implement sending it of course.

from feast.

My pull request fixes a typo.. But basically ingestion already supports setting a "no operation" store, for either serving or ingestion. The core api would still need to implement sending it of course.

So the next step is a PR which adds this functionality to core?

@zhilingc, possible to have a look at this?

from feast.

I'm also working on a PR to make ingestion accept an empty storage spec. Though in reality, that will interpreted as a StorageSpec.defaultInstance(). With empty string as the storeid. Ingestion will then treat this as a noop store.

from feast.

Ok. You might want to remove the validation check that makes declaring a store mandatory when registering a feature as well.

from feast.

Aren't you worried people will register features and they will just all go to noop when they leave it out though? We do need to implement inheriting a system default first.

And then a user needs to be able to override the default to a noop.

Which now that I think about might be a pain. As if they set the store id to an empty string, it will look the same as the default instance, so how will you know when to override it with the default or not?

Remember protos not having optional values means that there's no such thing as a null store id.

from feast.

Aren't you worried people will register features and they will just all go to noop when they leave it out though? We do need to implement inheriting a system default first.

And then a user needs to be able to override the default to a noop.

Which now that I think about might be a pain. As if they set the store id to an empty string, it will look the same as the default instance, so how will you know when to override it with the default or not?

Remember protos not having optional values means that there's no such thing as a null store id.

Isn't a better approach simply to have an option to switch off either store? That we we don't have to look at the value and try to guess whether it should be on or off. I think the default should be inheritance.

from feast.

When is the default declared? Should it be in the configuration when deploying core Feast? Or is it a necessary step that has to be done by the user before registering features (but after deployment)?

from feast.

When is the default declared? Should it be in the configuration when deploying core Feast? Or is it a necessary step that has to be done by the user before registering features (but after deployment)?

I think it should be done during initial installation. That way the user could theoretically have all data storage abstracted away from them.

from feast.

@zhilingc yes it should be a system default. So it should come with the system.

Users shouldn't even be creating new stores? I think we made store specs creatable accidentally to be honest, because all the others are. They should probably only ever be created by the person who deploys a feast instance.

from feast.

Can we close this issue now?

from feast.

Closing, but reopen if I missed something.

from feast.

Reopening the issue, core is still checking for presence of warehouse store. I am crafting a PR

from feast.

I am still having this error when trying to ingest data for feature having no warehouse store.
The feature spec looks like as follow:

id: "myentity.minute.feature_1"
name: "feature_1"
owner: "[email protected]"
description: "Test feature"
uri: "NA"
granularity: "MINUTE"
valueType: "DOUBLE"
entity: "myentity"
dataStores: 
  serving: 
    id: "REDIS1"

Exception:

Optional[Exception in thread "main" java.lang.IllegalArgumentException: no write transforms found
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
        at feast.ingestion.transform.SplitOutputByStore.expand(SplitOutputByStore.java:57)
        at feast.ingestion.transform.SplitOutputByStore.expand(SplitOutputByStore.java:45)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
        at feast.ingestion.values.PFeatureRows.apply(PFeatureRows.java:109)
        at feast.ingestion.transform.WarehouseStoreTransform.expand(WarehouseStoreTransform.java:45)
        at feast.ingestion.transform.WarehouseStoreTransform.expand(WarehouseStoreTransform.java:30)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
        at feast.ingestion.values.PFeatureRows.apply(PFeatureRows.java:109)
        at feast.ingestion.ImportJob.expand(ImportJob.java:184)
        at feast.ingestion.ImportJob.mainWithResult(ImportJob.java:126)
        at feast.ingestion.ImportJob.main(ImportJob.java:110)]

from feast.

Apparently the ingestion job succeed but all feature row didn't pass validation

from feast.