Tag Equals and StartsLike fails to return results about heroic HOT 10 CLOSED

Hey,

Can you paste or describe the configuration you are using and which commit of Heroic you are running?

from heroic.

I am using the latest master branch. Here is my yaml

port: 8080

cluster:
protocols:
- type: nativerpc
host: localhost
port: 1394

discovery:
type: static
nodes:
- "nativerpc://127.0.0.1:1394"

metrics:
backends:
- type: datastax
seeds:
- cassandra

metadata:
backends:
- type: elasticsearch
connection:
clusterName: BP2
type: single
pattern: metadata-%s
interval: 1w
seeds:
- elasticsearchclient

suggest:
backends:
- type: elasticsearch
connection:
clusterName: BP2
type: single
pattern: metadata-%s
interval: 1w
seeds:
- elasticsearchclient

consumers:

• type: kafka
  schema: com.spotify.heroic.consumer.schemas.Spotify100
  topics:
  • "metrics"
    config:
    group.id: heroic-consumer
    zookeeper.connect: zookeeper
    auto.offset.reset: smallest
    auto.commit.enable: true

from heroic.

Hey,

With the configuraiton you gave me, I get an error when trying to start Heroic.

An Elasticsearch configuration that does what you want should look like this:

metadata:                                                                                                                                                     
  backends:                                                                                                                                                   
    - type: elasticsearch                                                                                                                                     
      connection:                                                                                                                                             
        clusterName: BP2                                                                                                                                      
        seeds:                                                                                                                                                
          - elasticsearchclient                                                                                                                               
        index:                                                                                                                                                
          type: rotating                                                                                                                                      
          pattern: metadata-%s                                                                                                                                
          interval: 1w                                                                                                                                        

suggest:                                                                                                                                                      
  backends:                                                                                                                                                   
    - type: elasticsearch                                                                                                                                     
      connection:                                                                                                                                             
        clusterName: BP2                                                                                                                                      
        seeds:                                                                                                                                                
          - elasticsearchclient                                                                                                                               
        index:                                                                                                                                                
          type: rotating                                                                                                                                      
          pattern: metadata-%s                                                                                                                                
          interval: 1w  

I noticed now that there is a typo in the documentation for the type field.
I've updated the documentation in 71f6729.

I hope this helps!

from heroic.

We tried this and we get the same non-results. There are no error, just no results in the json result tag.

from heroic.

Because you are not seeing the error I did when trying to load the configuration. I'm questioning if you are loading the configuration at all.

Could you tell me how you run Heroic, and what the initial output is?

from heroic.

We don't know about elasticsearch, but we know that it can write to cassandra and retrieve by a key (see first comment above). So to us, this indicated that the config is being used.

We have all the components docker-fied and they all run in a solution. Here is the output of heroic, docker image, run by hand. dump.txt

and here is the yaml (as text) heroic.yaml.txt

from heroic.

So this could be an interoperability issue with the Elasticsearch client library we are using.

I've added an integration test that checks the the operation that you described, if you could run the following test suite it would be helpful:

$ mvn -P integration-tests -D it.elasticsearch.remote=true -D it.elasticsearch.seed=localhost clean verify

This will create a fair bit of clutter on the Cluster if its a permanent one, so if its persistent you might want to delete the templates and indexes created. They all have the pattern heroic-it-*.

$ curl -X DELETE http://localhost:9200/heroic-it-\*
$ curl -X DELETE http://localhost:9200/_template/heroic-it-\*

from heroic.

Hey,

So I think I've figured out what's going on. If an Elasticsearch cluster is not configured, filters partially work. But the starts with filter does not.

I believe you have indexes which are using the wrong mapping, configuring a cluster should add a template that you can look for:

$ curl http://localhost:9200/_template

If there is not template, you can add it by configuring the cluster:

$ tools/heroic-shell -C <config>
heroic> configure

You can make sure that this template has been applied to your indexes:

$ curl http://localhost:9200/heroic-\*/_mapping

If the mapping is wrong, the only way to get it to apply is to drop the indexes. The mapping is persistent in the already created indexes.

$ curl -X DELETE http://localhost:9200/heroic-\*

I hope this helps.

from heroic.

Hello,

@udoprog with your help we figured out what the problem was on our side. We had the elasticsearch index configured with an incorrect pattern. As you can see above, we set our elasticsearch connections with an index pattern of metadata-%s per the instructions here. Once we changed those be equal to heroic-a-%s we were able to get correct results using the query/metrics API. We think it'll be good to update the documentation.

Thanks for your help!

from heroic.

So the index configuration need to be consistent between the configure step, and the eventual backend configuration. I'll try to clarify this at some point, but If you can suggest an edit that makes this clearer I'd be happy to look at it.

from heroic.