indeedeng / iql Goto Github PK

NOTE: Indeed has discontinued supporting this project. Archiving will take place on 8/16/21.
If you are interested in taking over as the Maintainer, please contact Indeed at [email protected]

In reference to the "dimensions" code in ImhotepMetadataCache, Vladimir Markin writes:

dimensions.desc has 2 purposes:

1. Mark which fields in Lucene indexes contain numbers so that they can be treated as int fields. That's not needed since OSS IQL only supports Flamdex indexes.
2. Defining composite metrics like sjc=tsjc+bsjc. Composite metrics can easily lead to confusion when querying since they are currently shown side by side with real fields.

We should document this in the code and discourage the use of the dimensions functionality, which is currently undocumented elsewhere anyway.

See this discussion: https://groups.google.com/forum/#!topic/indeedeng-imhotep-users/SmyzrSViPzk

Upgrade com.fasterxml.jackson.core:jackson-databind to version 2.8.11.1 or later, due to security vulnerabilities.

Being able to share a shortlink to an IQL query is very powerful. At Indeed, we use such shortlinks all the time in email, IM, JIRA comments, etc.

MVP for v1 (S3-based):

• Support generating a shortlink of the form https://hostname/q/XXXXXX
• The shortlink code is generated, then regenerated if already in use (and not the same query). Then an S3 object is created with the code as the name, containing the text of the query, and stored under "shortlinks/" in an S3 bucket. (TBD: should the links be namespaced into a hierarchy to avoid large numbers of objects in a folder?)
  • TBD details of code generation: random?
  • TBD should there be prevention of creating multiple shortlinks for the same query? Maybe for a later version using a hash of the URL to a lookup table
• The lookup process is straightforward, and does a 301 redirect to the /iql URL.
• MVP contains no ability to list, edit, or delete shortlinks
• MVP contains no concept of shortlink ownership

When IQL was open-sourced, the front-end TypeScript application for the IQL web app was not. Instead, a compiled version of oss branch was included in the repo. The main reason for this is simply links to Indeed-internal things that are in the internal version. We have not yet revisited this problem.

In the meantime, lots of features and minor UI updates have happened on the internal version. This issue is to sync all those changes up and refresh the compiled app in this repo.

Creating new indexes does not populate the index selector control (the "from" input).

To reproduce:

• In IUploader, create a new dataset -- fooindextest
• Upload a simple TSV, the single-day NASA weblog for example
• Refresh the page around every 30 seconds until the UI indicates the TSV has been indexed
• In IQL, click the "from" input...

Documentation indicates you should "Allow a minute or two for your dataset to be available in the IQL web client.", but the dataset will never show up in the dropdown.

If you open an incognito window and navigate to the IQL webapp, you will see the dataset listed as an option.

It appears like the local storage value for {{iql_prod_getDatasets__cache}} never gets refreshed, even after hard reloads.

From imhotep created by ThomasBergman1 : indeedeng/imhotep#16

I saw an error like this when doing a particularly heavy query.

Query failed:
Looks like the IQL server got overloaded and is restarting.
Please take a look at your query and consider if it is too heavy.
Wiki:
https://wiki.indeed.com/display/INTEL/_Performance+Considerations+for+IQL+usage

Upgrade org.apache.lucene:lucene-core to version 7.1.0 or later, due to security vulnerabilities.

Need to incorporate how to provide field documentation to IQL webapp.

The current way (until we open source our metadata service) to set descriptions is to put it in a directory on the machine that runs IQL. The directory is set with ramses.metadata.dir config property and is typically /var/data/iql/ramses_metdata/.

Each index to be described has to have a subdirectory under that path with name matching the index name. The descriptions will go in a file called suggestions.xml under that subdirectory. Example:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN 2.0//EN" "http://www.springframework.org/dtd/spring-beans-2.0.dtd">

<beans default-lazy-init="true">
    <bean id="suggestionMap" class="java.util.HashMap">
        <constructor-arg>
            <map>
                <entry key="imhotepFieldName" value="This describes what is in the field" />
                <entry key="imhotepAnotherFieldName" value="This describes what is in another field" />
            </map>
        </constructor-arg>
    </bean>
</beans>

The descriptions need to be put as lines. The rest of the file will stay the same.

Other examples (for our public Imhotep datasets) are in the attached zip file.

suggestions.zip