is there hava a Sql Server connector?

#mysql table s3.source works
curl -X GET --header 'Accept: application/json' 'http://dns:8088/metacat-1.1.1/mds/v1/catalog/mysql-56-db/database/s3/table/source?includeInfo=true&includeDefinitionMetadata=true&includeDataMetadata=true'

#mysql table metastore.DBS error (trying curl with DBS or dbs both give error)
curl -X GET --header 'Accept: application/json' 'http://dns:8088/metacat-1.1.1/mds/v1/catalog/mysql-56-db/database/metastore/table/DBS?includeInfo=true&includeDefinitionMetadata=true&includeDataMetadata=true'

2018-09-13 02:09:59.700 ERROR 60262 --- [nio-8088-exec-1] c.n.metacat.main.api.ExceptionMapper : Unable to locate for mysql-56-db/metastore/dbs. Details: TABLE 'mysql-56-db/metastore/dbs' not found.
com.netflix.metacat.common.exception.MetacatNotFoundException: Unable to locate for mysql-56-db/metastore/dbs. Details: TABLE 'mysql-56-db/metastore/dbs' not found.
at com.netflix.metacat.main.api.RequestWrapper.processRequest(RequestWrapper.java:125) ~[metacat-main-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at com.netflix.metacat.main.api.v1.MetacatController.getTable(MetacatController.java:640) ~[metacat-main-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

This field is never used.

I am running docker version can you let me know how to load data from external mysql database to metacat

I've set the catalog folder location property in the application.yml file, but I can't find where to set the metacat.usermetadata.config.location; running a find function on the source files doesn't find any matching property name.

Hello dear metacat team,

I am trying to run metacat via docker, as explained in your project documentation but running ./gradlew metacatPorts yields the following error:

> Task :metacat-functional-tests:metacatPorts
jq: error (at <stdin>:263): Cannot iterate over null (null)

> Task :metacat-functional-tests:metacatPorts FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/home/XXXX/playground/metacat/metacat-functional-tests/build.gradle' line: 114

* What went wrong:
Execution failed for task ':metacat-functional-tests:metacatPorts'.
> Process 'command 'sh'' finished with non-zero exit value 10

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 35m 2s
93 actionable tasks: 51 executed, 42 up-to-date

Environment:

• Debian 10 Buster
• openjdk version "1.8.0_292"
• jq-1.5-1-a5b5cbe

Any pointers?

Can you please share a catalog samples of Snowflake?

I am very interested to explore metacat and appreciate Netflix's publishing to github for review. Apologies in advance for such a basic issue submittal, but after my initial clone of the repo, I attempted to run "./gradlew clean build" and receive a compilation error. I am wondering if there is a JDK version requirement, or perhaps something else simple. Could you advise?

This is an example of one of the errors I am seeing:

/Users/jhurff/projects/skunkworks/metacat/metacat-main/src/main/java/com/netflix/metacat/main/api/v1/MetacatController.java:805: error: unreported exception X; must be caught or declared to be thrown
                return table.orElseThrow(() -> new MetacatNotFoundException("Unable to find view: " + name));
                                        ^
  where X,T are type-variables:
    X extends Throwable declared in method <X>orElseThrow(Supplier<? extends X>)
    T extends Object declared in class Optional

Is there a requirement to use Java7, or does metacat work with Java8?

In the medium article, it was mentioned that metastore can act as a medium for converting the dataType of one datastore to another. Can someone give an example for the same.

Also, some documentation/example will be really helpful in using metacat by Thrift.

Thanks,
Sakshi

clickhouse is not support?

Hi guys,

well, looking for some data catalog solution at moment and found this as possible replacement of Atlas, but do you plan to support messaging systems like Kafka and/or Pulsar in near future?

Best regards,

Ladislav

We got the metacat war file up and running, but we wanted to use the elastic search functionality in it as well. Can you please tell us on how metacat publishes data to Elastic search along with the places where we should be making the necessary config changes.

CVE-2019-17571 CVE-2020-9488

Recommended upgrade version：
2.13.2

When I try to build the docker image according the the README:

./gradlew buildWarImage startMetacatCluster metacatPorts

I get:

Task 'buildWarImage' not found in root project 'metacat'.

Is the README out of date?

The program can potentially fail to release a database resource.

when i call http://localhost:8080/mds/v1/catalog, i get some error ：

There was an unexpected error (type=Unable to locate for getcatalognames. Details: Unable to locate any catalogs, status=404).
Unable to locate for getcatalognames. Details: Unable to locate any catalogs

I do not why ?how can i resolve this question?
and can you provide song docs?
thanks.

I got metacat running on my computer, and then I sent a post request to /catalog and it always come back as 201 not implemented. Inside the createCatalog method, the only code there is

throw new MetacatNotSupportedException("Create catalog is not supported.");

Was this by design and is there another way of creating a catalog?

can provide front pages

Hello, when integrating elastic search, I tried to call the API: "/ MDS / V1 / search / table? Q =" exception occurred: com. Netflix. Metacat. Common. Exception. Metacatexception: no such index. -- searchmetacatv1resource. Searchtables failed. I found that no index was created in ES. What is the index structure of ES?

In addition, how to save metadata changes to MySQL and es? Through event? I tried to modify the metadata of the table and didn't receive an event

Hi
I have successfully deployed the WAR file on Netcat. Unfortunately, I got this error message.
{"timestamp":1597209261983,"status":404,"error":"No message available","message":"No message available","path":"/akhil/"}
I have searched everywhere to resolve this issue but I couldn't. Any help would be highly appreciated.
Also please provide any documentation if you have any regarding Metacat.

CVE-2019-7611 CVE-2020-7021 CVE-2020-7019 CVE-2021-22135 CVE-2020-7018

Recommended upgrade version：
6.8.15

Where can I find the detailed documentation

Hello.
I wonder metacat can identify key fields of mysql or not.

I think key fields are important information,
but, it looks like metacat does not show it.

This is more a question than an issue. I browsed through this code base and I cannot locate any code related to data lineage. Is it part of Metacat?

You can reproduce this problem as follow:

• add a embedded Hive catalog with config javax.jdo.*
• Start service with java -jar metacat-app.*.jar

You will get this error:

Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
	... 90 more
Caused by: org.datanucleus.exceptions.NucleusUserException: Error : Could not find API definition for name "JDO". Perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH?
	at org.datanucleus.api.ApiAdapterFactory.getApiAdapter(ApiAdapterFactory.java:94)
	at org.datanucleus.AbstractNucleusContext.<init>(AbstractNucleusContext.java:115)
	at org.datanucleus.PersistenceNucleusContextImpl.<init>(PersistenceNucleusContextImpl.java:165)
	at org.datanucleus.PersistenceNucleusContextImpl.<init>(PersistenceNucleusContextImpl.java:154)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:419)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:308)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:217)
	at com.netflix.metacat.connector.hive.client.embedded.HivePersistenceManagerFactory.getpersistencemanagerfactory(HivePersistenceManagerFactory.java:97)
	at com.netflix.metacat.connector.hive.client.embedded.HivePersistenceManagerFactory.getPersistenceManagerFactory(HivePersistenceManagerFactory.java:60)

Are there any documents and/or discussion groups?

public void create(@nonnull final ConnectorRequestContext context, @nonnull final DatabaseInfo resource) {
......
JdbcConnectorUtils.executeUpdate(connection, "CREATE DATABASE " + databaseName);
......

create database can’t set character set and collation，so I can't create a replicated database correctly

This project looks interesting but without any documentation is note very useful

Any plans to add a (al least a mini) tutorial?

Regards
Matteo

Hi ajoymajumdar.Firstly,Thanks to your open source code, I am glad to explore metacat, but when I follow your readme,I have met some errors. I have try some solutions in these issues, but it did not work to me, can you help me? The error is shown in the following figure. Thank you

Hello - apologies for entering this as an issue (it's really just a question) - I asked this on the google group metacatoss, but I'm not sure anyone is checking that. I'm wondering since Teradata is referenced in the readme, which connector should be used for it? Obviously it allows jdbc connections, but jdbc isn't by itself a valid connector.

I use ./gradlew metacatPorts command to excute, at last i get

Task :metacat-functional-tests:metacatPorts
++ Container Exposed ports ++
+++ Metacat Web Port:32812
+++ Metacat Web Debug Port:32813
+++ Metacat Hive Thrift Port:32811
+++ Metacat Embedded Hive Thrift Port:32810
+++ Metacat Embedded Fast Hive Thrift Port:32809
+++ Metacat Hive Metastore Port:32807
+++ Metacat Hive Metastore Debug Port:32808

i can explore the web when i use http://localhost:32812/mds/v1/catalog.
But there is an error
<============-> 98% EXECUTING [23s] Attaching to metacat-test-cluster_storage-barrier_1
storage-barrier_1 | Waiting for postgresql:5432 ...... up!
storage-barrier_1 | Waiting for hive-metastore-db:3306 ......nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | .nc: getaddrinfo: Name does not resolve
storage-barrier_1 | . ERROR: unable to connect
metacat-test-cluster_storage-barrier_1 exited with code 1

:m+ '[' 0 -ne 0 ']'-tests:startMetacatCluster

why i got this message?

Trying to clone the git repo and get the permission denied error. How can get access?

git clone [email protected]:Netflix/metacat.git
Cloning into 'metacat'...
The authenticity of host 'github.com (192.30.255.112)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added 'github.com,192.30.255.112' (RSA) to the list of known hosts.
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Hello, this is Jaehwa.

I have quick question about Metacat.

I have read in several articles that Netflix uses the following batchid pattern to avoid eventual consistency issues.

• Never delete existing output in partitions.
• Each job inserts a unique sub-partition called "batchid".

If I store partitions in the Hive table using Metcat, does Metacat create the batchid?

Best Regards,
Jaehwa

Project needs documentation / wiki. Even a PDF would help :)

Hi Team,

While getting the databases for Postgres, we are getting only the schemas, not the databases.

Thanks,
Sakshi

Instead of clonning the repo. I downloaded sources from https://github.com/Netflix/metacat/releases/tag/v1.2.2

After that executing the

./gradlew clean build

fails with

...
 Download https://plugins.gradle.org/m2/org/ajoberstar/gradle-git/1.7.2/gradle-git-1.7.2.jar

> Configure project : 
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
org.ajoberstar.github-pages is deprecated will be removed in gradle-git 2.0.0. Users should migrate to org.ajoberstar.git-publish (https://github.com/ajoberstar/gradle-git-publish).
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.
Git repository not found at /Users/tperelmutov/Repos/RESERVOIR/metacat-1.2.2 -- nebula-release tasks will not be available. Use the git.root Gradle property to specify a different directory.


FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'metacat'.
> One of setGitDir or setWorkTree must be called.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 56s

Can anyone explain the difference between definition_metadata and data_metadata.

Thanks,
Sakshi

I have been built the war package,but how can I config the two basic configurations

HI Metecat team,

I GIT cloned and tried to do a clean build. Latest Gradle is 5.1.1, but build compile error says it uses Gradle 6. Build also fails.

My Gradle output
xyzs-MBP:metacat ssabat$ gradle -v

Gradle 5.1.1

Build time: 2019-01-10 23:05:02 UTC
Revision: 3c9abb645fb83932c44e8610642393ad62116807

Kotlin DSL: 1.1.1
Kotlin: 1.3.11
Groovy: 2.5.4
Ant: Apache Ant(TM) version 1.9.13 compiled on July 10 2018
JVM: 11.0.1 (Oracle Corporation 11.0.1+13)
OS: Mac OS X 10.11.6 x86_64

By build failure message is

Configure project :
Inferred project: metacat, version: 1.2.0-SNAPSHOT
org.ajoberstar.github-pages is deprecated will be removed in gradle-git 2.0.0. Users should migrate to org.ajoberstar.git-publish (https://github.com/ajoberstar/gradle-git-publish).

FAILURE: Build failed with an exception.

• Where:
  Build file '/Users/ssabat/metacat/build.gradle' line: 41

• What went wrong:
  An exception occurred applying plugin request [id: 'org.unbroken-dome.test-sets', version: '1.4.2']

Failed to apply plugin [id 'org.unbroken-dome.test-sets']
Could not create an instance of type org.unbrokendome.gradle.plugins.testsets.internal.DefaultTestSetContainer_Decorated.
> java.lang.StackOverflowError (no error message)

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

• Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 23s

Build fails for both Java 8 and 11. I am using OpenJDK.

Could you please advise?

Thanks,

Sunil Sabat

CVE-2017-3162 CVE-2016-5001 CVE-2017-3161

Recommended upgrade version：
2.3.4

when i call http://localhost:8080/mds/v1/catalog, i get some error ：

There was an unexpected error (type=Unable to locate for getcatalognames. Details: Unable to locate any catalogs, status=404). Unable to locate for getcatalognames. Details: Unable to locate any catalogs

I do not why ?how can i resolve this question? and can you provide song docs? thanks.

Once the build is completed, the metacat WAR file is generated under metacat-war/build/libs directory. Metacat needs two basic configurations:

metacat.plugin.config.location: Path to the directory containing the catalog configuration. Please look at catalog samples used for functional testing.

metacat.usermetadata.config.location: Path to the configuration file containing the connection properties to store user metadata.

So we need to update the above properties before build or after build. If it is after build where can i change the properties

./gradlew startMetacatCluster or ./gradlew metacatPort constantly fail because Cassandra take a few seconds to start: https://github.com/Netflix/metacat/blob/master/scripts/start_metacat_test_cluster.sh#L17

A simple hack would be just adding a sleep here, or we can poke on the Cassandra port and wait until it is available.

Is this repo accepting PRs right now? I can take a grab at this. Thanks for open sourcing this project!

Hi,
I see metacat support redshift connector, but still got some questions:

• Do you have an example of catalog configuration for redshift? (I dont see in metacat/metacat-functional-tests/metacat-test-cluster/etc-metacat/catalog/)
• How do I edit metacat.plugin.config.location to hold the catalog directory ? as environment variable? its not very clear in the README

Dose metacat can work with apache ranger? I have not find any document about it. Thanks a lot

I ran the following commands

./gradlew clean build
./gradlew metacatPorts

It keeps failing with

ERROR: for metacattestcluster_metacat_1 Cannot create container for service metacat: lstat /nail/home/smoy/external/metacat/metacat-functional-tests/build/logs
/metacat: not a directory

ERROR: for metacat Cannot create container for service metacat: lstat /nail/home/smoy/external/metacat/metacat-functional-tests/build/logs/metacat: not a direc
tory

So I work around the problem by making the directory ahead of metacatPorts as follow

mkdir -p /nail/home/smoy/external/metacat/metacat-functional-tests/build/logs/metacat

Hope this help others who ran into the same problem

can you post cassandra catalog example?

is presto a source of metadata for metacat? or can presto query data from metacat?

can you provide some sample configs for presto integration?

Travis are now recommending removing the sudo tag.

"If you currently specify sudo: false in your .travis.yml, we recommend removing that configuration"

CVE-2020-17521

Recommended upgrade version：2.5.13-01

can you post s3 catalog example (not hive)? i couldn't see any with bucket path, access keys.etc I assume metacat can import object tags.