logicalclocks / hops-examples Goto Github PK

Examples for Deep Learning/Feature Store/Spark/Flink/Hive/Kafka jobs and Jupyter notebooks on Hops

Java 0.53% Jupyter Notebook 93.05% Scala 0.46% Python 2.35% HTML 2.27% CSS 1.33%

tensorflow spark flink hive jupyter-notebook feature-store

hops-examples's Introduction

This Repository is archived and no longer maintained. Hopsworks tutorials can be found in the hopsworks-tutorial repository.

hops-examples's People

Contributors

Stargazers

Watchers

hops-examples's Issues

Script Integrity Attribute Not Implemented

Reported by the penetration testing:

A script tag with content from another domain is not using an integrity attribute.

<script type="text/javascript" async="" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/MathJax.js?config=TeX-MML-AM_CHTML">
</script>

We should do like

hops-examples/themes/berbera/layouts/_default/baseof.html

Line 84 in 4958157

 <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.3/umd/popper.min.js" integrity="sha384-vFJXuSJphROIrBnz7yo7oB41mKfc8JzQZiCq4NCceLEaO4IHwicKwpJf9c9IpFgh" 

and maybe also use the same version everywhere.

"online" parameter not recognized when creating featuregroup using pyspark

I get the following error when I try to create online feature store :

create_featuregroup() got an unexpected keyword argument 'online'
Traceback (most recent call last):
TypeError: create_featuregroup() got an unexpected keyword argument 'online'

I'm using hopsworks-0-10-0.tar.gz GCP image.

Thanks,
Vinay

Add a spark or sql transform

add support for JDK 11

mvn package crashes on JDK 11 with error

Caused by: java.lang.NoClassDefFoundError: javax/tools/ToolProvider

full log:

[INFO] --------------------< io.hops:hops-examples-spark >---------------------
[INFO] Building hops-examples-spark 0.10.0-SNAPSHOT [2/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hops-examples-spark ---
[INFO] Deleting /home/kaichi/Projects/hops-examples/spark/target
[INFO]
[INFO] --- maven-checkstyle-plugin:2.16:check (validate) @ hops-examples-spark ---
[INFO] Starting audit...
Audit done.
[INFO]
[INFO] --- build-helper-maven-plugin:3.0.0:add-source (add-source) @ hops-examples-spark ---
[INFO] Source directory: /home/kaichi/Projects/hops-examples/spark/src/main/scala added.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hops-examples-spark ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/kaichi/Projects/hops-examples/spark/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hops-examples-spark ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/kaichi/Projects/hops-examples/spark/target/classes
[WARNING] /home/kaichi/Projects/hops-examples/spark/src/main/java/io/hops/examples/spark/WorkflowExample.java: /home/kaichi/Projects/hops-examples/spark/src/main/java/io/hops/examples/spark/WorkflowExample.java uses or overrides a deprecated API.
[WARNING] /home/kaichi/Projects/hops-examples/spark/src/main/java/io/hops/examples/spark/WorkflowExample.java: Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) @ hops-examples-spark ---
Downloading from hops-snapshot-repository: https://bbc1.sics.se/archiva/repository/Hops/io/hops/hadoop-project-dist/2.8.2.8-SNAPSHOT/maven-metadata.xml
Downloaded from hops-snapshot-repository: https://bbc1.sics.se/archiva/repository/Hops/io/hops/hadoop-project-dist/2.8.2.8-SNAPSHOT/maven-metadata.xml (366 B at 1.2 kB/s)
Downloading from hops-snapshot-repository: https://bbc1.sics.se/archiva/repository/Hops/io/hops/hadoop-project/2.8.2.8-SNAPSHOT/maven-metadata.xml
Downloaded from hops-snapshot-repository: https://bbc1.sics.se/archiva/repository/Hops/io/hops/hadoop-project/2.8.2.8-SNAPSHOT/maven-metadata.xml (361 B at 3.1 kB/s)
Downloading from hops-snapshot-repository: https://bbc1.sics.se/archiva/repository/Hops/io/hops/hadoop-main/2.8.2.8-SNAPSHOT/maven-metadata.xml
Downloaded from hops-snapshot-repository: https://bbc1.sics.se/archiva/repository/Hops/io/hops/hadoop-main/2.8.2.8-SNAPSHOT/maven-metadata.xml (358 B at 3.1 kB/s)
[WARNING] Expected all dependencies to require Scala version: 2.11.12
[WARNING] org.apache.spark:spark-tags_2.11:2.4.0 requires scala version: 2.11.12
[WARNING] org.apache.spark:spark-streaming_2.11:2.4.0 requires scala version: 2.11.12
[WARNING] org.apache.kafka:kafka_2.11:1.1.0 requires scala version: 2.11.12
[WARNING] org.scala-lang:scala-reflect:2.11.12 requires scala version: 2.11.12
[WARNING] com.typesafe.scala-logging:scala-logging_2.11:3.7.2 requires scala version: 2.11.11
[WARNING] Multiple versions of scala libraries detected!
[INFO] /home/kaichi/Projects/hops-examples/spark/src/main/java:-1: info: compiling
[INFO] /home/kaichi/Projects/hops-examples/spark/src/main/scala:-1: info: compiling
[INFO] Compiling 2 source files to /home/kaichi/Projects/hops-examples/spark/target/classes at 1553523353016
[ERROR] error: java.lang.NoClassDefFoundError: javax/tools/ToolProvider
[INFO] at scala.reflect.io.JavaToolsPlatformArchive.iterator(ZipArchive.scala:301)
[INFO] at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[INFO] at scala.reflect.io.AbstractFile.foreach(AbstractFile.scala:92)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.traverse(ClassPath.scala:277)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.x$15$lzycompute(ClassPath.scala:299)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.x$15(ClassPath.scala:299)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.packages$lzycompute(ClassPath.scala:299)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:299)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:264)
[INFO] at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:358)
[INFO] at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:358)
[INFO] at scala.collection.Iterator$class.foreach(Iterator.scala:891)
[INFO] at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
[INFO] at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[INFO] at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[INFO] at scala.tools.nsc.util.MergedClassPath.packages$lzycompute(ClassPath.scala:358)
[INFO] at scala.tools.nsc.util.MergedClassPath.packages(ClassPath.scala:353)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply$mcV$sp(SymbolLoaders.scala:269)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply(SymbolLoaders.scala:260)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply(SymbolLoaders.scala:260)
[INFO] at scala.reflect.internal.SymbolTable.enteringPhase(SymbolTable.scala:235)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:260)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:211)
[INFO] at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1535)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:256)
[INFO] at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:73)
[INFO] at scala.tools.nsc.Global.rootMirror(Global.scala:71)
[INFO] at scala.tools.nsc.Global.rootMirror(Global.scala:39)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1390)
[INFO] at scala.tools.nsc.Global$Run.(Global.scala:1242)
[INFO] at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
[INFO] at scala.tools.nsc.MainClass.doCompile(Main.scala:23)
[INFO] at scala.tools.nsc.Driver.process(Driver.scala:51)
[INFO] at scala.tools.nsc.Driver.main(Driver.scala:64)
[INFO] at scala.tools.nsc.Main.main(Main.scala)
[INFO] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[INFO] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[INFO] at scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
[INFO] at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[INFO] java.lang.reflect.InvocationTargetException
[INFO] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[INFO] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[INFO] at scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
[INFO] at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[ERROR] Caused by: java.lang.NoClassDefFoundError: javax/tools/ToolProvider
[INFO] at scala.reflect.io.JavaToolsPlatformArchive.iterator(ZipArchive.scala:301)
[INFO] at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[INFO] at scala.reflect.io.AbstractFile.foreach(AbstractFile.scala:92)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.traverse(ClassPath.scala:277)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.x$15$lzycompute(ClassPath.scala:299)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.x$15(ClassPath.scala:299)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.packages$lzycompute(ClassPath.scala:299)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:299)
[INFO] at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:264)
[INFO] at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:358)
[INFO] at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:358)
[INFO] at scala.collection.Iterator$class.foreach(Iterator.scala:891)
[INFO] at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
[INFO] at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[INFO] at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[INFO] at scala.tools.nsc.util.MergedClassPath.packages$lzycompute(ClassPath.scala:358)
[INFO] at scala.tools.nsc.util.MergedClassPath.packages(ClassPath.scala:353)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply$mcV$sp(SymbolLoaders.scala:269)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply(SymbolLoaders.scala:260)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply(SymbolLoaders.scala:260)
[INFO] at scala.reflect.internal.SymbolTable.enteringPhase(SymbolTable.scala:235)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:260)
[INFO] at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:211)
[INFO] at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1535)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:256)
[INFO] at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:73)
[INFO] at scala.tools.nsc.Global.rootMirror(Global.scala:71)
[INFO] at scala.tools.nsc.Global.rootMirror(Global.scala:39)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1390)
[INFO] at scala.tools.nsc.Global$Run.(Global.scala:1242)
[INFO] at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
[INFO] at scala.tools.nsc.MainClass.doCompile(Main.scala:23)
[INFO] at scala.tools.nsc.Driver.process(Driver.scala:51)
[INFO] at scala.tools.nsc.Driver.main(Driver.scala:64)
[INFO] at scala.tools.nsc.Main.main(Main.scala)
[INFO] ... 6 more
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] hops-examples 0.10.0-SNAPSHOT ...................... SUCCESS [ 7.253 s]
[INFO] hops-examples-spark ................................ FAILURE [ 10.870 s]
[INFO] hops-examples-hive ................................. SKIPPED
[INFO] hops-examples-featurestore 0.10.0-SNAPSHOT ......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.351 s
[INFO] Finished at: 2019-03-25T15:15:53+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (default) on project hops-examples-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 240 (Exit value: 240) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hops-examples-spark

KeyError: 'HDFS_USER'

from hops import hdfs
hdfs.copy_to_local('./models/model.pkl')

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.