Topic: apache-spark Goto Github

👇 Here are 1589 public repositories matching this topic...

1duo / awesome-ai-infrastructures
apache-spark,Infrastructures™ for Machine Learning Training/Inference in Production.
User: 1duo
artificial-intelligence machine-learning deep-learning machine-learning-systems awesome-list deep-learning-framework kubernetes apache-spark apache-arrow apache-mesos

aloneguid / parquet-dotnet
apache-spark,Fully managed Apache Parquet implementation
User: aloneguid
Home Page: https://aloneguid.github.io/parquet-dotnet/
apache-parquet dotnet dotnet-core dotnet-standard apache-spark linux windows ios xamarin xbox

awesome-spark / awesome-spark
apache-spark,A curated list of awesome Apache Spark packages and resources.
Organization: awesome-spark
apache-spark pyspark awesome sparkr

awesome-spark / spark-gotchas
apache-spark,Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Organization: awesome-spark
apache-spark book guide pyspark

big-data-europe / docker-spark
apache-spark,Apache Spark docker image
Organization: big-data-europe
spark-kubernetes kubernetes k8s-spark docker apache-spark

cartershanklin / pyspark-cheatsheet
apache-spark,PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
User: cartershanklin
spark apache-spark pyspark big-data

cerndb / dist-keras
apache-spark,Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
Organization: cerndb
Home Page: http://joerihermans.com/work/distributed-keras/
machine-learning deep-learning apache-spark data-parallelism distributed-optimizers keras optimization-algorithms tensorflow data-science hadoop

cuebook / cuelake
apache-spark,Use SQL to build ELT pipelines on a data lakehouse.
Organization: cuebook
Home Page: https://cuelake.cuebook.ai
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion

databricks / learningsparkv2
apache-spark,This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
Organization: databricks
Home Page: https://learning.oreilly.com/library/view/learning-spark-2nd/9781492050032/
apache-spark spark structured-streaming spark-sql spark-mllib mllib mlflow delta-lake

databricks / spark-sklearn
apache-spark,(Deprecated) Scikit-learn integration package for Apache Spark
Organization: databricks
apache-spark scikit-learn grid-search parameter-tuning machine-learning

datamechanics / delight
apache-spark,A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.
Organization: datamechanics
Home Page: https://www.datamechanics.co/delight
spark spark-history-server delight monitoring cpu memory spark-ui apache-spark dashboard kubernetes

dmmiller612 / sparktorch
apache-spark,Train and run Pytorch models on Apache Spark.
User: dmmiller612
pytorch sparktorch deep-learning apache-spark pipelines inference distributed-computing

dotnet / spark
apache-spark,.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Organization: dotnet
Home Page: https://dot.net/spark
spark csharp dotnet analytics bigdata spark-streaming spark-sql machine-learning fsharp dotnet-core

ekampf / pyspark-boilerplate
apache-spark,A boilerplate for writing PySpark Jobs
User: ekampf
python apache-spark pyspark boilerplate

feathr-ai / feathr
apache-spark,Feathr – A scalable, unified data and AI engineering platform for enterprise
Organization: feathr-ai
Home Page: https://join.slack.com/t/feathrai/shared_invite/zt-1ffva5u6v-voq0Us7bbKAw873cEzHOSg
feature-engineering feature-store artificial-intelligence mlops data-engineering data-quality machine-learning apache-spark azure data-science

hortonworks-spark / spark-atlas-connector
apache-spark,A Spark Atlas connector to track data lineage in Apache Atlas
Organization: hortonworks-spark
apache-spark apache-atlas

hydrospheredata / mist
apache-spark,Serverless proxy for Spark cluster
Organization: hydrospheredata
Home Page: http://hydrosphere.io/mist/
apache-spark api big-data serverless

infoslack / awesome-kafka
apache-spark,A list about Apache Kafka
User: infoslack
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing

intel-analytics / bigdl-2.x
apache-spark,BigDL: Distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray
Organization: intel-analytics
Home Page: https://bigdl.readthedocs.io
apache-spark deep-neural-network distributed-deep-learning keras-tensorflow bigdl analytics-zoo python scala pytorch

japila-books / apache-spark-internals
apache-spark,The Internals of Apache Spark
Organization: japila-books
Home Page: https://books.japila.pl/apache-spark-internals
apache-spark spark book internals

japila-books / spark-sql-internals
apache-spark,The Internals of Spark SQL
Organization: japila-books
Home Page: https://books.japila.pl/spark-sql-internals
apache-spark spark-sql internals spark mkdocs-material book

japila-books / spark-structured-streaming-internals
apache-spark,The Internals of Spark Structured Streaming
Organization: japila-books
Home Page: https://books.japila.pl/spark-structured-streaming-internals
apache-spark spark structured-streaming internals book mkdocs-material

kubeflow / spark-operator
apache-spark,Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Organization: kubeflow
apache-spark google-cloud-dataproc kubernetes kubernetes-controller kubernetes-crd kubernetes-operator spark

lensacom / sparkit-learn
apache-spark,PySpark + Scikit-learn = Sparkit-learn
Organization: lensacom
scikit-learn apache-spark machine-learning distributed-computing python

lifeomic / sparkflow
apache-spark,Easy to use library to bring Tensorflow on Apache Spark
Organization: lifeomic
tensorflow apache-spark deep-learning dataframe spark-ml pipeline lifeomic

lucacanali / miscellaneous
apache-spark,Includes notes on using Apache Spark, Spark for Physics, a tool for running TPCDS on PySpark, a tool for performance testing CPUs, Jupyter notebook examples for Spark, Oracle and other DB systems.
User: lucacanali
apache-spark database jupyter-notebooks performance-analysis performance-monitoring performance-testing

lucacanali / sparkmeasure
apache-spark,This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
User: lucacanali
spark apache-spark performance-troubleshooting performance-metrics python scala

lw-lin / coolplayspark
apache-spark,酷玩 Spark: Spark 源代码解析、Spark 类库等
User: lw-lin
apache-spark spark spark-streaming sparkcore structured-streaming

lw-lin / streaming-readings
apache-spark,Streaming System 相关的论文读物
User: lw-lin
stream-processing streaming flink spark-streaming storm heron dataflow drizzle millwheel s4

mahmoudparsian / data-algorithms-book
apache-spark, MapReduce, Spark, Java, and Scala for Data Algorithms Book
User: mahmoudparsian
Home Page: http://mapreduce4hackers.com
hadoop-mapreduce java distributed-computing scala mapreduce data-algorithms python machine-learning pyspark distributed-algorithms

microsoft / data-accelerator
apache-spark,Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Organization: microsoft
spark spark-streaming spark-sql sparksql streaming-data streaming servicefabric nodejs docker hdinsight

microsoft / mobius
apache-spark,C# and F# language binding and extensions to Apache Spark
Organization: microsoft
spark apache-spark rdd dataframe dstream dataset streaming csharp mobius kafka-streaming

microsoft / synapseml
apache-spark,Simple and Distributed Machine Learning
Organization: microsoft
Home Page: http://aka.ms/spark
spark pyspark azure scala microsoft ml machine-learning databricks cognitive-services lightgbm

miguno / kafka-storm-starter
apache-spark,[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
User: miguno
Home Page: http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/
apache-kafka kafka apache-storm storm spark apache-spark scala integration avro apache-avro

miguno / wirbelsturm
apache-spark,[PROJECT IS NO LONGER MAINTAINED] Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
User: miguno
Home Page: http://www.michael-noll.com/blog/2014/03/17/wirbelsturm-one-click-deploy-storm-kafka-clusters-with-vagrant-puppet/
vagrant puppet kafka apache-kafka storm apache-storm spark apache-spark

mingchen0919 / learning-apache-spark
apache-spark,Notes on Apache Spark (pyspark)
User: mingchen0919
apache-spark pyspark-tutorial machine-learning

mlflow / mlflow
apache-spark,Open source platform for the machine learning lifecycle
Organization: mlflow
Home Page: https://mlflow.org
machine-learning ai ml mlflow apache-spark model-management

mrpowers / quinn
apache-spark,pyspark methods to enhance developer productivity 📣 👯 🎉
User: mrpowers
Home Page: https://mrpowers.github.io/quinn/
pyspark apache-spark

nchammas / flintrock
apache-spark,A command-line tool for launching Apache Spark clusters.
User: nchammas
apache-spark ec2 apache-spark-cluster orchestration spark-ec2

opencypher / morpheus
apache-spark,Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
Organization: opencypher
apache-spark cypher scala apache2 graph big-data

openscoring / openscoring
apache-spark,REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models
Organization: openscoring
api real-time pmml r scikit-learn apache-spark xgboost lightgbm

oryxproject / oryx
apache-spark,Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
Organization: oryxproject
Home Page: http://oryx.io
lambda-architecture oryx apache-spark machine-learning kafka apache-kafka java cloudera

ptyadana / sql-data-analysis-and-visualization-projects
apache-spark,SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
User: ptyadana
sql mysql mysql-notes exercises mysqlworkbench mysql-database data-analysis postgresql postgres sqlite

rjurney / agile_data_code_2
apache-spark,Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
User: rjurney
Home Page: http://bit.ly/agile_data_science
data-syndrome data data-science analytics apache-spark apache-kafka kafka spark predictive-analytics machine-learning

san089 / goodreads_etl_pipeline
apache-spark,An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
User: san089
etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3

spark-notebook / spark-notebook
apache-spark,Interactive and Reactive Data Science using Scala and Spark.
Organization: spark-notebook
apache-spark notebook scala data-science spark reactive

sparklyr / sparklyr
apache-spark,R interface for Apache Spark
Organization: sparklyr
Home Page: https://spark.rstudio.com/
r rstats apache-spark machine-learning dplyr sparklyr remote-clusters livy ide distributed

tirthajyoti / spark-with-python
apache-spark,Fundamentals of Spark with Python (using PySpark), code examples
User: tirthajyoti
pyspark spark apache-spark dataframe mlib machine-learning big-data database map-reduce python

treeverse / lakefs
apache-spark,lakeFS - Data version control for your data lake | Git for data
Organization: treeverse
Home Page: https://docs.lakefs.io
apache-spark apache-sparksql aws-s3 azure-blob-storage azure-storage data-engineering data-lake data-quality data-version-control data-versioning datalake datalakes git-for-data go golang google-cloud-storage hadoop-filesystem lakefs object-storage

tweag / sparkle
apache-spark,Haskell on Apache Spark.
Organization: tweag
apache-spark spark haskell analytics