fluid-cloudnative / community Goto Github PK
View Code? Open in Web Editor NEWInformation about the Fluid community.
License: Apache License 2.0
Information about the Fluid community.
License: Apache License 2.0
Dear Fluid TOC members and maintainers,
I am proposing to have Zifan Ni (GithubID: zifanni) as a new Fluid committer.
Zifan Ni, was a software engineer at Bilibili, works in the AI platform team, focusing on the storage and its accelaration for AI platforms. He is the core member of Fluid’s landing project in the Bilibili AI platform scene.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=nizifan
Jan 4, 2022
[BugFix] change mountTime to *v1.Time
Aug 6, 2021
disable fuse.shared.caching.reader.enabled for alluxio and goosefs, see Alluxio/alluxio#14909 (#1400)
Jan 2, 2021
Add remount check for hostpath mount during sync
April 4, 2022
Fix bug in helm template
When AI Meets Alluxio at Bilibili | Building an Efficient AI Platform for Data Preprocessing and Model Training
Building an Efficient AI Training Platform at bilibili with Alluxio - YouTube
Fluid v0.6.0:
Add remount check for hostpath mount during sync fluid-cloudnative/fluid#1340
The origion wrong youtube link lead to meeting 13, However, It should be meething 14, weblink: https://www.youtube.com/watch?v=9A7HG5HuZ1w.
Dear Fluid TOC members and maintainers,
I am proposing to have Lingwei Qiu (GithubID: yangyuliufeng) as a new Fluid maintainer.
Lingwei Qiu, is a software engineer at China Telecom, works in the Paas R&D team, focusing on the planning and construction of container platforms and also works on the combination of Fluid and AI platforms. He is the core staff member of Fluid’s landing project in the China Telecom Cloud Native scene.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=yangyuliufeng
Fluid Community Meeting talks: Plan Discussion for Fluid Operation Refactoring
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20221124)
Fluid Community Meeting talks: Plan Discussion for BigData App Integration for Fluid
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20211027)
Fluid Community Meeting talks: Optimize Pod scheduling through Fluid
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20210728)
Fluid v0.4.0:
Split the Fluid controller into runtime controller and dataset controller: fluid-cloudnative/fluid#271
Refactor the start command and start function: fluid-cloudnative/fluid#232
Fluid v0.5.0:
Redesign the Cache runtime lifecycle and support backuping and restoring of Dataset: fluid-cloudnative/fluid#564, fluid-cloudnative/fluid#591
Fluid v0.6.0:
Cache-aware scheduling algorithm optimization: fluid-cloudnative/fluid#725
Dataset mount point dynamic update: fluid-cloudnative/fluid#954
Fluid v0.7.0:
Optimize CI process: fluid-cloudnative/fluid#1309, fluid-cloudnative/fluid#1368
help community members to design useful solutions for mount point automatically recovered
Fluid v0.8.0:
Redesign the startup logic of controller so that it can sense the change of certificate: fluid-cloudnative/fluid#1295,fluid-cloudnative/fluid#1420
Feel free to +1 or -1, starting with my +1.
Thanks,
Rong
Dear Fluid TOC members and maintainers,
I am proposing to have Dongdong Lv (GithubID: ldd91) as a new committer.
Dongdong Lv, Deep Learning Platform architect of Unisound,He is responsible for Unisound's large-scale distributed machine learning platform architecture design and new function evolution, deep learning algorithm application optimization and AI model acceleration. Research areas include large-scale cluster scheduling, high performance computing, distributed file storage, distributed cache, etc. Participate in Fluid, Alluxio, JuiceFS, Volcano, Cube Studio and other open source projects.
As an early adopter of the Fluid, Actively participate in community weekly meetings and discussions, Contributed to the growth of community , expand the user case of fluid. cooperative development with JuiceData Team to add new runtime of fluid – JuicefsRuntime
Talks:
2022 Alluxio Day 11: Speeding Up In Atlas Deep Learning Platform with Fluid+Alluxio
Fluid Community Meeting talks: Add JuiceFSRuntime Engine To Fluid Backend
Blogs:
Speeding Up In Atlas Deep Learning Platform with Fluid+Alluxio
Speeding Up the AI Supercomputing Platform - Practice at Unisound
v0.3.0:
v0.7.0:
Dear Fluid TOC members and maintainers,
I am proposing to have Tao Wang(GithubID: frankleaf) as a new committer.
Tao Wang, is a software engineer at Alibaba Cloud, works in the Open Source Big Data team, focusing on the combination of Fluid and JindoFS and also works on the development of Fluid and JindoFS core system, He is the core staff member of Fluid’s landing project on Alibaba inner scenarios and many AI and bigdata accelerating scene like weibo、zuoyebang etc.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=frankleaf
Fluid Community Meeting talks: Introduction of Fluid JindoRuntime
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20210923))
Fluid v0.5.0:
Fluid Integrate With JindoRuntime Engine: fluid-cloudnative/fluid#493
Support JindoRuntime DataLoader: fluid-cloudnative/fluid#764
Fluid v0.6.0:
Refactor Dataloader Code Structure To Engine Level: fluid-cloudnative/fluid#826
Fluid v0.7.0:
Support QuotaList Regex With Multi DiskPath: fluid-cloudnative/fluid#678
JindoFuse Recover Automatically: fluid-cloudnative/fluid#1385
Feel free to +1 or -1, starting with my +1.
Thanks,
Rong
To enhance the fluid dataset manager capability, i summary our commany dataset usage:
Unisound Atlas AI Platform(网络隔离):
数据存储方式:
非对象存储,为分布式文件系统存储,分布式文件挂载到所有物理节点的固定目录,所有组和用户有统一的入口
数据的来源:
数据的备份:
数据的使用场景和使用方式:
使用数据的痛点:
Dear Fluid TOC members and maintainers,
I am proposing to have Sichen Zhao (GithubID: hahchenchen) as a new Fluid committer.
Sichen Zhao, is a software engineer at Shopee, works in the Paas team,focuses on cloud native AI infrastructure and other kubernetes related products, and also works on the combination of Fluid and AI platforms. He likes to explore and contribute to open source technologies, contributed to other communities like kubevela, Newlib, RTEMS, and fluid community.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=hahchenchen
Jan 8, 2023
Feature: Support sharedOptions in dataset
Dec 10, 2022
Optimization: set dataset mount.path of local/pvc , do no take effect. the
Dec 4, 2022
Optimization: when master worker delete, the dataset and runtime status do not changed
Dec 1, 2022
Feature: Support Fuse ds support onDelete updateStrategy for alluxioRuntime
Nov 11, 2022
Optimization: Fix issue: 2294, set tieredstore in AlluxioRuntime do not affect memory request.
Nov 9, 2022
Optimization: Fix issue: 2198, when set alluxio runtime master port, do not take effect
Nov 7, 2022
Optimization: Fix the bug set Port Properties In alluxioRuntime, alluxio Pod crash
Sep 16, 2022
Optimization: Fix when alluxioruntime fuse args set, add mountPath in args
Blogs:
https://blog.csdn.net/hahachenchen789/article/details/128957437
https://blog.csdn.net/hahachenchen789/article/details/128957498
https://blog.csdn.net/hahachenchen789/article/details/128957585
Fluid v0.8.3:
Support sharedOptions and sharedEncryptOptions in dataset: fluid-cloudnative/fluid#2574
Fluid v0.8.2:
Support dataset use mount path of local/pv: .fluid-cloudnative/fluid#2406
Fix alluxio runtime bugs, set Port, set tieredstore, status changed when master/worker deleted: fluid-cloudnative/fluid#2388, fluid-cloudnative/fluid#2371, fluid-cloudnative/fluid#2295, fluid-cloudnative/fluid#2287
Optimization for set Port Properties In alluxioRuntime : fluid-cloudnative/fluid#2263
Fluid v0.8.1:
Optimization for Alluxio runtime fuse args: fluid-cloudnative/fluid#2135
Feel free to +1 or -1, starting with my +1.
Thanks,
Rong
Dear Fluid TOC members and maintainers,
I am proposing to have Lingwei Qiu (GithubID: yangyuliufeng) as a new Fluid committer.
Lingwei Qiu, is a software engineer at China Telecom, works in the Paas R&D team, focusing on the planning and construction of container platforms and also works on the combination of Fluid and AI platforms. He is the core staff member of Fluid’s landing project in the China Telecom AI platform scene.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=yangyuliufeng
Jan 4, 2022
Store certs of webhook server in a secret so that webhook can have more replicas.
Aug 6, 2021
Optimize the function of dynamic updating of Dataset mount points. Solve some critical bugs to make Fluid more stable
Jan 2, 2021
Design the webhook framework and optimize cache-aware scheduling algorithm so that we can optimize Pod scheduling through Fluid.
Feb 26, 2021
Contributed the Databackup controller. Re-design the cache runtime lifecycle to help more cache engine to be integrated into Fluid.
Fluid Community Meeting talks: Plan Discussion for BigData App Integration for Fluid
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20211027)
Fluid Community Meeting talks: Optimize Pod scheduling through Fluid
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20210728)
Fluid v0.4.0:
Split the Fluid controller into runtime controller and dataset controller: fluid-cloudnative/fluid#271
Refactor the start command and start function: fluid-cloudnative/fluid#232
Fluid v0.5.0:
Redesign the Cache runtime lifecycle and support backuping and restoring of Dataset: fluid-cloudnative/fluid#564, fluid-cloudnative/fluid#591
Fluid v0.6.0:
Cache-aware scheduling algorithm optimization: fluid-cloudnative/fluid#725
Dataset mount point dynamic update: fluid-cloudnative/fluid#954
Feel free to +1 or -1, starting with my +1.
Thanks,
Rong
Dear Fluid TOC members and maintainers,
I applied to wushiming (GithubID: wushiming540) as a new committer.
Wu Shiming, senior engineer of Huawei's NAIE platform, is responsible for large-scale distributed machine learning platform architecture design and new feature evolution, deep learning algorithm application optimization, and AI model acceleration. Research areas include large-scale cluster scheduling, high-performance computing, distributed file storage, distributed cache, etc. Participate in open source projects such as Fluid, Alluxio, and Volcano.
As an in-depth user of Fluid, I actively participate in weekly community meetings and discussions, contributing to the growth of the community and expanding Fluid's user cases.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=wushiming540
Aug 16, 2023
docs: build fluid compnent
https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20231019
Dear Fluid TOC members and maintainers,
I am proposing to have Shouzhuo Sun (GithubID: ssz1997) as a new committer.
Shouzhuo Sun is a software engineer of Alluxio. He works in Alluxio data team, focuses on cloud native applications, AI on Kubernetes, and other kubernetes related products. He is the point of contact from Alluxio for Alluxio runtime.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=ssz1997
Nov 5, 2021
Enable the fluid diagnose tool to obtain logs of Alluxio components.
Feb 19, 2022
Mitigate Alluxio dataload slowness issue.
Talks:
Fluid Community Meeting talks: Alluxio 2.9 new feature Sharing
https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20221222
Fluid Community Meeting talks: Recent Work on Alluxio Fuse
https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20220216
v0.7.0:
Enable the fluid diagnose tool to obtain logs of Alluxio components: fluid-cloudnative/fluid#1060
Mitigate Alluxio dataload slowness issue: fluid-cloudnative/fluid#1450
v0.8.0:
Add Fluid with S3 documentation: fluid-cloudnative/fluid#1494
Feel free to +1 or -1, starting with my +1.
Thanks,
Rong
Dear Fluid TOC members and maintainers,
I am proposing to have Hao Ma (GithubID: allenhaozi) as a new Fluid committer.
Hao Ma is a senior engineer for 4Paradigm. He works on the 4paradigm machine learning platform, Committed to the exploration, construction and implementation of MLOPS concept on the company's machine learning platform. He believes in open source, loves open source, willing to contribute to open source.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=allenhaozi
Cache Runtime Support Container Network:
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20220428) .
build kubernetes csi development env by kind on Mac:
(https://medium.com/@allenhaozi/build-kubernetes-csi-development-env-by-kind-on-mac-d61947b71661) .
KubeMeet 杭州站开放报名,一起击破「云原生应用管理」痛点!
(https://mp.weixin.qq.com/s/84VzBqrddi3LPSQuDrRnaQ) .
v0.8.2
support image pull secrets #2185
v0.8.0
Container Network with short-circuit read support #1287
v0.7.0
unify the file format under cmd, alluxio jindo dataset #1388
v0.6.0
add build fluid depvelopment env with kind on mac doc #760
Dear Fluid TOC members and maintainers,
I am proposing to have Yuandong Xie (GithubID: xieydd) as a new committer.
Yuandong Xie, is a senior engineer of Tencent Cloud. He works in Tencent Kubernetes Engine(TKE) team, focuses on cloud native AI infrastructure and other kubernetes related products. Yuandong also focus on elastic kubernetes service on tencent cloud. He likes to explore and contribute to open source technologies, is also member of kubeflow、Istio and fluid community.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=xieydd
Feb 26th, 2022
Contributed to the growth of community , expand the user case of fluid. Add new runtime of fluid – GooseFSRuntime, Provides fluid monitoring、stability enhancements and enterprise-level needs.
Talks:
KubeCon 2021 China: Fluid : Kubernetes 原生分布式数据集协调器和加速器
https://www.lfasiallc.com/kubecon-cloudnativecon-open-source-summit-china/program/schedule/
KubeCon 2021 NorthAmerican: Fluid- Build Data Orchestration in Kubernetes.
https://sched.co/lV2U
Fluid Community Meeting talks: Plan Discussion for BigData App Integration for Fluid
https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20210630
Blogs:
https://cloud.tencent.com/developer/article/1868108
v0.3.0:
Support the underlay storage which is configured only with non-root: fluid-cloudnative/fluid#168
v0.4.0:
Support deploying multiple dataset in the same node: fluid-cloudnative/fluid#242
v0.5.0:
Enhance Prometheus monitoring support for AlluxioRuntime: fluid-cloudnative/fluid#466
v0.6.0:
Enhance HA support for cache Runtime: fluid-cloudnative/fluid#781
Support new cache Runtime:GooseFS: fluid-cloudnative/fluid#912
Feel free to +1 or -1, starting with my +1.
Thanks,
Rong
Dear Fluid TOC members and maintainers,
I am proposing to have Ruofeng Lei (GithubID: abowloflrf) as a new committer.
Ruofeng Lei, is a senior software engineer of Baidu AI Cloud. He works in Baidu Cloud Container Engine (CCE) Team, focuses on cloud native AI infrastructure, helps users to develop, build and deploy AI applications on Kubernetes. He has a great contribution to Fluid's landing on Baidu AI Cloud and made it used in production. Besides Fluid, he also works on container GPU sharing and GPU scheduling, etc.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=abowloflrf
v0.7.0:
Bug Fix: Fix update dataload status error
fluid-cloudnative/fluid#1433
v0.8.0
Bug Fix: Fix: runtime deletion issue
fluid-cloudnative/fluid#1971
Optimizations: Update dataset mountPoint min-length to 5
fluid-cloudnative/fluid#2010
Enhancement: Support leader election for controller and webhook
fluid-cloudnative/fluid#2011
fluid-cloudnative/fluid#2014
Enhancement: Update node /etc/updatedb.conf
config in csi plugin
fluid-cloudnative/fluid#2078
fluid-cloudnative/fluid#2075
Dear Fluid TOC members and maintainers,
I am proposing to have Weiwei Zhu(GithubID: zwwhdls) as a new maintainer.
Weiwei Zhu, is a fullstack engineer at Juicedata, maintainer of JuiceFS CSI Driver, focusing on the development of JuiceFS in cloud native scenarios and also works on combination of Fluid and JuiceFS.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=zwwhdls
Sep 16, 2022: Add ThinRuntime in order to make general storage easier to integrate Fluid.
Mar 31, 2022: Support JuiceFS Cloud Service edition in Fluid.
Mar 13, 2022: Add fluid app controller for serverless offline task lifecycle management.
Jan 1, 2022: Support Fluid fuse recovery feature.
Nov 22, 2021: Add a new runtime for fluid - JuiceFSRuntime
Talks:
KubeCon + CloudNativeCon Europe 2022 Office Hour
Fluid Community Meeting talks: Fluid Support Serverless Offline Task Lifecycle Management Design and Demonstration
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20220512)
Design Proposal for Fuse Recovery Feature
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20220120)
Blogs:
https://juicefs.com/zh-cn/blog/solutions/fluid-eci-juicefs
https://blog.hdls.me/16523685168321.html
https://juicefs.com/blog/cn/posts/fluid-with-juicefs
v0.7.0:
Add JuiceFSRuntime engine to fluid backend:
fluid-cloudnative/fluid#1047
Mountpoint recover automatically:
fluid-cloudnative/fluid#1263
Support juicefs in serverless:
fluid-cloudnative/fluid#1427
Add feature gate to csi:
fluid-cloudnative/fluid#1444
0.8.0:
Add fluid app controller:
fluid-cloudnative/fluid#1481
Add app label in dataload & skip dataload pod inject:
fluid-cloudnative/fluid#1542
Support juicefs cloud service:
fluid-cloudnative/fluid#1683
Add label selector in controller cache:
fluid-cloudnative/fluid#1785
Add configs in juicefsruntime:
fluid-cloudnative/fluid#1804
Add eventDriven & cache selector in JuiceFSRuntime Controller:
fluid-cloudnative/fluid#1844
Support tolerate in juicefsruntime:
fluid-cloudnative/fluid#2083
Add thinRuntime:
fluid-cloudnative/fluid#2098
Support emtpyDir for juicefs tiredstore volumeType:
fluid-cloudnative/fluid#2109
Add juicefs diagnose script and fix alluxio diagnose script:
fluid-cloudnative/fluid#2156
Add e2e test for juicefs:
fluid-cloudnative/fluid#2505
Feel free to +1 or -1, starting with my +1.
Thanks,
Rong
Dear Fluid TOC members and maintainers,
I am proposing to have Weiwei Zhu(GithubID: zwwhdls) as a new committer.
Weiwei Zhu, is a software engineer at Juicedata, Inc. She has made quite a lot of contribution to the Fluid open source project.
The commit history: https://github.com/fluid-cloudnative/fluid/commits?author=zwwhdls
Fluid Community Meeting talks: Fluid Support Serverless Offline Task Lifecycle Management Design and Demonstration
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20220512)
Design Proposal for Fuse Recovery Feature
(https://github.com/fluid-cloudnative/community/wiki/Meeting-Agenda-and-Notes#20220120)
Blogs:
https://blog.hdls.me/16523685168321.html
https://juicefs.com/blog/cn/posts/fluid-with-juicefs
v0.7.0:
Add JuiceFSRuntime engine to fluid backend:
fluid-cloudnative/fluid#1047
mount point recover automatically
fluid-cloudnative/fluid#1263
support juicefs in serverless
fluid-cloudnative/fluid#1427
add feature gate to csi
fluid-cloudnative/fluid#1444
0.8.0:
add fluid app controller
fluid-cloudnative/fluid#1481
add app label in dataload & skip dataload pod inject
fluid-cloudnative/fluid#1542
support juicefs cloud service
fluid-cloudnative/fluid#1683
add label selector in controller cache
fluid-cloudnative/fluid#1785
add code-gen in makefile
fluid-cloudnative/fluid#1796
add configs in juicefsruntime
fluid-cloudnative/fluid#1804
add eventDriven & cache selector in JuiceFSRuntime Controller
fluid-cloudnative/fluid#1844
Feel free to +1 or -1, starting with my +1.
Thanks,
Rong
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.