mplegendre / spindle Goto Github PK
View Code? Open in Web Editor NEWThis project forked from hpc/spindle
Scalable dynamic library and python loading in HPC environments
License: Other
This project forked from hpc/spindle
Scalable dynamic library and python loading in HPC environments
License: Other
============================================================================= == SPINDLE: Scalable Parallel Input Network for Dynamic Load Environments == ============================================================================= Authors: SPINDLE: Matthew LeGendre (legendre1 at llnl dot gov) W.Frings <W.Frings at fz-juelich dot de> COBO: Adam Moody <moody20 at llnl dot gov> Version: 0.10 (March 2017) Summary: =========== Spindle is a tool for improving the performance of dynamic library and python loading in HPC enviornments. Documentation: ============ https://computation-rnd.llnl.gov/spindle/ Overview: ============ Using dynamically-linked libraries is common in most computational environments, but they can cause serious problem when used on large clusters and supercomputers. Shared libraries are frequently stored on shared file systems, such as NFS. When thousands of processes simultaneously start and attempt to search for and load libraries, it resembles a denial-of-service attack against the shared file system. This "attack" doesn't just slow down the application, but impacts every user on the system. We encountered cases where it took over ten hours for a dynamically-linked MPI application running on 16K processes to reach main. Spindle presents a novel solution to this problem. It transparently runs alongside your distributed application and takes over its library loading mechanism. When processes start to load a new library, Spindle intercepts the operation, designates one process to read the file from the shared file system, then distributes the library's contents to every process with a scalable broadcast operation. Spindle is very scalable. On a cluster at LLNL the Pynamic benchmark (which measures library loading performance) was unable to scale much past 100 nodes. Even at that small scale it was causing significant performance problems that were impacting everyone on the cluster. When running Pynamic under Spindle, we were able to scale up to the max job size at 1,280 nodes without showing any signs of file-system stress or library-related slowdowns. Unlike competing solutions, Spindle does not require any special hardware, and libraries do not have to be staged into any special locations. Applications can work out-of-the-box do not need any special compile or link flags. Spindle is completely userspace and does not require kernel patches or root privileges. Spindle can trigger scalable loading of dlopened libraries, dependent library, executables, python modules and specified application data files. Compilation: ============ Please see INSTALL file in the Spindle source tree. Usage: ====== Put 'spindle' before your job launch command. E.g: spindle mpirun -n 128 mpi_hello_world
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.