swift-lang / swift-t Goto Github PK
View Code? Open in Web Editor NEWSwift/T: High Performance Parallel Scripting Language
Home Page: http://swift-lang.org/Swift-T
License: Apache License 2.0
Swift/T: High Performance Parallel Scripting Language
Home Page: http://swift-lang.org/Swift-T
License: Apache License 2.0
From @GoogleCodeExporter on April 22, 2015 19:4
The current API for interacting with local files from Tcl is crude and
undocumented
We should fix this.
Original issue reported on code.google.com by [email protected]
on 19 Sep 2014 at 7:09
Copied from original issue: j-woz/exm-issues#743
I will fix it.
From @GoogleCodeExporter on April 22, 2015 19:2
Ketan ran this and got a misleading MPI-related error message. We should be
able to handle this better.
Try
https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/swift-galaxy/swift/script.swift
r7833
Original issue reported on code.google.com by [email protected]
on 7 May 2014 at 3:20
Copied from original issue: j-woz/exm-issues#683
From @GoogleCodeExporter on April 22, 2015 19:4
Try building fork executor.
Original issue reported on code.google.com by [email protected]
on 15 Aug 2014 at 8:59
Copied from original issue: j-woz/exm-issues#734
From @GoogleCodeExporter on April 22, 2015 19:1
https://bitbucket.org/saws/saws/wiki/Home
Original issue reported on code.google.com by [email protected]
on 14 Mar 2014 at 9:28
Copied from original issue: j-woz/exm-issues#623
From @GoogleCodeExporter on April 22, 2015 19:2
Our current intended model is that the user provides PROCS and PPN and we
divide PROCS/PPN to obtain NODES. SLURM does not currently follow this model.
Original issue reported on code.google.com by [email protected]
on 7 Apr 2014 at 7:32
Copied from original issue: j-woz/exm-issues#648
From @GoogleCodeExporter on April 22, 2015 19:4
Currently these don't support parallel tasks.
Original issue reported on code.google.com by [email protected]
on 7 Nov 2014 at 8:31
Copied from original issue: j-woz/exm-issues#753
From @GoogleCodeExporter on April 22, 2015 19:5
We don't free blob values if they are stored in local arrays. E.g. if we have
a leaf function that accpets an array of blobs. This leads to memory leaks.
E.g. see test 851
Original issue reported on code.google.com by [email protected]
on 20 Jan 2015 at 3:23
Copied from original issue: j-woz/exm-issues#761
From @GoogleCodeExporter on April 22, 2015 19:5
xpt_init2 is hard-coded into turbine.tcl .
Original issue reported on code.google.com by [email protected]
on 4 Mar 2015 at 4:59
Copied from original issue: j-woz/exm-issues#786
From @GoogleCodeExporter on April 22, 2015 19:3
Should we close stdin in the Turbine startup script?
Cf. exm-users 5/9/2014.
Original issue reported on code.google.com by [email protected]
on 27 May 2014 at 2:34
Copied from original issue: j-woz/exm-issues#695
From @GoogleCodeExporter on April 22, 2015 19:3
The current syntax for foreign tcl functions is clunky and doesn't support all
use cases well since it only supports a few fixed formats. It also requires
extending the syntax to add other types of foreign functions. Something
cleaner might be:
foreign tcl (int o) f (string x) {
pkg = "mypkg";
pkg_version = "0.0";
func = "myfunc"
}
foreign tcl (int o) f (string x) {
template = "set <<o>> <<x>>"
}
Original issue reported on code.google.com by [email protected]
on 22 May 2014 at 10:14
Copied from original issue: j-woz/exm-issues#694
From @GoogleCodeExporter on April 22, 2015 19:3
The current syntax for foreign tcl functions is clunky and doesn't support all
use cases well since it only supports a few fixed formats. It also requires
extending the syntax to add other types of foreign functions. Something
cleaner might be:
foreign tcl (int o) f (string x) {
pkg = "mypkg";
pkg_version = "0.0";
func = "myfunc"
}
foreign tcl (int o) f (string x) {
template = "set <<o>> <<x>>"
}
Original issue reported on code.google.com by [email protected]
on 22 May 2014 at 10:14
Copied from original issue: j-woz/exm-issues#694
From @GoogleCodeExporter on April 22, 2015 19:4
Currently there's not really much documentation about buffer sizes for ADLB_Get
and ADLB_Put. In practice it depends on the buffer provided to ADLB_Get being
large enough to handle any task the user code calls ADLB_Put for. If this is
not true, a buffer overflow results. We should probably modify the ADLB_Get
signature to take an additional argument specifying the size of the provided
buffer.
The right thing to do would be to only match it to tasks that are small enough.
This is a minor pain and will require modifying the server task matching logic
a bit.
A less nice solution would be to just say that the user code is required to
post buffers big enough for any task and abort the program if we detect an
issue.
Original issue reported on code.google.com by [email protected]
on 19 Jun 2014 at 4:08
Copied from original issue: j-woz/exm-issues#712
From @GoogleCodeExporter on April 22, 2015 19:4
It would be helpful in some situations to be able to link with a different
command to the compile command. Currently some build files assume that $(CC)
is the linker command.
Original issue reported on code.google.com by [email protected]
on 11 Jul 2014 at 6:30
Copied from original issue: j-woz/exm-issues#720
From @GoogleCodeExporter on April 22, 2015 19:4
The vanilla machine files, e.g. machine.vanilla-gcc.sh, rely on sniffing
dependencies, e.g. stc, turbine, tclsh from the user's path. Then genleaf
makes certain assumptions about the layout of these components, e.g. the tcl
layout is: $TCL_HOME/bin/tclsh8.6 $TCL_HOME/lib/libtcl8.6.so.
There are problems with this in several scenarios:
* The user does not have these components on their path.
* The user has the wrong version of the component on their path (e.g. a Tcl
different to the one Turbine was compiled with)
* The user's system does not match the expected layout. This is true for
common distributions, including Ubuntu linux's Tcl packages.
I just ran into these issues trying to use genleaf on my system and gave up. I
don't think I can use it without reconfiguring Turbine to use a different Tcl
or rewriting the script.
We've already got a mechanism to handle all this in the configure script: why
not just use the information in turbine-build-config.sh?
Original issue reported on code.google.com by [email protected]
on 31 Jul 2014 at 9:20
Copied from original issue: j-woz/exm-issues#727
From @GoogleCodeExporter on April 22, 2015 19:5
Basic enums in Swift.
Original issue reported on code.google.com by [email protected]
on 5 Feb 2015 at 9:34
Copied from original issue: j-woz/exm-issues#770
From @GoogleCodeExporter on April 22, 2015 19:3
The MPI_T interfaces should allow us to determine the unexpected message queue
length. This is useful to understand whether servers are congested.
Original issue reported on code.google.com by [email protected]
on 10 Jun 2014 at 12:10
Copied from original issue: j-woz/exm-issues#705
From @GoogleCodeExporter on April 22, 2015 19:2
We need to define calling conventions for calling C functions with Swift/T
data. I'm enumerating some of the initial issues that I can think of.
+-------------------------+
| Function signatures |
+-------------------------+
- Do we want to pass in any additional metadata arguments? E.g. Swift/T state,
MPI communicators, etc
- C input arguments match Swift/T input arguments
- How to support multiple outputs?
-> Ideally single outputs can be handled as return value
- How to support varargs??
+-------------------------+
| Data types |
+-------------------------+
Scalar/simple types
- numeric types -> native ADLB representation (int64_t, double, etc)
- strings -> null-terminated
Uninterpreted data
- blob -> struct? or two arguments?
Composite types
- Arrays (sparse) -> ???
- Structs -> ???
+-------------------------+
| Options |
+-------------------------+
1. Pass native ADLB representations into C function
2. Unpacked representations based (loosely?) on SWIG conventions
3. Unpacked representations not based on SWIG
Original issue reported on code.google.com by [email protected]
on 2 May 2014 at 7:25
Copied from original issue: j-woz/exm-issues#682
From @GoogleCodeExporter on April 22, 2015 19:4
If user cannot write to out.tcl because of permissions, you get a Java stack.
Original issue reported on code.google.com by [email protected]
on 27 Aug 2014 at 8:38
Copied from original issue: j-woz/exm-issues#741
From @GoogleCodeExporter on April 22, 2015 19:2
Ketan ran this and got a misleading MPI-related error message. We should be
able to handle this better.
Try
https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/swift-galaxy/swift/script.swift
r7833
Original issue reported on code.google.com by [email protected]
on 7 May 2014 at 3:20
Copied from original issue: j-woz/exm-issues#683
The basic cause is that we're calling out to system, which requires taking our nicely tokenized args, munging them into a single string, and hoping that system will re-tokenize the string back into the original form. This is possible if you escape the string correctly (turbine::sync_exec doesn't).
Test 6392 is one example where this breaks. There will be others.
We could avoid that whole mess if we just call execvp() or similar with our tokenized args instead http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
From @GoogleCodeExporter on April 22, 2015 19:3
We build nicely out of the box with no configuration if we're compiling against
mpich. OpenMPI requires modifying MPI library names etc in turbine. Should be
able to build against system OpenMPI or compiled-from-source OpenMPI with the
same minimal effort.
Original issue reported on code.google.com by [email protected]
on 13 Jun 2014 at 1:55
Copied from original issue: j-woz/exm-issues#706
From @GoogleCodeExporter on April 22, 2015 19:5
There is a bug in the frontend of the compiler where variable definitions from
imported modules are not visible in modules that imported them. I believe the
issue is the order in which the files are compiled. See test 061
Original issue reported on code.google.com by [email protected]
on 2 Mar 2015 at 8:16
Copied from original issue: j-woz/exm-issues#784
From @GoogleCodeExporter on April 22, 2015 19:5
Currently the order is:
type the_new_type the_old_type;
typedef alias_for_old_type the_old_type;
In C the order is reversed:
typedef the_old_type alias_for_old_type;
Alternatives:
1. Do nothing
2. Swap them
3. Support both - infer which is meant
4. Detect C ordering, suggest user swaps it (either error or warning)
Alt 1:
Pros:
* Easy
* Backwards compatible with existing scripts
* Consistent with struct declarations: type <new type name> <definition of type>
type the_new_type { ... }
type the_new_type the_old_type
Cons:
* Some ongoing confusion for developers familiar with C
Alt 2:
Pros:
* Consistent with C
Cons:
* Somewhat inconsistent with the struct declaration order
* Breaks backwards compatibility
Alt 3:
Pros:
* Whichever alternative user writes will work - no read error->edit->compile
cycle required
Cons:
* Some minor implementation work
* More than one way to do same thing - overhead of documentation, inconsistency
between user's code, etc
* Errors messages may be slightly less helpful, since we don't know which way
was intended.
Alt 4:
Pros:
* Backwards compatibility/consistency (see Alt 1)
* No changes to language/docs
Cons:
* Some minor implementation work
* Requires C user to do another read warning/error -> edit -> compile cycle
Original issue reported on code.google.com by [email protected]
on 6 Feb 2015 at 3:57
Copied from original issue: j-woz/exm-issues#773
From @GoogleCodeExporter on April 22, 2015 19:1
http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/projekt
e/vampirtrace
Original issue reported on code.google.com by [email protected]
on 16 Jan 2014 at 5:01
Copied from original issue: j-woz/exm-issues#604
From @GoogleCodeExporter on April 22, 2015 19:4
Swift/T coasters is functional, but the user experience could be greatly
improved
* One-step start of the Coaster service through Turbine or Swift/T
* Better packaging - don't bundle separate Swift/K distro
* Improved configuration mechanism - file?
Original issue reported on code.google.com by [email protected]
on 3 Nov 2014 at 3:23
Copied from original issue: j-woz/exm-issues#750
From @GoogleCodeExporter on April 22, 2015 19:3
E.g. if we're in the ADLB_Store(id, ..) call, we should be able to pass a debug
symbol in as an argument. The compiler knows this info when it's generating
the code, and we'd like to include this info in log/error messages.
Original issue reported on code.google.com by [email protected]
on 3 Jun 2014 at 9:02
Copied from original issue: j-woz/exm-issues#696
From @GoogleCodeExporter on April 22, 2015 19:5
Maybe we should have the option to configure (either globally or per-variable)
the policy for placing data. E.g. on the nearest server or randomly.
Thoughts, Justin?
Original issue reported on code.google.com by [email protected]
on 17 Apr 2015 at 3:23
Copied from original issue: j-woz/exm-issues#794
From @GoogleCodeExporter on April 22, 2015 19:5
Using this syntax and finding it pretty concise but still redundant:
file list<argv("list")>;
Wondering if this would be neat:
main (file list, string name, int n) { ... }
This would automate the use of argv and type conversion, creating the
appropriate local variables.
Original issue reported on code.google.com by [email protected]
on 23 Feb 2015 at 9:58
Copied from original issue: j-woz/exm-issues#782
From @GoogleCodeExporter on April 22, 2015 19:3
As did OmpSs
Original issue reported on code.google.com by [email protected]
on 9 Jun 2014 at 7:18
Copied from original issue: j-woz/exm-issues#703
From @GoogleCodeExporter on April 22, 2015 19:5
http://google-opensource.blogspot.com/2015/03/farewell-to-google-code.html
On August 24, 2015 it's going read-only
Original issue reported on code.google.com by [email protected]
on 12 Mar 2015 at 6:25
Copied from original issue: j-woz/exm-issues#788
From @GoogleCodeExporter on April 22, 2015 19:5
We have:
type MyStruct {
int x;
}
Justin suggested we should instead have:
struct MyStruct {
int x;
}
Alternatives:
1. Remove/deprecated "type" support
2. Parallel support for both forms
Pros:
* Familiarity for people coming from C
Cons:
* Adds a reserved keyword
* Another inconsistency with Swift/K to resolve
* (Alt 1) Breaks existing scripts
* (Alt 2) Leaves more than one way to do same thing - additional documentation
and mental overhead for developers
Original issue reported on code.google.com by [email protected]
on 6 Feb 2015 at 3:44
Copied from original issue: j-woz/exm-issues#772
From @GoogleCodeExporter on April 22, 2015 19:5
Jonathan Ozik ran into an issue where he linked against the non-system Python
during build but at runtime it picked up his system python. I guess Python
wasn't added to the rpath correctly.
Original issue reported on code.google.com by [email protected]
on 20 Apr 2015 at 2:17
Copied from original issue: j-woz/exm-issues#796
From @GoogleCodeExporter on April 22, 2015 19:3
We currently use a hash table to map from (id, subscript) to a list of
notifications. This works great for supporting lookup of a (id, subscript)
given that we know the pair we're interested in. However, it doesn't work for
the following:
* Locating all subscripts for a ID (e.g. if ID is closed, and want to work out
if any subscribes will never be fulfilled).
* Locating all subscripts that have (id, subscript) as a prefix.
A radix tree would work for this.
Original issue reported on code.google.com by [email protected]
on 5 Jun 2014 at 6:03
Copied from original issue: j-woz/exm-issues#699
From @GoogleCodeExporter on April 22, 2015 19:5
Release 0.8.0
Original issue reported on code.google.com by [email protected]
on 12 Mar 2015 at 2:29
Copied from original issue: j-woz/exm-issues#787
From @GoogleCodeExporter on April 22, 2015 19:2
In Hemant's NFHEDM app and possibly one of Ketan's apps we are caching data and
open file descriptors in user memory. The user has no way to finalize these
resources. Add a shutdown hook.
Original issue reported on code.google.com by [email protected]
on 15 Apr 2014 at 3:58
Copied from original issue: j-woz/exm-issues#665
From @GoogleCodeExporter on April 22, 2015 19:4
Add user documentation for checkpointing feature.
Original issue reported on code.google.com by [email protected]
on 25 Sep 2014 at 3:56
Copied from original issue: j-woz/exm-issues#745
From @GoogleCodeExporter on April 22, 2015 19:4
Currently Aget/AMGet have a limitation that you can't have simultaneous
requests for different work types - or more generally, that requests can't be
matched out of order.
Original issue reported on code.google.com by [email protected]
on 7 Nov 2014 at 8:35
Copied from original issue: j-woz/exm-issues#754
From @GoogleCodeExporter on April 22, 2015 19:2
What needs more documentation?
Calling a CUDA function from Swift/T.
Preferably a file/function name
http://www.mcs.anl.gov/exm/local/guides/leaf.html
Original issue reported on code.google.com by [email protected]
on 2 May 2014 at 6:52
Copied from original issue: j-woz/exm-issues#681
From @GoogleCodeExporter on April 22, 2015 19:4
Must be set before running swift-t, because by the time the settings script is
sourced, turbine-config has already been sourced. Fix this.
Original issue reported on code.google.com by [email protected]
on 14 Nov 2014 at 9:31
Copied from original issue: j-woz/exm-issues#755
From @GoogleCodeExporter on April 22, 2015 19:4
Very rarely, tests 355 or 425 at O1 will fail, usually when run under valgrind.
The problem turns out to be an idle detection bug related to notifications.
The situation is:
- Given 3 servers, Master, A, and B
- Master is checking idleness
- A does remote subscribe to data on B
- Idle check request comes before B has a chance to reply
- A says (incorrectly) that it is idle
- At some point before or after B says it is idle
This is unlikely to happen in practice with any frequency.
Original issue reported on code.google.com by [email protected]
on 13 Aug 2014 at 4:17
Copied from original issue: j-woz/exm-issues#731
From @GoogleCodeExporter on April 22, 2015 19:3
We don't get rank numbers on the BG/Q - I am providing an option for ADLB to
produce them.
Original issue reported on code.google.com by [email protected]
on 5 Jun 2014 at 2:46
Copied from original issue: j-woz/exm-issues#698
From @GoogleCodeExporter on April 22, 2015 19:2
system() as in Swift/K. Interested?
Original issue reported on code.google.com by [email protected]
on 9 Apr 2014 at 6:47
Copied from original issue: j-woz/exm-issues#653
From @GoogleCodeExporter on April 22, 2015 19:3
In place of issue 200. Provide some sort of support to watch particular
variables in user program. More scalable/less noisy than logging everything.
Original issue reported on code.google.com by [email protected]
on 13 Jun 2014 at 3:41
Copied from original issue: j-woz/exm-issues#707
From @GoogleCodeExporter on April 22, 2015 19:5
I was thinking about how to automatically distribute Swift arrays/ADLB
containers in such a way that we keep small containers on one server, but split
larger containers once they grow. I think we really want this to be
transparent to ADLB clients (unlike the old distributed containers design).
We could use consistent hashing for this - every time the master server for a
container reaches some threshold, increase the number of servers the container
is split between. Readers/writers could be forwarded to the correct server if
they attempt to read on master. This could potentially be a performance boost
if clients were able to cache the number of shards of the container (with hash
table/LRU) to avoid contacting the master.
I think there are some tricky aspects to it. First is managing the migration.
We don't want to block server/worker operations while it's in progress, but we
could potentially just block reads/writes to that particular container while
it's happening.
Second is reference counting. I think we probably want to have the master be
responsible for detecting when the refcount goes to zero... but we don't want
all refcount increments/decrements going to the master (that would eliminate
most of the perf/scalability benefits). We need some kind of scheme. Possibly
when the master has many refcounts it could delegate some of them to each
sub-container. If it's own refcounts went to zero for a container, it would
then have to try to gather additional refcounts from sub-containers.
Original issue reported on code.google.com by [email protected]
on 17 Apr 2015 at 5:03
Copied from original issue: j-woz/exm-issues#795
From @GoogleCodeExporter on April 22, 2015 19:4
There's an interaction between multiple worker types (issue 503) and worker
services (issue 524). Should the same service get started on all worker types
uniformly, or should there be different options for different worker types?
The "default" behaviour that will occur if we don't modify anything is that the
services will only get launched on CPU workers.
Original issue reported on code.google.com by [email protected]
on 26 Jun 2014 at 4:13
Copied from original issue: j-woz/exm-issues#714
From @GoogleCodeExporter on April 22, 2015 18:50
Detect circular dependencies leading to deadlock.
E.g.
a = b;
b = c;
c = a;
or with conditionals
c = a + 2;
if (...) {
a = b;
} else {
a = c;
}
The basic strategy to do this would be to construct a dataflow graph and then
check for cycles or dead-ends. E.g. in a procedure there will be some set of
variables A which start off with an initial value (including arguments, and
variables which are initialised to a constant value). If we follow all
"depends-on" links from a variable v back through the graph, all paths should
terminate at a variable in the set A. If there is a cycle, or if the path
terminates at an uninitialised variable, we have a deadlock. There are a
couple of complications I see:
* conditional statements - the value of the variable could come from one branch
or another, so need to have some kind of OR operator (like the Phi operator
from SSA) to represent this
* loops - I'm not sure what the implications of these are yet
* assigning to array members. Sometimes we can work out exactly which array
index is assigned, but in general we don't know, so we will have to have
special treatment in the dataflow graph for the unknown array indices
Original issue reported on code.google.com by [email protected]
on 8 Feb 2012 at 3:38
Copied from original issue: j-woz/exm-issues#79
From @GoogleCodeExporter on April 22, 2015 19:2
Use MCS machine Frisbee. Swift/T has apparently regressed on the Mac.
Yadu, are you interested in this?
Original issue reported on code.google.com by [email protected]
on 24 Apr 2014 at 3:42
Copied from original issue: j-woz/exm-issues#671
From @GoogleCodeExporter on April 22, 2015 19:5
The ADLB API lets you to provide an arbitrary array of work types, e.g. if you
wanted work types 5,1,25,4.
This is mostly ignored by our implementation of ADLB, which assumes types are
0,1,2,...,n.
I don't think there's that much value in supporting this remapping of types at
the ADLB level, since there's a fixed number of types and the same thing can be
just as well accomplished at the application level if the application really
wants to use some alternative scheme for numbering types. Maybe we should just
remove the feature from the API?
Original issue reported on code.google.com by [email protected]
on 17 Mar 2015 at 2:32
Copied from original issue: j-woz/exm-issues#789
From @GoogleCodeExporter on April 22, 2015 19:4
Provide a method to implement third-party command line executors.
Should be able to ship a third-party executor as a Swift include + Tcl package.
Constraints are:
* Must be able to dynamically load as Tcl package (current design can do this)
* May have declarations in user script/header script (TODO)
* Must not require changes to Turbine Tcl/C code to add new executor
* Must not require changes to STC to add new executor
I think the third-party executor would need:
- to implement the turbine_executor interface, and register itself with Turbine
when the package is loaded
- declare its existence in the Swift include and specify a "run" tcl proc that
should be called to launch the task
As a test to confirm it's working, we should be able to replace any
special-cased coaster logic in STC and Turbine with generic logic.
Original issue reported on code.google.com by [email protected]
on 15 Aug 2014 at 9:21
Copied from original issue: j-woz/exm-issues#735
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.