nim-lang / atlas Goto Github PK
View Code? Open in Web Editor NEWThe Atlas Package cloner. It manages an isolated workspace that contains projects and dependencies.
License: MIT License
The Atlas Package cloner. It manages an isolated workspace that contains projects and dependencies.
License: MIT License
Consider using git clone --depth=1
for pulling dependencies.
This:
For further possible interacting with the pulled repos there's git fetch --unshallow
.
✝ ~/code/timbrel ✝ atlas use file://../saintnim/
[Info] (/Users/saint/code/timbrel) is the current workspace
[Info] (/Users/saint/code/timbrel/timbrel.nimble) updated
[Info] (timbrel) installing dependencies for timbrel.nimble
[Info] (print) updated package to a9df499a441d997ab9c8658b014e84c561f5528a
/Users/saint/code/atlas/src/atlas.nim(583) atlas
/Users/saint/code/atlas/src/atlas.nim(576) main
/Users/saint/code/atlas/src/atlas.nim(474) main
/Users/saint/code/atlas/src/atlas.nim(222) installDependencies
/Users/saint/code/atlas/src/atlas.nim(197) traverseLoop
/Users/saint/code/atlas/src/depgraphs.nim(564) getCfgPath
/Users/saint/.choosenim/toolchains/nim-2.0.2/lib/system/fatal.nim(53) sysFatal
Error: unhandled exception: index out of bounds, the container is empty [IndexDefect]
I can't seem to add it this way at all. I'm able to get it to work by manually editing the nim.cfg
and adding --path:"/Users/saint/code/saintnim/src"
Any idea how to fix this? Thanks!
koch.nim
commands.atlas use ormin
and watch Atlas do some stuff with bootstrapping Nim.nimble install https://github.com/nim-lang/atlas@\#head
/tmp/nimble_1894/githubcom_nimlangatlas_#head/src/lockfiles.nim(11, 12) Warning: imported and not used: 'sha1' [UnusedImport]
Success: atlas installed successfully.
Tip: 7 messages have been suppressed, use --verbose to show them.
packageparser.nim(395) readPackageInfo
Error: Could not read package info file in /home/runner/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2/atlas.nimble;
... Reading as ini file failed with:
... Invalid section: .
... Evaluating as NimScript file failed with:
... /home/runner/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2/atlas.nimble(2, 12) Error: cannot open file: src/pkgversion
Locally I get:
Error: The package inside "/Users/jaremycreechley/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2" is invalid.
Details: Could not read package info file in /Users/jaremycreechley/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2/atlas.nimble;
... Reading as ini file failed with:
... Invalid section: .
... Evaluating as NimScript file failed with:
... /Users/jaremycreechley/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2/atlas.nimble(2, 12) Error: cannot open file: src/pkgversio
I've tried setting up atlas for a project, but I don't know if I like that there is no way (that I know of) to connect an atlas workspace to the root toolchain folder of the host system (or any other arbitrary folder).
I understand if this is by design, and it does make a lot of sense if this is the case.
What I was thinking is having the option to (explicitly, manually) setup paths where atlas can search for dependencies, so that the same libraries don't end up duplicated 10 times because of having 10 different workspaces.
My idea would be:
1. Setup alternative folder to search for (explicitly)
2. `atlas use mydepend`
2.1 atlas searches for the dependency in the alternative folders, if they are any
---> does not redownload the library, just uses the ones in the other folder
---> choose by prioritizing the last folder found, like nim.cfg does (or first, or whatever other rule)
2.2 no alternatives found. atlas continues with the current behavior where the libraries are duplicated once for each workspace
That would be my feature request if this is not possible.
Or just my question if the feature exists and I haven't found it (sorry if that's the case).
I've spent way more time than I care to think about trying to figure out a generic cross compiling scheme. conan
and conda
sorta offer this but not very well.
All of that said, Nim already has decent cross compiling support but it's just not very centralized and requires each dev to sorta re-invent things. It also depends on each dev to figure out how to get and install cross compilers.
It'd be amazing if atlas
could provide a way to provide a sequence like "install cross-xc compilers then use them to build" sequence. From what little I've seen of Atlas this wouldn't seem to impractical too add. Especially if it could be added as just another builder / extractor with possibly a bit of sugar on the nim.cfg side. Perhaps another concept of a "provider" could be added too, e.g. recipe for building and installing a specific Nim compiler or non-Nim compile tools.
I'm creating this issue to find out if others would be interested in as well. It's something I'd be keen on adding if it'd be accepted. Though it'd have to be non-invasive and not complicate the simple build path, otherwise I think it'd be going down the wrong path.
I've tried the following:
atlas use argparse==4.0.1
atlas use argparse=4.0.1
atlas use [email protected]
atlas use 'argparse == 4.0.1'
atlas use 'argparse = 4.0.1'
How do tell it to use a particular version?
If there are two projects of the same name but of different origin (i.e coming from different URLs), namespace the projects. This means to turn the project name into $project.$author.$github
.
I upgraded from 2.0.2 to 2.0.8 and notice a severe regression : atlas use
no longer generate a correct nim.cfg files.
Workspace configuration saved to atlas.workspace
uses absolute file paths which breaks portability/dual boot usage.
Possible solution: save the ondisk
property of the workspace as a relative path to the config file, save the ondisk
property of the packages as relative paths to the workspace directory.
And not just the available git tags.
It must be reworked in general, the existing code is not clean enough.
The parser that process project.nimble
is what I call an extractor
. Custom extractors allow to extend this mechanism to cover other package managers, including ones for different programming languages. An extractor can be written in any programming language you like.
The entry in the configuration would look somewhat like:
"extractor": {"pattern": "requirements.txt", "language": "python", "parser": "<url to extension>", "resolver": "SemVer"}
I'm trying to make atlas understand local development packages that already exist in the workspace.
I don't want to use URL, the point of the folder structure is to not need url at all
But im having a hard time making atlas understand this structure
The documentation reads:
A workspace contains one or multiple "projects". These projects can use each other and it is easy to develop multiple projects at the same time.
https://github.com/nim-lang/atlas/blob/master/doc/atlas.md#projects
But I'm not quite getting how that is going to happen if atlas use mypkg
is not finding the folder right next to itself, which exists inside the workspace structure already.
I know about atlas use /path/to/package
.... but that is adding requires "file://path/to/package"
to the nimble file, and that is incompatible syntax with nimble, which makes nimble completely break and not able to run at all.
(I may not want to use nimble myself, but I cannot force that choice on everybody, specially with atlas being in such early dev state. So I don't want to make my nimble files unusable just because I chose atlas)
The main problem is that I don't want to have my dependencies in a subfolder. I have to maintain a set of modules like this:
And the list is only growing. Which is why I'm turning to atlas, in the hopes that I can finally use nimble develop
like I used to do without issues, but since that feature is very broken in Nimble, I'm hoping that atlas can solve the problem.
I have even tried nim.cfg with both nimble and raw nim c
and no nimble at all... and they both fail to resolve the sub-dependencies. Which is extremely frustrating, because it means I have to reupload the package, remove it from local, redownload my own package, and repeat.... for every single line I change, which is just ridiculous.
The main goal is being able to keep my current folder structure of dependencies all stored at the root of the workspace, and make each dependency understand that its dependencies exist in the folder right next to them.
How can I approach this with atlas?
Atlas should have another option,
to just put local paths in the nim.cfg
.
This way we could developing a module and use it directly in the consuming application.
For example it could be:
atlas develop /path/to/module
This would then add the path to the module to the nim.cfg
.
It could also add this to the nimble file?
We currently have 2 generators:
nim.cfg
file.project.nimble
This mechanism could be extended so that it can generate other files like a requirements.txt
. Ideally it also supports to setup/edit environment variables or the command that is used for a "builder" (see the ticket about builders).
Running atlas use norm
in a workspace does not fetch db_connectors
required by Norm's dependencies lowdb
.
In order to get norm to compile it is necessary to install db_connectors
manually.
The fetcher is the component that does the following:
Myself and several others have used the "vendoring" style where a sub-directory is used as the Atlas workspace. This is a very common pattern for many language package managers like Npm or Elixir's mix and each project can have it's own versions checked out. Golang also uses a vendor directory for project specific deps pinning in addition to it's global GOPATH
workspace.
For many projects it's convenient to setup the main parent repo as a Nim project that also contains the configs to install dependencies for that project rather than needing to create an explicit workspace. Often this is most useful for folks who want to clone a repo, install the deps locally and try out a project. This is pretty big for people who see a Nim project and just want to "try it out". Though this can be handy for a user who has a workspace but doesn't want to pollute it with some project's dependencies before they check it out. Lastly, it also enables a smooth transition path for Status project which use the nimbus-build-system that uses git submodules in a vendor
folder.
The previous implementation of this in Atlas was very hacky and searched directy subdirectories in the current folder for workspaces in normal workspaces weren't found. That's problematic for lots of reasons and caused issues with normal atlas workspaces containing other workspaces. However, there should be a couple of solutions to enabling a vendoring style setup with Atlas properly.
Here's a couple of feature goals:
./vendor
folderHere's a few approaches that have come up in a previous PR and I've thought of trying:
atlas.config
project config file which could specify a default vendored workspace subdirectory in addition to other per-project configsatlas.vendor
file that just contains the path to a vendor folder (similar to some of git's setups)atlas.workspace
configuration that specifies a vendored workspaceatlas --workspace=./vendor xyz
-l
uses a pre-defined folder like vendor
as the workspace, similar to the atlas -g
support for globalI'm partial towards either option 1 or option 5 of having an atlas.config
or a -l
local option, respectively. They're both explicit.
Having a workspace config in the current project seems sort of confusing with Atlas's standard usage. The atlas.vendor
link file are sort of annoying in Nimble and Git.
Many commands like use
and pin
and rep
can work on a Workspace or an individual project. We should offer project.use
and workspace.use
as aliases that enforce a respective interpretation. If the qualified command syntax is not used then if the current working directory is a project the command is mapped to project.command
and else if the working directory is a workspace it is mapped to workspace.command
otherwise it is an error as it's not clear what was meant.
Notes for re-adding support for namespaces.
It seems the proc's pkgUrlToDirname
and extractProjectName
both need updating and maybe refactoring. Also, the GitHub search will need re-adding if it's not present. It would be good to review the package resolution in version 0.6.3 to ensure all the edge cases are handled.
Also, it'll likely be a good time to fix up the on disk directory paths to support relative paths from the workspace.
It'd be ideal to write unit tests for this before setting up integration tests.
The update to the dependency graph broke this.
This breaks any projects using Atlas where users want to change to a different git repo for a dependency for testing or waiting for PR's to get merged.
Due to nimble moving the src path, and atlas having the opposite behavior, install hooks not created for atlas in mind will fail.
Presently the following results in micros
being installed in the workspace directory instead of .deps
.
mkdir workspace
cd workspace
atlas init --deps=".deps"
atlas use micros
Looking at the atlas.workspace
file deps
is not populated:
{
"deps": "",
"overrides": "",
"plugins": "",
"resolver": "SemVer",
"graph": null
}
Even if one populates deps:".deps"
using atlas use micros
still installs it in the workspace root.
It would be nice to be able to do "atlas use [email protected]:..." on private packages
$ atlas init
$ atlas clone arraymancer
$ atlas use nimjl # Because it uses arraymancer as a dependency
This creates 2 repo arraymancer : "Arraymancer" and "arraymancer" (capital A) difference.
IMO that shouldn't be possible & package name should be case insensitive. I don"t want to have 3 different packages "Tools", "tools", "TOOLS" that are different things
Atlas currently lacks any support for a "build" step. Ideally the implementation is flexible enough to integrate cmake
etc.
Choosenim is the best way to have the latest Nim version around while still being able to jump to devel versions or a local copy.
Is it though? Atlas can do the same already.
This comment seems to imply that you can use atlas as a choosenim replacement. There is anything about this in the README? How does it work? What commands does it need to run?
Nimble parser needs to handle this properly:
/Users/jaremycreechley/.atlas/nim-testutils/testutils.nimble(10, 17) Error: assignments to 'srcDir' must be string literals
Caused by #
in:
bin = @["ntu"]
#srcDir = "testutils"
This happend using nim v2.0.0.
✘-INT Clonkk ~$ nimble install https://github.com/nim-lang/atlas
Downloading https://github.com/nim-lang/atlas using git
Warning: No nimblemeta.json file found in /home/rcaillaud/.nimble/pkgs2/nimlsp-0.4.4-b624dd472b343ca6dcb5e825bbd47f7d1388f077
Verifying dependencies for atlas@0.8.0
Installing atlas@0.8.0
Building atlas/atlas using c backend
/tmp/nimble_128084/githubcom_nimlangatlas/src/packagesjson.nim(10, 8) Warning: imported and not used: 'context' [UnusedImport]
/tmp/nimble_128084/githubcom_nimlangatlas/src/sat.nim(559, 3) Warning: unreachable else, all cases are already covered [UnreachableElse]
/tmp/nimble_128084/githubcom_nimlangatlas/src/nimblechecksums.nim(9, 12) Warning: use command `nimble install checksums` and import `checksums/sha1` instead; sha1 is deprecated [Deprecated]
/tmp/nimble_128084/githubcom_nimlangatlas/src/depgraphs.nim(180, 41) Error: 'nimbleContents' is not a mutable location; it cannot be moved
Prompt: Build failed for 'https://github.com/nim-lang/[email protected]', would you like to try installing 'https://github.com/nim-lang/atlas@#head' (latest unstable)? [y/N]
atlas use malebolgia
[Info] (.) is the current workspace
[Info] (malebolgia) could not resolve by name or repo; searching GitHub
[Error] (malebolgia) package not found by github search
[Warning] (malebolgia) cannot resolve package name
I expected malebolgia
to be found. I expected the readme to have a package that is easy to find.
Instead of "There were problems" collect warnings and errors and output them at the end of a run. Then the user would not have to scroll through the potentially long log to see what went wrong.
Using non-existing version or impossible version constraint doesn't raise an error :
If I have a project with a "myproject.nimble" file containing the following :
requires "arraymancer >= 0.10.0 & < 0.7.20"
And I call "atlas use myproject.nimble", atlas doesn't fail when it is impossible to resolve the version.
Reverting this commit fixes the issue. Error log:
atlas clone https://github.com/planetis-m/raylib-game-template
[Info] (.) is the current workspace
Cloning into 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
Cloning intp 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
Cloning into 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
Cloning into 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
Cloning 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
[Error] (https://github.com/planetis-m/raylib-game-template) exernal program failed: git clone --recursive https://github.com/planetis-m/raylib-game-template https://github.com/planetis-m/raylib-game-template
Found this bug updating one of my earlier projects with a lockfile. Looks like instead of using the URL, atlas for some reason falls back to the local dir.
I plan to add some integration tests for lockfiles as I don't believe we have any currently. Or if we do, there needs to be more.
Expected lockfile:
{
"items": {
"stew": {
"dir": "$deps/nim-stew",
"url": "https://github.com/status-im/nim-stew",
"commit": "7afe7e3c070758cac1f628e4330109f3ef6fc853",
"version": "0.1.0"
},
...
}
But getting:
"items": {
"stint": {
"dir": "$deps/nim-stint",
"url": "file:///Users/jaremycreechley/projs/status/nim-codex-dht/vendor/nim-stint",
"commit": "86621eced1dcfb5e25903019ebcfc76ed9128ec5",
"version": "0.0.1"
},
...
}
Would it make sense to move AtlasContext into it's own module like context.nim
?
It'd let the main atlas.nim
be broken up into more self contained modules -- something like a cloning.nim
module for handling that functionality. I think it'd help making unit tests as well and to use logging more places.
Atlas errors out on a project I'm trying to include: stack_string. Since stack_strings
is a valid module it should be supported.
Currently, it is possible to output the dependencies with a graph representation using GraphViz.
See: https://github.com/nim-lang/atlas/blob/master/src/atlas.nim#L446
« Ideally Atlas would output a graph representation that is not only useful for GraphViz. Maybe in JSON and then some external tool would render that in the browser. » Araq from this forum post.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.