Github lib support and vessel support about mops HOT 16 CLOSED

@tomijaga Merged and deployed 🚀

Soon I will create a post on the forum

from mops.

@tomijaga hey, great feature! 🔥

Or would it be better to concatenate the version with the repo's URL?

Let's use concatenation, it will be the same as in the config file.

I want to change it to

public type Dependency = {
    name: PackageName; // max 50
    repo: Text; // max unknown
    version: Text; // max 20
};

Ok, you can set max to 250, I think that's quite enough and can be changed later.

from mops.

Yh, this works. I plan to only save the org and package name in the canister
(repo = "NatLabs/Itertools.mo"), so the size limit can be reduced in future.

I think it can be saved as is, we won't have any memory problems because of this. Also, it will not be necessary to convert from one format to another and vice versa.

from mops.

Other options would be to add additional fields to the Dependency type or create a new type specifically for GitHub dependency packages. What do you think the best approach is?

I can suggest an approach where the version field is empty for github packages and just save the version in repo field, like https://github.com/NatLabs/Itertools.mo#v0.1.0

For example in mops.toml:

Itertools = "https://github.com/NatLabs/Itertools.mo#v0.1.0"

on backend:

{
  name = "Itertools";
  repo = "https://github.com/NatLabs/Itertools.mo#v0.1.0";
  version = "";
}

and skip version check for github packages on the backend.

from mops.

Do you think it's best to remove the progress bar for the GitHub downloads or to make a pre-request to GitHub for the size of each repo before downloading them?

If there is a way to make a request to get the repo size, that would be nice.
A fake progress bar is fine too.

As an option, start downloading with an approximate repo size of 2mb and in parallel make a request to get the real size of the repo and then update the size.

from mops.

Ok, you can set max to 250, I think that's quite enough and can be changed later.

Yh, this works. I plan to only save the org and package name in the canister
(repo = "NatLabs/Itertools.mo"), so the size limit can be reduced in future.

from mops.

I think it can be saved as is, we won't have any memory problems because of this.

Okay, that makes sense. It was just some premature optimizations 😅
I'll update the code to save the whole url.

Also, it will not be necessary to convert from one format to another and vice versa.

Do you mean these conversions: #6 (comment), #6 (comment)

from mops.

The GitHub packages have multiple ways a user can specify a package version. The user can specify a releases version number with a v prefix (v0.1.23), a branch name and even a checksum hash for a release 🥲 .
I'm saving each of them in theversion field of the Dependency like vessel does, but it causes issues with the backend's version validation. It rejects any value that doesn't fit this format 0.1.23-beta.*

I think a solution is to update the version validation to allow alphanumeric values when the repo field has a value.

    let dep = {
        name = "Itertools";
        repo = "https://github.com/NatLabs/Itertools.mo";
        version = "main";
    }

    if (dep.repo.size() == 0){
         Version.validate(dep.version)
    }else{
        // store data in canister
    }

Other options would be to add additional fields to the Dependency type or create a new type specifically for GitHub dependency packages. What do you think the best approach is?

from mops.

Do you mean these conversions: #6 (comment), #6 (comment)

I thought there were conversions like https://github.com/NatLabs/Itertools.mo <-> NatLabs/Itertools.mo

from mops.

I can suggest an approach where the version field is empty for github packages and just save the version in repo field, like https://github.com/NatLabs/Itertools.mo#v0.1.0

Okay, this works as well. It's probably best to have them in a single field so it's immediately identifiable.

I thought there were conversions like https://github.com/NatLabs/Itertools.mo <-> NatLabs/Itertools.mo

Yes, I was making those conversions as well. I just wanted to clarify between the two.

from mops.

I've added functionality to display the progress bar while the GitHub packages are downloading, but most packages don't have the content-length sent in the request.

All the repos except this one, https://github.com/dfinity/motoko-base#a6301031d21449d38eb64a72e3e9dd0ad8447a86, had the content-length. Unfortunately, I didn't figure out why.

I put a default size of 2MB for all downloads, but it's usually way over the size of an avg repo, so the progress bar barely gets filled during the download.

Do you think it's best to remove the progress bar for the GitHub downloads or to make a pre-request to GitHub for the size of each repo before downloading them?

from mops.

After testing again, I found that the repo size is retrieved in the request.
It just had a slight delay which was causing issues before.
It works fine now that the default repo size is 2mb.

It's been great working on this feature this past week. 😁
I've implemented all the features I planned for the vessel support.
If there is any other thing you want to add, let me know.
If not, I think this is ready for a full review.

from mops.

It's been great working on this feature this past week. 😁
I've implemented all the features I planned for the vessel support.

Great job! Great feature, will allow developers to make a smooth transition from vessel to mops

If not, I think this is ready for a full review.

Done. Found performance issues in the sources command.

from mops.

While importing sha2 works fine, when I went to try a specific github repository: https://github.com/dfinity/ICRC-1/blob/main/ref/Account.mo it persists an error "file "~.../.mops/_github/icrc1accountconverter | icrc1ac#main | icrc1#e953a" (for package icrc1accountconverter | etc) does not exist Motoko(M0012)."

I also tried forking and adding a toml package declaration, but that did not work either.

I'm aware there's an existing implementation, I'm just curious on how the importing from Github Motoko packages is supposed to work.

from mops.

Hi. Not sure if that repo is supposed to use in projects, probably you want to try https://github.com/NatLabs/icrc1?

dfinity/ICRC-1 will not work because package files located in ref dir and not in src.
If you fork and put mops.toml with:

[package]
baseDir = "ref"

probably will work.

from mops.

from mops.