Git Product home page Git Product logo

mini_repo's Introduction

MiniRepo

⚠️ MiniRepo is deprecated in favour of an offical self-hosting solution from the Hex team, mix hex.registry build (available from Hex v0.21+).

MiniRepo allows self-hosting of Hex packages.

Features:

  • Pluggable storage. MiniRepo ships with following adapters:

    • Local filesystem

    • S3

  • Mirroring

  • Publishing packages via HTTP API

  • Hosting of multiple repositories and mirrors

    MiniRepo exposes following URLs for API and repository access:

    • http://some_url/api/

    • http://some_url/repos/

Learn more about Hex specifications here.

Setup

Clone and install dependencies:

git clone [email protected]:wojtekmach/mini_repo.git
cd mini_repo
mix deps.get

Start a development server:

iex -S mix

By default, the development server is configured with two repositories:

  • test_repo is a custom repository

  • hexpm_mirror is a mirror of the official Hex.pm repository, configured to only fetch package decimal.

Both repositories are configured to store files locally. See config/dev.exs for more information.

Make sure to also read "Deployment with releases" section below.

Usage

Once you have the MiniRepo server running, here is how you can use it with Mix or Rebar3.

Usage with Mix

Let's create a new package and publish it to our test_repo repository:

$ mix new foo
$ cd foo

Make following changes to that package's mix.exs:

  def project() do
    [
      app: :foo,
      version: "0.1.0",
      elixir: "~> 1.9",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      description: "Some description",
      package: package(),
      hex: hex(),
    ]
  end

  defp deps() do
    []
  end

  defp package() do
    [
      licenses: ["Apache-2.0"],
      links: %{}
    ]
  end

  defp hex() do
    [
      api_url: "http://localhost:4000/api/repos/test_repo",
      # make sure to change it, see `auth_token` in config/releases.exs
      api_key: "secret"
    ]
  end

Now publish the package:

$ mix hex.publish package

Finally, let's use this package from another project.

First, configure Hex to use the custom repository:

$ cd /path/to/mini_repo
$ mix hex.repo add test_repo http://localhost:4000/repos/test_repo --public-key priv/test_repo_public.pem

Now, create a new Mix project:

$ mix new bar
$ cd bar

And configure the dependency, note the repo configuration option.

  defp deps() do
    [
      {:foo, "~> 0.1", repo: "test_repo"}
    ]
  end

Since the development server includes a hexpm_mirror repo, let's try that too:

$ HEX_MIRROR_URL=http://localhost:4000/repos/hexpm_mirror mix hex.package fetch decimal 1.8.0

See Hex.pm guide on publishing packages and Hex docs, in particular mix help hex.config, for more information.

Usage with Rebar3

Let's create a new package and publish it to our test_repo repository:

$ rebar3 new lib baz
$ cd baz

Now, let's configure Rebar3 to use our custom repo, put the following into your global rebar3 configuration:

%% ~/.config/rebar3/rebar.config
{plugins, [rebar3_hex]}.
{hex, [
  {repos, [
    #{
      name => <<"test_repo">>,

      %% make sure to change it, see `auth_token` in config/releases.exs
      api_key => <<"secret">>,
      api_url => <<"http://localhost:4000/api/repos/test_repo">>,
      api_repository => undefined,

      repo_url => <<"http://localhost:4000/repos/test_repo">>,
      repo_organization => undefined,
      repo_public_key => <<"-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEAxfUmzcCs9+rHvGiTvethBN0dVNgvJKss2z48mMjgOd9owiBMvHWQ
wBSncGgZHbahVJbz3bRfvKVAi1mgWx1233MlWJHR+qc2iQyXKW35cYsUOJtGAgmM
10kLvKhxKXdMgJASb02logFVuz2Ov3a/blHGDSqH6HCXok7tUY6ZwRIv7+zsQTga
ttpaLngmgGA2vPUQjUHIDSR6+j65szripj7BLyzqfncCcZK0nKYalBkwcXbrOln0
FucLkxYiy1saxxJlfHQ9W7j9YmjbZDDmSgnbJfi2/WpOgclptthYNA9+OYbz9peD
X9EXqozUvq0yXdgoqOnUfzYTrFOHg/MIHQIDAQAB
-----END RSA PUBLIC KEY-----">>
    }
  ]}
]}.

And publish the package specifying the repo:

$ rebar3 hex publish -r test_repo

Finally, let's use this package from another project:

$ rebar3 new lib qux
$ cd qux

Add the dependency to the project's rebar.config:

{erl_opts, [debug_info]}.
{deps, [
  {baz, "0.1.0"}
]}.

And make sure the dependecy was correctly resolved:

$ rebar3 deps
(...)
===> Verifying dependencies...
baz (package 0.1.0)

See Hex.pm guide on publishing packages with Rebar3 and Rebar3 docs for more information.

Deployment with releases

It's recommended to deploy MiniRepo with Elixir releases.

Let's now assemble the release locally:

$ MIX_ENV=prod mix release
* assembling mini_repo-0.1.0 on MIX_ENV=prod
(...)

And start it:

PORT=4000 \
MINI_REPO_URL=http://localhost:4000 \
MINI_REPO_AUTH_TOKEN=secret \
_build/prod/rel/mini_repo/bin/mini_repo start

As you can see, some configuration can be set by adjusting system environment variables, see config/releases.exs

Warning: MiniRepo by default has very basic authorization uses pre-generated public/private keys for repository signing. Make sure to generate your own public/private keys and consider adding authentication that makes sense in your organization.

Also, see mix help release for general information on Elixir releases.

More information

See following modules documentation to learn more about given feature:

Contributing

The goal of MiniRepo is to provide a minimal server that implements Hex specifications. Why minimal? By keeping the project focused on bare minimum we hope it's easy to understand and serves as a good starting point for a more complete solution that makes sense in a given organization. A production grade system should include infrastructure for monitoring, backups, SSL, and more, not to mention features like user management, SSO and such, but that's out of the scope of MiniRepo project.

We welcome anyone to contribute to the project, especially around documentation and guides, but features specific to narrow set of users likely won't be accepted. For a complete and production-grade implementation see source code of Hex.pm, however keep in mind it's optimised for running the public registry for the community and thus have different features and constraints than a self-hosted solution might have.

License

Copyright (c) 2019 Plataformatec Copyright (c) 2020 Dashbit

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

mini_repo's People

Contributors

alesasnouski avatar anthonator avatar ebostijancic avatar ejpcmac avatar isaacsanders avatar tlux avatar wojtekmach avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mini_repo's Issues

Fails on get dependencies

I'm doing an installation according the manuals but it fails on get deps stage:

$ mix deps.get
** (CompileError) config/config.exs:1: module Config is not loaded and could not be found
    (elixir) lib/code.ex:232: Code.eval_string/3
    (mix) lib/mix/config.ex:220: Mix.Config.eval!/2

I've elixir v 1.9.2

timeout errors

I've relatively small set of libs to mirror. I've the following in my config file.

      # only mirror following packages
      only: ~w(decimal nimble_csv csv ex_aws ex_aws_s3),

But we get a lot of timeout errors and hence it never goes beyond csv package.
After errors it restarts but it starts again from the start. And then again timeout error occurs on csv package only.

19:21:54.360 [error] GenServer :hexpm_mirror terminating
** (stop) exited in: Task.Supervised.stream(5000)
    ** (EXIT) time out
    (elixir 1.11.2) lib/task/supervised.ex:304: Task.Supervised.stream_reduce/7
    (elixir 1.11.2) lib/enum.ex:3461: Enum.reduce/3
    (mini_repo 0.2.0) lib/mini_repo/mirror/server.ex:106: MiniRepo.Mirror.Server.sync_created_packages/3
    (mini_repo 0.2.0) lib/mini_repo/mirror/server.ex:62: MiniRepo.Mirror.Server.sync/1
    (mini_repo 0.2.0) lib/mini_repo/mirror/server.ex:28: MiniRepo.Mirror.Server.handle_info/2
    (stdlib 3.14) gen_server.erl:689: :gen_server.try_dispatch/4
    (stdlib 3.14) gen_server.erl:431: :gen_server.loop/7
    (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: {:continue, :sync}
State: %MiniRepo.Mirror{name: "hexpm_mirror", only: ["decimal", "nimble_csv", "csv", "ex_aws", "ex_aws_s3"], registry: %{}, store: {MiniRepo.Store.S3, [bucket: "minirepo", options: [region: "ap-southeast-1"]]}, sync_interval: 300000, sync_opts: [max_concurrency: 1], upstream_name: "hexpm", upstream_public_key: "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApqREcFDt5vV21JVe2QNB\nEdvzk6w36aNFhVGWN5toNJRjRJ6m4hIuG4KaXtDWVLjnvct6MYMfqhC79HAGwyF+\nIqR6Q6a5bbFSsImgBJwz1oadoVKD6ZNetAuCIK84cjMrEFRkELtEIPNHblCzUkkM\n3rS9+DPlnfG8hBvGi6tvQIuZmXGCxF/73hU0/MyGhbmEjIKRtG6b0sJYKelRLTPW\nXgK7s5pESgiwf2YC/2MGDXjAJfpfCd0RpLdvd4eRiXtVlE9qO9bND94E7PgQ/xqZ\nJ1i2xWFndWa6nfFnRxZmCStCOZWYYPlaxr+FZceFbpMwzTNs4g3d4tLNUcbKAIH4\n0wIDAQAB\n-----END PUBLIC KEY-----\n", upstream_url: "https://repo.hex.pm"}

Can we set timeout as config option or can we set it to infinity?

Does mini_repo have an UI ?

Can we see how many packages have been mirrored on mini_repo via some UI ?

Can we maybe use hexpm(self-hosted version) to serve as UI for mini_repo ?

Checksum mismatch against registry

Hello everyone,

Thank you for this great app. mini_repo has worked perfectly for several months now. But from one day to another I wasn’t able to mix deps.get anymore - without changing anything in my mini_repo setup. mix deps.get raises the following error:

** (RuntimeError) Checksum mismatch against registry (outer)
    (hex) lib/hex/scm.ex:187: Hex.SCM.update/1
    (mix) lib/mix/dep/fetcher.ex:61: Mix.Dep.Fetcher.do_fetch/3
    (mix) lib/mix/dep/converger.ex:190: Mix.Dep.Converger.all/9
    (mix) lib/mix/dep/converger.ex:123: Mix.Dep.Converger.all/7
    (mix) lib/mix/dep/converger.ex:108: Mix.Dep.Converger.all/4
    (mix) lib/mix/dep/converger.ex:51: Mix.Dep.Converger.converge/4
    (mix) lib/mix/dep/fetcher.ex:16: Mix.Dep.Fetcher.all/3
    (mix) lib/mix/tasks/deps.get.ex:31: Mix.Tasks.Deps.Get.run/1

However, before the issue appeared for the first time, I accidentally published the same version of the same package twice with different contents (thus changing the checksums for the package version). Possibly this has something to do with it? Anyway, the bug is NOT limited to that particular package anymore. In fact, I can't install any of my self-hosted packages at the moment.

I noticed that the mini_repo registry in the MiniRepo.Repository struct contains the same version for that particular package multiple times with different checksums, which seems odd to me. Maybe it would help to somehow rebuild the entiry registry.

This is my first attempt on rebuilding the registry, which did not help.

repo_name = :main
repo_config = Application.fetch_env!(:mini_repo, :repositories)[repo_name]
{_, [root: store_path]} = repo_config[:store]

registry =
  [store_path, "repos", to_string(repo_name), "tarballs", "*.tar"]
  |> Path.join()
  |> Path.wildcard()
  |> Enum.sort()
  |> Enum.reduce(%{}, fn tarball_path, registry ->
    {:ok, {package_name, version_info}} =
      tarball_path
      |> File.read!()
      |> MiniRepo.Utils.unpack_tarball()

    Map.update(registry, package_name, [version_info], &[version_info | &1])
  end)

MiniRepo.Repository
|> struct!([name: to_string(repo_name)] ++ repo_config)
|> Map.put(:registry, registry)
|> MiniRepo.RegistryBackup.save()

Edit: I also took a look into your wm-outer-checksum branch, which actually did the trick. Is there any chance of brushing things up and merging it into the master?

hex search only returns results from hex.pm mirror, never local repo

I am evaluating mini_repo for hosting internal hex packages and I have verified that "publish" and "fetch" of deps works but I have not been able to get "search" working against the local repo. When I do a "rebar3 hex search baz" I receive no results after publishing baz and then using baz as dep in another project. Also I see it in the repo directory of the server.

Feature request: Public Key GET Endpoints

When adding new repos to Mini Repo, we already have to specify the public and private keys in the list of repositories. Adding one of these as an upstream hex repo locally requires sharing the same key out of band. It would be terrific if there was an endpoint where we could ask the upstream repo for its public key (thinking something along the lines of GET /api/repos/:repo/public_key), so that the setup locally can be done by running a curl command entirely in band of the upstream repo. This is analogous to the key being listed on https://hex.pm/docs/public_keys for the original hex repo.

Upload/Download path don't use the same configuration

Hello,

We recently ran mini_repo in a kubernetes cluster, and we found a bug with the paths for the local storage. Uploading (via the API) seem to use the path that is configurable as an environment variable, whereas downloading seems to be hard-coded to {:mini_repo, "data"} which resolves to /app/lib/mini_repo-0.2.0/data in the container. The issue is in endpoint.ex, where the path is not honoring the configured value.

It would be handy if the configured values were reported on start as well, since finding that path at runtime was not particularly fun 😄

mix hex.search support?

https://github.com/hexpm/hex/blob/master/lib/hex/api/package.ex#L12 seems to be calling a GET <hex_api_url>/packages URL which mini_repo does not support currently. That does not seem to be part of the spec though so I am unsure what the proper resolution here is.

As a workflow complication, I'm imagining a user having several separate mini_repo instances which would all need to be searched simultaneously, and may not be on the same server or even intranet, so the right thing to do would probably be to improve hex to support multi-repo search by (perhaps) using repo URLs in some capacity versus the :api_url config.

Implement api key

I realize mini_repo is meant to be a simple hex, and it is great that it starts to address adoption concerns for Enterprise. At certain Enterprise scale different departments of orgs will and should have access control to self-hosted package hosts. Excluding the api key could be viewed as a security concern and become a non-starter as a self-hosted solution.

I would advocate for an opt-in for api key. Keep the default simple as it currently is. With a config option to generate / issue api keys for orgs that require that additional access control layer.

Missing `/public_key` with Local Store

Hello there. Thank you for your work on this project. It is a great platform upon which we can build a custom package management solution.

I'm currently running a modified version of the master branch as a release with the default test_repo. The following occurs when I request the public key:

14:19:59.939 [info]  GET /repos/test_repo/public_key
14:19:59.939 [info]  Sent 404 in 218µs

Requests to /names and /versions succeed as expected. Digging in further, it appears: (1) the public key is configured correctly, according to Application.get_all_env/1, but (2) the key is not present in the data directory.

Do you have any advice? Thank you for your time.

Minirepo behind a proxy

Hello,

I wish to run Minirepo behind a proxy server and use it as a local cache / hex mirror.

With direct internet connectivity (without a proxy server), minirepo starts and automatically fetches packages. However, if minirepo is behind a proxy, it is not able to reach / fetch packages. I tried setting environment variables - e.g. "https_proxy" and "HTTPS_PROXY"; however, I felt the environment variables were ignored.

Am I missing anything? If yes, could you please provide appropriate pointers?

Regards
Bala

Guidance / Documentation Request For Certs

Hex allows overriding a cert path (via cacerts_path or HEX_CACERTS_PATH env variable) to point to a specific PEM file, which works when attempting to pull just private files (repo: "my_repo") from a mini repo using a custom SSL cert. How then do you continue to use the SSL cert for the original hex packages? You either get an SSL error for mix deps.get for private packages, or an SSL error for the public packages. One can elect to use unsafe_https but this is an unsatisfying solution. It seems as though the CACERTS should somehow tie to the repo perhaps?

Inner checksum upload and outer checksum download errors with Hex > 0.20.5

I believe related to the discussion in hexpm/hex#753 and the subsequent work in https://github.com/wojtekmach/mini_repo/tree/wm-outer-checksum

When downloading a dependency of the form {:my_dep, "~> 1.2.3", repo: "my_repo"}, mix deps.get results in the error:

** (RuntimeError) Checksum mismatch against registry (outer)
    (hex) lib/hex/scm.ex:187: Hex.SCM.update/1
    (mix) lib/mix/dep/fetcher.ex:61: Mix.Dep.Fetcher.do_fetch/3
    (mix) lib/mix/dep/converger.ex:190: Mix.Dep.Converger.all/9
    (mix) lib/mix/dep/converger.ex:123: Mix.Dep.Converger.all/7
    (mix) lib/mix/dep/converger.ex:108: Mix.Dep.Converger.all/4
    (mix) lib/mix/dep/converger.ex:51: Mix.Dep.Converger.converge/4
    (mix) lib/mix/dep/fetcher.ex:16: Mix.Dep.Fetcher.all/3
    (mix) lib/mix/tasks/deps.get.ex:31: Mix.Tasks.Deps.Get.run/1

and when publishing via mix hex.publish, the logs print

HTTP status code: 500
Publishing failed

with the server logs:

18:20:14.875 [error] GenServer :risk terminating
** (stop) {:gpb_type_error, {{:missing_fields, [:inner_checksum], :Release}, [value: %{checksum: <<36, 164, 185, 217, 137, 83, 6, 217, 75, 0, 242, 235, 245, 170, 228, 97, 27, 58, 170, 34, 185, 252, 155, 12, 123, 196, 155, 128, 244, 121, 136, 236>>, dependencies: [REDACTED], version: "1.2.3"}, path: :"Package.releases"]}}
    (hex_core) /build/deps/hex_core/src/hex_pb_package.erl:1426: :hex_pb_package.mk_type_error/3
    (hex_core) /build/deps/hex_core/src/hex_pb_package.erl:1247: :hex_pb_package."-v_msg_Package/3-lc$^0/1-0-"/3
    (hex_core) /build/deps/hex_core/src/hex_pb_package.erl:1248: :hex_pb_package.v_msg_Package/3
    (hex_core) /build/deps/hex_core/src/hex_pb_package.erl:64: :hex_pb_package.encode_msg/3
    (mini_repo) lib/mini_repo/registry_builder.ex:70: MiniRepo.RegistryBuilder.build_package/3
    (mini_repo) lib/mini_repo/registry_builder.ex:23: MiniRepo.RegistryBuilder.build_partial/3
    (mini_repo) lib/mini_repo/repository/server.ex:129: MiniRepo.Repository.Server.build_partial_registry/2
    (mini_repo) lib/mini_repo/repository/server.ex:113: MiniRepo.Repository.Server.update_registry/3

Getting 404 when trying to pull package using S3 adapter

When I try to mix deps.get packages hosted on our private mini_repo we get a 404 . Looking at our server logs I see /repos/our_repo/packages/our_package but it returns a 404. When I look at our S3 bucket I can see the packages and tarballs. I have the public access policy documented in releases.exs on the bucket. I've turned off all public access restrictions to the bucket.

Not sure what I'm doing wrong.

I'm assuming mini_repo proxies all package fetches to S3. Is that the case? Should I be going straight to S3?

Could not start mini_repo with aws store.

Our config file is as follows: (we copied it from https://github.com/wojtekmach/mini_repo/blob/master/lib/mini_repo/store/s3.ex#L22-L27)

config :ex_aws,
  access_key_id: "OUR-AWS-ACCESS-KEY",
  secret_access_key: "OUR-AWS-SECRETE-KEY",
  json_codec: Jason

store =
  {MiniRepo.Store.S3,
   bucket: "minirepo",
   options: [
     region: "ap-southeast-1"
   ]}

This gives the following error. And project refuses to start.

** (EXIT) an exception was raised:
        ** (ArgumentError) expected a keyword list, but an entry in the list is not a two-element tuple with an atom as its first element, got: {"content-length", 0}
            (elixir 1.11.2) lib/keyword.ex:475: Keyword.keys/1
            (ex_aws 2.1.1) lib/ex_aws/auth.ex:163: ExAws.Auth.build_canonical_request/5
            (ex_aws 2.1.1) lib/ex_aws/auth.ex:146: ExAws.Auth.signature/8
            (ex_aws 2.1.1) lib/ex_aws/auth.ex:130: ExAws.Auth.auth_header/7
            (ex_aws 2.1.1) lib/ex_aws/auth.ex:45: ExAws.Auth.headers/6
            (ex_aws 2.1.1) lib/ex_aws/request.ex:29: ExAws.Request.request_and_retry/7
            (ex_aws 2.1.1) lib/ex_aws/operation/s3.ex:39: ExAws.Operation.ExAws.Operation.S3.perform/2
            (mini_repo 0.2.0) lib/mini_repo/store/s3.ex:113: MiniRepo.Store.S3.fetch/3
** (Mix) Could not start application mini_repo: MiniRepo.Application.start(:normal, []) returned an error: shutdown: failed to start child: MiniRepo.Repository.Server
    ** (EXIT) an exception was raised:
        ** (ArgumentError) expected a keyword list, but an entry in the list is not a two-element tuple with an atom as its first element, got: {"content-length", 0}
            (elixir 1.11.2) lib/keyword.ex:475: Keyword.keys/1
            (ex_aws 2.1.1) lib/ex_aws/auth.ex:163: ExAws.Auth.build_canonical_request/5
            (ex_aws 2.1.1) lib/ex_aws/auth.ex:146: ExAws.Auth.signature/8
            (ex_aws 2.1.1) lib/ex_aws/auth.ex:130: ExAws.Auth.auth_header/7
            (ex_aws 2.1.1) lib/ex_aws/auth.ex:45: ExAws.Auth.headers/6
            (ex_aws 2.1.1) lib/ex_aws/request.ex:29: ExAws.Request.request_and_retry/7
            (ex_aws 2.1.1) lib/ex_aws/operation/s3.ex:39: ExAws.Operation.ExAws.Operation.S3.perform/2
            (mini_repo 0.2.0) lib/mini_repo/store/s3.ex:113: MiniRepo.Store.S3.fetch/3

Sync Memory Spike At Startup

The application spikes to ~1.5GB on my system at startup and then settles down to ~200MB at idle. Looking through the code, the only suspect part is the RegistryDiff code. Consider changing the diff algorithm to not be greedy for both local packages/versions and remote, but instead compare package by package or something like that. This problem will get worse as more packages get added to hex.

Hex Mirror Sync Progress Markers

The way the code is currently written, a sync is only marked as succeeded once it succeeds fully. Given that we progress package by package, we should be able to save checkpoint intermediate progress if the program crashes for some reason.

An alternative implementation that worked for me locally in such a scenario was:

deleted = sync_deleted_packages(mirror, config, diff)
mirror = update_in(mirror.registry, fn registry -> Map.drop(registry, deleted) end)
RegistryBackup.save(mirror)

chunk_size = 5

mirror =
  diff.releases
  |> Enum.chunk_every(chunk_size)
  |> Enum.reduce(mirror, fn packages, mirror ->
    updated = sync_releases(mirror, config, packages)
    mirror = update_in(mirror.registry, fn registry -> Map.merge(registry, updated) end)
    RegistryBackup.save(mirror)

    mirror
  end)

  mirror =
  diff.packages.created
  |> Enum.chunk_every(chunk_size)
  |> Enum.reduce(mirror, fn packages, mirror ->
    created = sync_created_packages(mirror, config, packages)
    mirror = update_in(mirror.registry, fn registry -> Map.merge(registry, created) end)
    RegistryBackup.save(mirror)

    mirror
  end)

and changing the sync functions to accept packages and to not dig into the diff map contents anymore. Would you be interested in such a change?

Additionally, in the delete part, the code in master is currently calling Map.delete/2 with a list instead of Map.drop/2 with the list of names to drop which is a bug.

That list is likely to be much smaller, but could be given the same treatment as create/update for consistency:

mirror =
  diff.packages.deleted
  |> Enum.chunk_every(chunk_size)
  |> Enum.reduce(mirror, fn packages, mirror ->
    deleted = sync_deleted_packages(mirror, config, packages)
    mirror = update_in(mirror.registry, fn registry -> Map.drop(registry, deleted) end)
    RegistryBackup.save(mirror)

    mirror
  end)

Receive a 500 when attempting to retire a package

When attempting to retire a package I get the following error.

2019-10-31T12:12:27.108306+00:00 app[web.1]: 12:12:27.104 [error] GenServer :<repo> terminating
2019-10-31T12:12:27.108308+00:00 app[web.1]: ** (KeyError) key :reason not found in: %{"message" => "Not wanted", "reason" => "other"}
2019-10-31T12:12:27.108311+00:00 app[web.1]: (stdlib) :maps.get(:reason, %{"message" => "Not wanted", "reason" => "other"})
2019-10-31T12:12:27.108313+00:00 app[web.1]: (elixir) lib/map.ex:273: Map.update!/3
2019-10-31T12:12:27.108316+00:00 app[web.1]: (mini_repo) lib/mini_repo/repository/server.ex:52: anonymous fn/4 in MiniRepo.Repository.Server.retire/4
2019-10-31T12:12:27.108319+00:00 app[web.1]: (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
2019-10-31T12:12:27.108321+00:00 app[web.1]: (mini_repo) lib/mini_repo/repository/server.ex:50: anonymous fn/3 in MiniRepo.Repository.Server.retire/4
2019-10-31T12:12:27.108324+00:00 app[web.1]: (elixir) lib/map.ex:780: Map.update!/3
2019-10-31T12:12:27.108326+00:00 app[web.1]: (mini_repo) lib/mini_repo/repository/server.ex:98: MiniRepo.Repository.Server.update_registry/3
2019-10-31T12:12:27.108329+00:00 app[web.1]: (elixir) lib/agent/server.ex:23: Agent.Server.handle_call/3
2019-10-31T12:12:27.108331+00:00 app[web.1]: Last message (from #PID<0.5322.0>): {:update, #Function<3.17456623/1 in MiniRepo.Repository.Server.retire/4>}
2019-10-31T12:12:27.108337+00:00 app[web.1]: State: #MiniRepo.Repository<name: "<repo>", public_key: "<public key>", registry: %{"hello_repo" => [...]}, store: {...}, ...>
2019-10-31T12:12:27.108339+00:00 app[web.1]: Client #PID<0.5322.0> is alive
2019-10-31T12:12:27.108342+00:00 app[web.1]: 
2019-10-31T12:12:27.108344+00:00 app[web.1]: (stdlib) gen.erl:167: :gen.do_call/4
2019-10-31T12:12:27.108346+00:00 app[web.1]: (elixir) lib/gen_server.ex:1006: GenServer.call/3
2019-10-31T12:12:27.108348+00:00 app[web.1]: (mini_repo) lib/mini_repo/api_router.ex:52: anonymous fn/5 in MiniRepo.APIRouter.do_match/4
2019-10-31T12:12:27.108351+00:00 app[web.1]: (mini_repo) lib/plug/router.ex:259: MiniRepo.APIRouter.dispatch/2
2019-10-31T12:12:27.108353+00:00 app[web.1]: (mini_repo) lib/mini_repo/api_router.ex:1: MiniRepo.APIRouter.plug_builder_call/2
2019-10-31T12:12:27.108355+00:00 app[web.1]: (mini_repo) lib/mini_repo/endpoint.ex:1: MiniRepo.Endpoint.plug_builder_call/2
2019-10-31T12:12:27.108357+00:00 app[web.1]: (plug_cowboy) lib/plug/cowboy/handler.ex:12: Plug.Cowboy.Handler.init/2
2019-10-31T12:12:27.108359+00:00 app[web.1]: (cowboy) /usr/local/mini_repo/deps/cowboy/src/cowboy_handler.erl:41: :cowboy_handler.execute/2
2019-10-31T12:12:27.108361+00:00 app[web.1]: 
2019-10-31T12:12:27.108364+00:00 app[web.1]: 12:12:27.106 [error] #PID<0.5322.0> running MiniRepo.Endpoint (connection #PID<0.5321.0>, stream id 1) terminated
2019-10-31T12:12:27.108366+00:00 app[web.1]: Server: <host>:80 (http)
2019-10-31T12:12:27.108368+00:00 app[web.1]: Request: POST /api/repos/<repo>/packages/hello_repo/releases/0.0.0/retire
2019-10-31T12:12:27.10837+00:00 app[web.1]: ** (exit) exited in: GenServer.call(:<repo>, {:update, #Function<3.17456623/1 in MiniRepo.Repository.Server.retire/4>}, 5000)
2019-10-31T12:12:27.108372+00:00 app[web.1]: ** (EXIT) an exception was raised:
2019-10-31T12:12:27.108375+00:00 app[web.1]: ** (KeyError) key :reason not found in: %{"message" => "Not wanted", "reason" => "other"}
2019-10-31T12:12:27.108377+00:00 app[web.1]: (stdlib) :maps.get(:reason, %{"message" => "Not wanted", "reason" => "other"})
2019-10-31T12:12:27.108379+00:00 app[web.1]: (elixir) lib/map.ex:273: Map.update!/3
2019-10-31T12:12:27.108381+00:00 app[web.1]: (mini_repo) lib/mini_repo/repository/server.ex:52: anonymous fn/4 in MiniRepo.Repository.Server.retire/4
2019-10-31T12:12:27.108384+00:00 app[web.1]: (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
2019-10-31T12:12:27.108386+00:00 app[web.1]: (mini_repo) lib/mini_repo/repository/server.ex:50: anonymous fn/3 in MiniRepo.Repository.Server.retire/4
2019-10-31T12:12:27.108387+00:00 app[web.1]: (elixir) lib/map.ex:780: Map.update!/3
2019-10-31T12:12:27.10839+00:00 app[web.1]: (mini_repo) lib/mini_repo/repository/server.ex:98: MiniRepo.Repository.Server.update_registry/3
2019-10-31T12:12:27.108392+00:00 app[web.1]: (elixir) lib/agent/server.ex:23: Agent.Server.handle_call/3

Always tries to publish to hexpm

Whenever I try to publish to my hosted mini_repo it always tries to publish to hexpm.

When I mix hex.publish package

I always get the message.

Publishing package to public repository hexpm.
not found

Here's my mix.exs

defmodule HelloRepo.MixProject do
  use Mix.Project

  def project do
    [
      app: :hello_repo,
      description: "Hello, repo!",
      version: "0.0.0",
      elixir: "~> 1.9",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      hex: hex(),
      package: package()
    ]
  end

  def application do
    [
      extra_applications: [:logger]
    ]
  end

  defp deps do
    []
  end

  defp hex do
    [
      api_url: "...",
      api_key: "..."
    ]
  end

  defp package do
    [
      licenses: [],
      links: %{}
    ]
  end
end

Feature Request - On demand hexpm_mirror caching

@wojtekmach What would you think about having an option to fetch packages from hexpm on demand instead of specifying only the packages to download.

I'm thinking it would work something like Nexus where packages are downloaded when they are requested.

The reason I think this request is needed is because its difficult to list every package that we may need using mini_repo from hexpm with the :only option. It would allow us to fetch all packages internally on demand and our deps.get would be faster because we are relying on cache from mini_repo instead of hexpm public. (For our CI since local dev caches).

I have been playing around with the code and this shouldn't be very hard to accomplish.

From what I can tell, packages are fetched with two requests from hex.package fetch package version

/repos/hexpm_mirror/packages/<package_name>
/repos/hexpm_mirror/tarballs/<tarball>

Currently these are matched with the Plug.Static:

  plug Plug.Static,
    at: "/repos",
    from: {:mini_repo, "data/repos"}

I would propose that we add two new endpoint matches something like:
get "/repos/hexpm_mirror/packages/:name"
and
get "/repos/hexpm_mirror/tarballs/:tarball"

We then manually check if the package exists in the Application.app_data(:mini_repo) "data/hexpm_mirror/..." location. If that exists great then send_file (conn, 200, path) the package and the tarball.. But if not, we go fetch the package from hex and if it exists, write the file and serve it up. Then all future requests for the same package and version will be found and served up.

We could accomplish pretty easy with a callback into server.ex and a little code to fetch a single package or a list of packages.

What do you think? I'd be happy to code this, but wanted to first see if you would want a feature like this. Sorry if this was a little all over the place.

Minimum IAM permissions for S3 storage

It would be great to have documentation for the minimum IAM permissions necessary for MiniRepo to work. Could be documented in MiniRepo.Store.S3 similar to how the bucket policy is documented.

Open to using bakeware on this project?

Hello,

I was reading about the new tool Bakeware (Compile Elixir applications into single, easily distributed executable binaries) - and it struck me that a project like mini_repo may be a good fit for using it. You could start your own local repo after downloading and running a single file.

I thought probably best to raise this idea in an issue before starting on the PR, just to guage whether this is good/bad idea 😄

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.