Git Product home page Git Product logo

hex_core's Introduction

hex_core

Build Status

Reference implementation of Hex specifications: https://github.com/hexpm/specifications.

Usage

Let's use default config for now. See "Configuration" section below for customization.

Config = hex_core:default_config().

WARNING The built-in httpc-based HTTP client adapter has insecure default settings and thus it is not recommended for usage, in particular for API write operations. Use HTTP clients with secure defaults like hackney and Mint. See "Configuration" and "Wrapper Module" sections for more information on customising your usage of hex_core.

Repository

Get all package names:

> hex_repo:get_names(Config).
{ok,{200, ...,
     #{packages => [
           #{name => <<"package1">>},
           #{name => <<"package2">>},
           ...]}}}

Get all package versions from repository:

> hex_repo:get_versions(Config).
{ok,{200, ...,
     #{packages => [[
           #{name => <<"package1">>, retired => [], versions => [<<"1.0.0">>]},
           #{name => <<"package2">>, retired => [], versions => [<<"0.5.0">>]},
           ...]}}}

Get package releases from repository:

> hex_repo:get_package(Config, <<"package1">>).
{ok,{200, ...,
     #{releases => [
           #{checksum => ..., version => <<"0.5.0">>, dependencies => []}],
           #{checksum => ..., version => <<"1.0.0">>, dependencies => []}],
           ...]}}}

API

For a full list of all parameters and returned objects for the API, check out the API docs: https://github.com/hexpm/specifications/blob/master/http_api.md.

Get package from HTTP API:

> hex_api_package:get(Config, <<"package1">>).
{ok, {200, ...,
    #{
        <<"name">> => <<"package1">>,
        <<"meta">> => #{
           <<"description">> => ...,
           <<"licenses">> => ...,
           <<"links">> => ...,
           <<"maintainers">> => ...,
        },
        ...,
        <<"releases">> => [
            #{<<"url">> => ..., <<"version">> => <<"0.5.0">>}],
            #{<<"url">> => ..., <<"version">> => <<"1.0.0">>}],
            ...
        ]
    }}}

Get package tarball:

{ok, {200, _, Tarball}} = hex_repo:get_tarball(Config, <<"package1">>, <<"1.0.0">>).

Publish package tarball:

{ok, {200, _Headers, _Body} = hex_api_package:publish(Config, Tarball).

Package tarballs

Unpack package tarball:

{ok, #{outer_checksum := Checksum, contents := Contents, metadata := Metadata}} = hex_tarball:unpack(Tarball, memory).

Remember to verify the outer tarball checksum against the registry checksum returned from hex_repo:get_package(Config, Package).

Create package tarball:

{ok, #{tarball := Tarball,
       inner_checksum := InnerChecksum,
       outer_checksum := OuterChecksum}} = hex_tarball:create(Metadata, Contents).

Configuration

The default configuration, provided by hex_core:default_config/0, uses built-in httpc-based adapter and Hex.pm APIs: https://hex.pm/api and https://repo.hex.pm.

HTTP client configuration can be overridden as follows:

Config = maps:merge(hex_core:default_config(), #{
  http_adapter => {my_hackney_adapter, my_hackney_adapter_config},
  http_user_agent_fragment => <<"(my_app/0.1.0) (hackney/1.12.1) ">>
}),
hex_repo:get_names(Config).

%% my_hackney_adapter.erl
-module(my_hackney_adapter).
-behaviour(hex_http).
-exports([request/5]).

request(Method, URI, ReqHeaders, Body, AdapterConfig) ->
    %% ...

See the hex_core module for more information about the configuration.

Wrapper Module

It's recommended to write a wrapper module because a lot of decisions are left to the user, e.g.: where to get configuration from, how to handle caching, failures etc.

For a sample, see: examples/myapp_hex.erl. Here's an excerpt:

-module(myapp_hex).
-export([
    get_api_package/1,
    get_repo_tarball/2,
    get_repo_versions/0
]).

%%====================================================================
%% API functions
%%====================================================================

get_api_package(Name) ->
      case hex_api_package:get(config(), Name) of
          {ok, {200, _Headers, Payload}} ->
              {ok, Payload};

          Other ->
              Other
      end.

get_repo_versions() ->
      case hex_repo:get_versions(config()) of
          {ok, {200, _Headers, Payload}} ->
              {ok, maps:get(packages, Payload)};

          Other ->
              Other
      end.

%%====================================================================
%% Internal functions
%%====================================================================

config() ->
    Config1 = hex_core:default_config(),
    Config2 = put_http_config(Config1),
    Config3 = maybe_put_api_key(Config2),
    Config3.

put_http_config(Config) ->
    maps:put(http_user_agent_fragment, <<"(myapp/1.0.0) (httpc)">>, Config).

maybe_put_api_key(Config) ->
    case os:getenv("HEX_API_KEY") of
        false -> Config;
        Key -> maps:put(api_key, Key, Config)
    end.

Installation

Rebar3

Add to rebar.config:

{deps, [
  {hex_core, "0.10.3"}
]}

Mix

Add to mix.exs:

defp deps do
  [
    {:hex_core, "~> 0.10.3"}
  ]
end

Development

  • Run rebar3 as dev compile to re-generate protobuf files
  • Run rebar3 as test proper for property-based tests
  • Run rebar3 ex_doc to generate documentation

hex_core's People

Contributors

aerosol avatar aj-foster avatar cgerling avatar ericmj avatar ferd avatar isaacsanders avatar jadeallenx avatar jjcarstens avatar joladev avatar josecfreittas avatar kianmeng avatar kubaodias avatar lostkobrakai avatar michaelklishin avatar pablocostass avatar paulo-ferraz-oliveira avatar starbelly avatar supersimple avatar the-mikedavis avatar tsloughter avatar vkatsuba avatar wojtekmach avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hex_core's Issues

Run tests on OTP 17

There's a regression on master that was only caught when I tried vendoring master into Hex. proper plugin does not compile on otp 17, so we need to again try fixing that or add some workaround.

Ensure API params have binary keys

In some cases we manually use binary keys:

https://github.com/hexpm/hex_core/blob/v0.6.1/src/hex_api_key.erl#L20:

Params = #{<<"name">> => Name, <<"permissions">> => Permissions},

but in some we allow user to pass the map:

https://github.com/hexpm/hex_core/blob/v0.6.1/src/hex_api_release.erl#L45

hex_api:post(Config, Path, Params).

And so the user may pass atom keys like we used to do here:

wojtekmach/mini_repo@f6c8f39#diff-5008c7016636a9b7531bd02fa4abb5ffL52

We should ensure the params are always a map of binary keys.

Fix typespec for hex_tarball:create/2

The specs are:

-spec create(metadata(), files()) -> {ok, {tarball(), checksum()}}.
-type contents() :: #{filename() => binary()}.
-type filename() :: string().
-type files() :: [filename() | {filename(), filename()}] | contents().

but we don't actually accept a map as files on create, we're incorrectly re-using contents type that is only for unpack.

Unpacking tarball leaves an empty leftover `memory` directory

5> filelib:is_dir("memory").
false
6> {ok, #{tarball := Tarball}} = hex_tarball:create(#{name => <<"foo">>}, [{"a.txt", <<"">>}]).
{ok,#{inner_checksum =>
          <<184,50,90,164,129,244,147,39,115,65,24,50,16,59,114,74,
            241,242,173,155,50,158,184,8,99,131,...>>,
      outer_checksum =>
          <<201,62,52,115,78,250,211,233,23,206,11,244,10,34,208,
            229,14,117,17,49,49,180,220,114,195,169,...>>,
      tarball =>
          <<86,69,82,83,73,79,78,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
            0,0,0,...>>}}; 
7> hex_tarball:unpack(Tarball, memory).
{ok,#{contents => [{"a.txt",<<>>}],
      inner_checksum =>
          <<184,50,90,164,129,244,147,39,115,65,24,50,16,59,114,74,
            241,242,173,155,50,158,184,8,99,131,...>>,
      metadata => #{<<"name">> => <<"foo">>},
      outer_checksum =>
          <<201,62,52,115,78,250,211,233,23,206,11,244,10,34,208,
            229,14,117,17,49,49,180,220,114,195,169,...>>}}
8> filelib:is_dir("memory").
true
9> file:list_dir("memory").
{ok,[]}

Validate package metadata

Validate package metadata against spec errors, do not include superfluous fields and do not allow type errors.

We can also add separate, optional functions to validate package metadata for warnings, we can pass it a list of policies that can be used, like (hexpm/public/private). We could handle the tarball size validation the same way to not be locked into hexpm specifics.

NOTE: Do not include maintainers in the package metadata.

vendor script seems broken

Not too sure if there's something off in how I'm using it (on linux):

λ hex_erl → master → ./vendor.sh
Usage: vendor.sh TARGET_DIR PREFIX
λ hex_erl → master → ./vendor.sh ../rebar3/src/ hex_
sed: -e expression #1, char 1: unknown command: `.'
λ hex_erl → master → ./vendor.sh ~/code/rebar3/src/ hex_
./vendor.sh: line 39: /home/ferd/code/rebar3/src//hex_hex_erl.hrl: No such file or directory
λ hex_erl → master → ./vendor.sh ~/code/rebar3/src hex_
./vendor.sh: line 39: /home/ferd/code/rebar3/src/hex_hex_erl.hrl: No such file or directory

I couldn't get it to do anything else but these things.

Oldest OTP version that hex_core needs to support?

Back in Nov 2020, hex_core needed to support OTP 17:

hex_core needs to support OTP 17. This is because it needs to be usable by Hex which in turns needs to be usable by Elixir 1.0.x. At some point Hex will no longer support old Elixir versions but not quite yet.

I'm wondering what the oldest OTP version is that hex_core needs to support now? The background for my question is that for gpb, I've retained compatibility with older Erlang versions, but would like to know if I can drop some? For example for maps, Erlang 18 added support for using variables as map keys, so still needing to supporting Erlang 17 in gpb would impose some limitations on how the code could be phrased. As far as I'm aware, hex_core is the project that needs support for the oldest Erlang for gpb. A recent issue in gpb, tomas-abrahamsson/gpb#217 prompted this question.

I tried looking for info on the oldest Elixir still supported by Hex, but failed to find it. Related: the latest Elixir, currently 1.13.4, says that the oldest Elixir to still receive security patches is 1.9, which is compatible with Erlang versions 20-22. Some more research: Debian old-stable, currently buster, ships Elixir 1.7.4 (link), which is compatible with Erlang 19-22.

hex_tarball:unpack/2, create/2 API changes

Currently the API is:

-type contents() :: #{filename() => binary()}.
-type tarball() :: binary().
%% ...

-spec unpack(tarball(), memory) ->
                {ok, #{checksum => checksum(), metadata => metadata(), contents => contents()}} |
                {error, term()};
            (tarball(), filename()) ->
                {ok, #{checksum => checksum(), metadata => metadata()}} |
                {error, term()}.

so we are only able to accept binaries. I'm wondering if we should accept filenames too, so instead of binary() we accept {binary, binary()} | filename:filename() and deprecate the former? Eventually, at say v1.0, we'd accept {binary, binary()} | filename:filename_all().

This is a low-level library so I'm ok with pretty low level interface but I think this change would make it more convenient to use and I wish I'd have started with that.

Thoughts? cc @ferd @tsloughter

Roadmap

v0.1.0:

  • Add script to vendor hex_erl into other builds tools (rewrite vendor_hex_erl task in bash or something; include hex_erl version in generated files) (#7)
  • Add ETag support to hex_repo (#8)
  • Add hex_api client per https://github.com/hexpm/specifications/blob/master/http_api.md
    • public endpoints (get package, get release etc) (#11)
    • search endpoint
    • private endpoints (keys, owners etc) (#26)
    • organizations support (#26)
    • create user endpoint (#26)
    • publish package endpoint (#26)
    • Add ETag support
    • Rate limit headers support (no longer applies, we return all headers)
    • URI.encode (in Erlang) URLs (#26)

v0.2.0:

Default tarball_max_size used by rebar3_hex too small

When trying to publish docs for aws-beam/aws-erlang I ran into an issue where trying to publish the docs resulted in the following error: "tarball error, unknown POSIX error".

After patching OTP to include the actual error as per my local OTP patch I figured the actual error is too_big which comes from

{error, {tarball, too_big}};

rebar3_hex uses hex_tarball:create_docs/1 and does not allow overriding the default configs coming from:

create_docs(Files, hex_core:default_config()).

One can argue for one of two things:

  1. rebar3_hex should not use create_docs/1 but should instead use create_docs/2 and allow the overriding of tarball_max_size and tarball_max_uncompressed_size.
  2. The defaults chosen in hex_core are too small and hence should be increased.

The spec for retirement_reason / retirement_params may be incorrect.

The retirement reason / params is specified as as atoms and the keys for params as atoms. However, this does not work. iirc, at one point in time atoms were allowed, but that changed, and we can see that here : https://github.com/hexpm/hexpm/blob/e1251217d56ec91cdb104a27d9dd70eb08aa0f47/lib/hexpm_web/erlang_format.ex#L18

It may be there was intention to serialize atoms to strings from the client side via protobuf, but that doesn't appear to work. Testing locally nets me an http 400.

I'm remember all this being discussed some years ago, but the memory is fuzzy around the details now. In other words, either this is a bug in encoding within hex_code (it does not fail in hex_core to be clear) or the specs need to be adjusted to reflect reality.

Do not crash when unpacking empty packages

E.g.:

mix hex.package fetch loki 0.1.0 --unpack
** (MatchError) no match of right hand side value: {:error, :enoent}
    (hex 0.20.1) src/mix_hex_tarball.erl:220: :mix_hex_tarball.copy_metadata_config/2
    (hex 0.20.1) src/mix_hex_tarball.erl:209: :mix_hex_tarball.finish_unpack/1
    (hex 0.20.1) lib/hex.ex:183: Hex.unpack_tar!/2
    (hex 0.20.1) lib/mix/tasks/hex.package.ex:114: Mix.Tasks.Hex.Package.fetch/4
    (mix 1.10.0-dev) lib/mix/task.ex:330: Mix.Task.run_task/3
    (mix 1.10.0-dev) lib/mix/cli.ex:83: Mix.CLI.run_task/2

hex_core does not return the body of an error message when an invalid semver is provided for revertion

If an invalid semver is provided as an argument when attempting to revert a package (i.e., "eh?") hexpm will return a 400 with an appropriate message (can only delete a release up to one hour after publication) however this message is not properly passed back to the caller. It's not clear as to why. @wojtekmach believes the body is simply not passed back to the caller in this case.

The raw error returned from hex_core:

{error,{rebar3_hex_revert,{api_error,<<"truecoat">>,<<"me">>,[72,84,84,80,32,115,116,97,116,117,115,32,99,111,100,101,58,32,"400"]
}}}

Or in other words : "HTTP status code: 400"

Edit:

Note this issue was originally opened on hexpm as hexpm/hexpm#1015. If it turns out this a problem at in hexpm vs hex_core we can simply re-open the other, but my assumption is the same as @wojtekmach's on this one.

The built in httpc requests throw MatchErrors on connection loss

https://github.com/hexpm/hex_core/blob/master/src/hex_http_httpc.erl#L14

If your connection is gone the left hand side match fails because you actually get {:error, {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :nxdomain}]}}

It'd be nice if that error tuple was passed through instead of having to wrap all calls in try and rescuing the MatchError. I feel that the fact that it returns a ok tuple implies that it would also return an error tuple in this case

If this sounds reasonable I can make a PR

When hex_erl is included as project dependency it tries to run protobuf provider hooks

I have a project that includes hex_erl in rebar.config e.g.

{deps,[
    {hex_erl, {git, "https://github.com/hexpm/hex_erl.git", {branch, "master"}}}
]}.

When I run rebar3 compile in my project I get the following error:

===> Verifying dependencies...
===> Fetching hex_erl ({git,"https://github.com/hexpm/hex_erl.git",
                                  {ref,
                                      "6c8973c2ce3acea1ebec8f08d23b4951f272361b"}})
===> Compiling hex_erl
===> Unable to run pre hooks for 'compile', command 'compile' in namespace 'protobuf' not found.

It seems that when protobuf is added as project plugin it isn't fetched as project dependency. https://github.com/hexpm/hex_erl/blob/6c8973c2ce3acea1ebec8f08d23b4951f272361b/rebar.config#L3-L6
https://github.com/hexpm/hex_erl/blob/6c8973c2ce3acea1ebec8f08d23b4951f272361b/rebar.config#L18-L23
Please refer to end of this section in rebar3 documentation: http://www.rebar3.org/v3/docs/using-available-plugins#section-project-plugins-and-overriding-commands

Running mix with kaffe deps fails to download pc package from hex

Environment : Running wsl Ubuntu behind a corporate proxy

Error Message:
===> Errors loading plugin pc. Run rebar3 with DEBUG=1 set to see errors.
===> Expanded command sequence to be run: []
===> Running provider: do
===> Expanded command sequence to be run: [app_discovery,{bare,compile}]
===> Running provider: app_discovery
===> Found top-level apps: [snappyer]
using config: [{src_dirs,["src"]},{lib_dirs,["apps/","lib/","."]}]
===> Getting definition for package pc from repo hexpm
===> hex_repo:get_package failed for package <<"pc">>: {badmatch,
{error,enotdir}}
===> throw {error,{rebar_app_utils,{missing_package,<<"pc">>,undefined}}} [{rebar_app_utils,
update_source,
3,
[{file,
Heres the mix file
defmodule Sputnik.MixProject do
use Mix.Project

def project do
[
app: :sputnik,
version: "0.1.0",
elixir: "~> 1.14.3",
start_permanent: Mix.env() == :prod,
deps: deps()
]
end

Run "mix help compile.app" to learn about applications.

def application do
[
extra_applications: [:logger,:kaffe],
mod: {ElixirKaffeCodealong.Application, []}
]
end

Run "mix help deps" to learn about dependencies.

defp deps do
[
{:certifi, "> 2.10"},
{:pc, "
> 1.13.0"},

add to your existing deps

{:kaffe, "~> 1.9"}

]

end
end

Proxy configuration

We should either document how to set up proxy (#36 (comment)) or have first-class support for setting it (either there's a contract for it or adapter-specific configuration is passed down so we can e.g. httpc:set_options() in the httpc adapter)

Ref: wojtekmach/mini_repo#8

Quoting etag required

I noticed our caching wasn't working and discovered the need to do the quoting of the etag when building the hex core config map:

#{http_etag => <<"\"", ETag/binary, "\"">>},

I can send a patch if you're ok with this being done hex_http instead, but does it need to check and not add extra quotes or can it just assume the user will not send a quoted etag?

Request release

Would it be possible to release master (e.g. 0.7.1) so as to benefit from #105? Thanks much.

Disk test checksum mismatch

Failures:

  1) hex_tarball_tests:disk_test/0
     Failure/Error: {error,
                        {badmatch,
                            <<"BB994E006EA06B74A6A4C72C0C42B565C0E510A3B24D211D46EEF77C4CFED49F">>},
                        [{hex_tarball_tests,'-disk_test/0-fun-0-',0,
                             [{file,
                                  "/home/hq1/dev/hex_erl/_build/test/lib/hex_erl/test/hex_tarball_tests.erl"},
                              {line,24}]},
                         {hex_tarball_tests,in_tmp,1,
                             [{file,
                                  "/home/hq1/dev/hex_erl/_build/test/lib/hex_erl/test/hex_tarball_tests.erl"},
                              {line,188}]}]}
     Output:


Top 10 slowest tests (0.038 seconds, 18.4% of total time):
  hex_registry_tests:signed_test/0
    0.010 seconds
  hex_tarball_tests:disk_test/0
    0.005 seconds
  hex_tarball_tests:unpack_error_handling_test/0
    0.005 seconds
  hex_tarball_tests:timestamps_and_permissions_test/0
    0.004 seconds
  hex_repo_tests:get_names_test/0
    0.003 seconds
  hex_repo_tests:get_package_test/0
    0.003 seconds
  hex_repo_tests:get_tarball_test/0
    0.003 seconds
  hex_tarball_tests:build_tools_test/0
    0.002 seconds
  hex_repo_tests:get_versions_test/0
    0.002 seconds
  hex_tarball_tests:memory_test/0
    0.001 seconds

support setting a httpc profile

Rebar3 sets up an httpc profile for proxy settings and such and uses it in requests. It looks like hex_core has no support for setting a profile?

I think a good option would be to just add it to the config (using default by default) and then add the argument to the request call to hex_httpc. Does that sound good?

Should it just be named http_profile in the config? Or be more specific?

Theoretical new hex library for sharing command logic between hex and rebar3_hex

Discuss the possibility of a new hex library to share command logic, such as hex.outdated, between hex, rebar3_hex, and hex related libraries and/or plugins. This requires enumerating the differences between the two as is and enumerating what's missing from rebar3_hex and other hex related libraries where it might be beneficial to share code between projects in particularly where it's the logic is a bit complicated, but also with the idea that getting new commands into plugins such as rebar3_hex in a more expedient fashion.

hex_tarball enhancment: Report max size in error or provide functions to obtain max size(s)

We have a need in rebar3_hex to report to the user what the max size allowed for a package is. There are at least 4 ways I can see to do this :

  1. update hex_tarball:create/2 to return {error, {tarball, too_big, ?TARBALL_MAX_SIZE}}, ?TARBALL_MAX_UNCOMPRESSED_SIZE}}
  2. hex_core could formulate an error message and return that {error, {tarball, too_big, Msg}}
  3. hex_core could maybe standardize on maps as the second element in an error tuple and put messages, sizes, etc. in said map. {error, #{}}or {error, {tarball, #{}}}
  4. hex_core couldhex_tarball:max_size/0 or hex_tarball:max_compressed_size/0 and hex_tarball:max_uncompressed_size/0 and consumers of hex_core would simply refer to these functions to assist in formulating a message.

It's probably easiest to just provide a function that returns a max size. I'm also wondering if we can get away with just telling the user in an error message only the compressed size (absolute max size), as it gets confusing once you start trying to explain compressed size and uncompressed size in an error message. Perhaps giving the user the uncompressed max size is more helpful vs the compressed max size.

Happy to do a PR, but figured a little discussion around this first would be proper.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.