Hey,

I'm curious about the decision to return Result instead of panicking here:

Providing the wrong lengths of buffers doesn't seem like a runtime error that one can recover from. It seems more like a logical error by the developer. Adding the overhead of this check and later matching on the result seems unnecessary.

Thanks!

Is there some reason why FromHexError does not derive Eq? It seems common in the Rust std lib to do so, here are a bunch of example errors that do so: https://doc.rust-lang.org/std/cmp/trait.Eq.html

Happy to do a PR but its so trivial I thought there might be some reason I can't work out.

Thanks

Hi,

My code heavily relies on hex::encode from your crate. I'm not sure that performance of hex::encode is a bottleneck in my project, but for some of my methods a single call to hex::encode takes about 1/4 of overall time the method takes. It's too long for my crypto-related stuff, yet hex is a very nice crate.

I started an investigation and added a bench for hex::encode in my fork.
What's interesting is the fact that the performance of hex::encode depends on a length of input non-lineary.

Here is an output of cargo bench for information.

running 6 tests
test tests::bench_encode_512bits ... bench:         962 ns/iter (+/- 139)
test tests::bench_encode_256bits ... bench:         518 ns/iter (+/- 61)
test tests::bench_encode_128bits ... bench:         287 ns/iter (+/- 42)
test tests::bench_encode_64bits  ... bench:         163 ns/iter (+/- 93)
test tests::bench_encode_32bits  ... bench:         123 ns/iter (+/- 15)
test tests::bench_encode_16bits  ... bench:          95 ns/iter (+/- 63)

What I'm going to do is to try to improve the performance of this method.
Please vote up / comment in a case my investigation and pull-requests are appreciated here.

Thanks!

The serde support of this crate is just plain awesome. I use #[serde(with = "hex::serde")] all the time. However, I just ran into the situation where I want a Vec<Vec<u8>> to be encoded as a list of hex strings. This is a lot less straightforward to implement, sadly.

To make this more ergonomic, I suggest adding a serialization wrapper newtype that I may use instead of serde_with. I could then write something like Vec<HexBytes> or Vec<Hex<Vec<u8>>> and it would just work.

It would be really nice if you could tag the commits that are used for releases, that would make it really easy for tools like Dependabot to link to which commits are included in a new release when evaluating an update PR.

Could be nice to have a struct that enforce at compile time that a string is expected to be Hex encoded.

struct HexString(String);
struct HexSlice<'a>(&'a str)

The idea behind this is related to mongodb gridfs, the field md5 is expected to be Hex encoded, so I though, wouldn't be nice to have a struct that ensure a field is hex encoded ?

This is common in TLS crate for example, https://docs.rs/native-tls/0.2.4/native_tls/struct.Certificate.html.

I think the trait implementation could belong either here or to the generic-array crate. Gut feeling says that here would be better. Obviously, this should be feature gated either way.

it's relatively minor, but the endianness of the output of .to_hex() may differ from the output of POSIX hexdump:

hexdump:

0000000 7c21 6c1f 6b2c 274c 9ec3 55b4 48df 86fd 
0000010 eddb 6191 73f4 b073 1ec9 2d12 314c a80b 
0000020 

to_hex: "217c1f6c2c6b4c27c39eb455df48fd86dbed9161f47373b0c91e122d4c310ba8"

Being able to choose would be a nice feature.

@KokaKiwi, Could you publish the latest commit to support no_std?

Once const-generics are further stabilized, one could write an encode/decode function that does not error:

#![feature(const_generics, const_evaluatable_checked)]
use std::array::IntoIter;

const HEX_CHARS_LOWER: &[u8; 16] = b"0123456789abcdef";

fn generate_iter(len: usize) -> impl Iterator<Item = (usize, usize)> {
    (0..len).step_by(2).zip((0..len).skip(1).step_by(2))
}

const fn byte2hex(byte: u8, table: &[u8; 16]) -> (u8, u8) {
    let high = table[((byte & 0xf0) >> 4) as usize];
    let low = table[(byte & 0x0f) as usize];

    (high, low)
}

pub fn encode<const N: usize>(bytes: [u8; N]) -> [u8; N * 2] {
    let mut output = [0; { N * 2 }];

    for (byte, (i, j)) in IntoIter::new(bytes).zip(generate_iter(N * 2)) {
        let (high, low) = byte2hex(byte, HEX_CHARS_LOWER);
        output[i] = high;
        output[j] = low;
    }

    output
}

fn main() {
    assert_eq!(encode(*b"kiwi"), *b"6b697769");
}

playground

I think this should be considered before the 1.0 release (this could either be a new function or it could replace an existing one?).

The error.rs is being imported as mod error; and not pub mod error;. I was trying to match the error returned but couldn't do it I can't access it from my crate.

Feedback from shssoichiro/avsser#2 (comment):

I will say I wish that the hex crate could deserialize into a fixed-size array, like rustc-serialize could. I found a better way to do it in this crate (because the hex strings were UUIDs), but it would still be a good functionality to have.

https://doc.rust-lang.org/nightly/core/fmt/trait.LowerHex.html
https://doc.rust-lang.org/nightly/core/fmt/trait.UpperHex.html

it would be great to directly display u8 slices to the console like this:

let data: &[u8] = ...;
println!("{:x}", data);

because we currently have to do this:

let mut s = String::new();
data.write_hex(&mut s).unwrap();
println!("{}", s);

which isn't great

Currently, encode_to_slice documentation says:

The output buffer, has to be able to hold at least input.len() * 2 bytes, otherwise this function will return an error.

However the code checks for an exact size:

Which side of the doc-vs-code split should be fixed? (I'm inclined to just clarify the docs, as the resulting/current API is quite ergonomic)

Decoding hex value "banana" I get Invalid character \'n\' at position 2
Decoding "abcdefg" I get Invalid character \'g\' at position 7

Such as:

pub trait ToHexExt {
    /// Encode the hex strict representing `self` into the result with prefix `0x`. Lower case
    /// letters are used (e.g. `f9b4ca`)
    fn encode_hex_with_prefix<T: iter::FromIterator<char>>(&self) -> T;

    /// Encode the hex strict representing `self` into the result with prefix `0X`. Upper case
    /// letters are used (e.g. `F9B4CA`)
    fn encode_hex_upper_with_prefix<T: iter::FromIterator<char>>(&self) -> T;
}


impl <T: ToHex + AsRef<[u8]> >ToHexExt for T {
    fn encode_hex_with_prefix<U: iter::FromIterator<char>>(&self) -> U {
        encode_to_iter(HEX_CHARS_LOWER, &["0x".as_ref(), self.as_ref()].concat())
    }

    fn encode_hex_upper_with_prefix<U: iter::FromIterator<char>>(&self) -> U {
        encode_to_iter(HEX_CHARS_UPPER, &["0X".as_ref(), self.as_ref()].concat())
    }
}

But I find that maybe need a ext subdir for doing such work for all apis.

Latest 0.4.1 release broke support for older Rust versions compared to 0.4.0, for example, accessing enum variants via Self were not available for Rust 1.36.0:

In my case this version of hex were used as a dependency in the Cargo.toml,

[dependencies]
hex = "0.4.0"

and after the new release, cargo automatically started using 0.4.1 as they are assumed to be compatible.

Generally it seems unexpected to update MSRV (Minimal Supported Rust Version) in the patch versions, so it would be nice to clarify your policy on the compiler versions support.
Usual options are either to test specific Rust version in the CI workflow or to put a note in the README file that this project is compatible only with the latest stable Rust version.

EDIT: this discussion might be interesting too.

Using hex-simd as a backend can provide a performance boost for "free".

Benchmarks: docs/benches-v050.md

How to append hex-encoded bytes to a string without allocation?

It might like this:

pub fn encode_to_string(data: impl AsRef<[u8]>, output: &mut String)

Would it be possible to make a new release to crates.io?

I think the point of FromHex and ToHex is to allow other librarys to accept any kind of type, that implements those traits (but why? to decode or encode them? why not do that before passing them to the function?).

The encoding and decoding traits should be symmetric (currently they aren't), by that I mean that any type, that can be encoded should be decodable too.

The encode and decode function can only work with an input of Iterator<Target = u8> or Iterator<Target = char> (where the char can be converted to an u8...), this includes&[u8], String, &str or [u8; n].

Therefore it might be better to provide a single trait like this

trait ToBytes {
    fn to_bytes(&self) -> Iterator<Target = u8>;
}

and make the hex function signatures like this

fn encode<T: ToBytes>(value: T)

I omitted the return type, because I don't know what type should be returned. If you are too generic it can no longer be used with for example no_std, or it makes the code slow/too complicated.

There is also some duplication inside the library, like several encode functions? (not sure, it could also have been decode)

Overall I think it would be better to just remove them.

There used to be write_hex that encoded straight into Writer, but now it's gone and I'm not sure what's the most idiomatic way to replace it without temporarily allocation.

The problematic parts of code:

fn encode_to_iter<T: iter::FromIterator<char>>(table: &'static [u8; 16], source: &[u8]) -> T {
    BytesToHexChars::new(source, table).collect()
}

impl<T: AsRef<[u8]>> ToHex for T {
    fn encode_hex<U: iter::FromIterator<char>>(&self) -> U {
        encode_to_iter(HEX_CHARS_LOWER, self.as_ref())
    }

    fn encode_hex_upper<U: iter::FromIterator<char>>(&self) -> U {
        encode_to_iter(HEX_CHARS_UPPER, self.as_ref())
    }
}

By returning FromIterator you are forcing the function user, to collect the data into something (most likely something from alloc like String or Vec).

I would propose to change the bounds to IntoIterator<Item = char>, you could still collect it into for example a String
value.encode_hex().collect::<String>();, but with the benefit of being able to loop though the entire thing without extra allocations.

For example #8 needs a wrapper type, that implements fmt::UpperHex, with the current definition you would have to write it like this

use crate::ToHex;
use core::fmt;

pub struct Hex<T>(pub T);

impl<T: ToHex> fmt::UpperHex for Hex<T> {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "{}", self.0.encode_hex_upper::<String>())
    }
}

As you can see you have to create a temporary String, which would require alloc.

If the trait would return IntoIterator<Item = char> you could do this

use crate::ToHex;
use core::fmt;

pub struct Hex<T>(pub T);

impl<T: ToHex> fmt::UpperHex for Hex<T> {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
       for c in self.0.encode_hex_upper() {
              write!(f, "{}", c)?;
       }
       Ok(())
    }
}

which does not allocate anywhere.

It'd be great to be able to use hex in no-std projects but as it currently depends on std this is not possible. I think this would be a breaking change so before submitting a PR I wanted to check if this is something you'd be willing to consider merging?

I'd probably port over code from https://github.com/rphmeier/rustc-hex that uses iterators.

Is there any way of serializing a Vec<Vec<u8>> into a Vec<T> where each T is a hex encoded string?

Sorry if it has been asked before but I couldn't find any related issue.

At the moment there is no way to choose between case mode when using allocation free encode_to_slice. Current behavior defaults to lowercase.

My proposal is to:
change encode_to_slice -> encode_to_slice_lower add encode_to_slice_upper`

let (high, low) = byte2hex(*byte, HEX_CHARS_LOWER);
HEX_CHARS_LOWER could be easily parametrized

2cbeff7

https://github.com/rust-lang/rust/blob/master/src/libserialize/hex.rs

It looks like this project is based on hex.rs from the compiler. Shouldn't this project retain the license headers and copyright for rust-lang/rust?

Update FromHex::from_hex to accept &[u8] instead of &str.

It seems to be overkill to pass unicode into a function that operates on ASCII bits. The first operation that happens is converting the unicode to bytes anyways. Consider a couple scenarios:

Current design:

Vec::from_hex("hello")
Vec::from_hex(str::from_utf8(b"hello").unwrap())

With my proposal:

Vec::from_hex("hello".as_bytes())
Vec::from_hex(b"hello")

Stop accepting whitespace in the input to FromHex::from_hex

Is there any reason for this? I feel like if the user passes in superfluous characters that are not hex to from_hex, it should probably be result in an error.

Thoughts?

I came here to find information on the change between 0.4.2 -> 0.4.3 but no changelog.md or information in releases.
Is it worth adding a changelog?

https://keepachangelog.com/en/1.0.0/

So, has support for this stalled? Given the heavy usage of this crate, does it need a new maintainer? I see no releases in three years, lots of open pull requests, and no movement towards a 1.x release. Thoughts?

> cargo new foo && cd foo
     Created binary (application) `foo` package
> cargo add --vers 0.4.2 hex --no-default-features
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding hex v0.4.2 to dependencies
> echo 'fn main() { dbg!(hex::encode(b"1234")); }' > src/main.rs
> cargo update --precise 0.4.2 -p hex
    Updating crates.io index
    Updating hex v0.4.3 -> v0.4.2
> cargo run
   Compiling hex v0.4.2
   Compiling foo v0.1.0 (/tmp/tmp.EwT6eRV1y2/foo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.51s
     Running `/home/nemo157/.cargo/shared-target/debug/foo`
[src/main.rs:1] hex::encode(b"1234") = "31323334"
> cargo update -p hex
    Updating crates.io index
    Updating hex v0.4.2 -> v0.4.3
> cargo run
   Compiling foo v0.1.0 (/tmp/tmp.EwT6eRV1y2/foo)
error[E0425]: cannot find function `encode` in crate `hex`
 --> src/main.rs:1:23
  |
1 | fn main() { dbg!(hex::encode(b"1234")); }
  |                       ^^^^^^ not found in `hex`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0425`.
error: could not compile `foo`

To learn more, run the command again with --verbose.

Moving an existing API under a feature flag is a breaking change to all users with default-features = false specified.

Hi,

It would be useful to be able to do something like:

println!("something: {}", hex::Hex(bytes))

This would allow hexlify things without creating temporary buffers.

What do you think?

This issue was automatically generated. Feel free to close without ceremony if
you do not agree with re-licensing or if it is not possible for other reasons.
Respond to @cmr with any questions or concerns, or pop over to
#rust-offtopic on IRC to discuss.

You're receiving this because someone (perhaps the project maintainer)
published a crates.io package with the license as "MIT" xor "Apache-2.0" and
the repository field pointing here.

TL;DR the Rust ecosystem is largely Apache-2.0. Being available under that
license is good for interoperation. The MIT license as an add-on can be nice
for GPLv2 projects to use your code.

Why?

The MIT license requires reproducing countless copies of the same copyright
header with different names in the copyright field, for every MIT library in
use. The Apache license does not have this drawback. However, this is not the
primary motivation for me creating these issues. The Apache license also has
protections from patent trolls and an explicit contribution licensing clause.
However, the Apache license is incompatible with GPLv2. This is why Rust is
dual-licensed as MIT/Apache (the "primary" license being Apache, MIT only for
GPLv2 compat), and doing so would be wise for this project. This also makes
this crate suitable for inclusion and unrestricted sharing in the Rust
standard distribution and other projects using dual MIT/Apache, such as my
personal ulterior motive, the Robigalia project.

Some ask, "Does this really apply to binary redistributions? Does MIT really
require reproducing the whole thing?" I'm not a lawyer, and I can't give legal
advice, but some Google Android apps include open source attributions using
this interpretation. Others also agree with
it.
But, again, the copyright notice redistribution is not the primary motivation
for the dual-licensing. It's stronger protections to licensees and better
interoperation with the wider Rust ecosystem.

How?

To do this, get explicit approval from each contributor of copyrightable work
(as not all contributions qualify for copyright, due to not being a "creative
work", e.g. a typo fix) and then add the following to your README:

## License

Licensed under either of

 * Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
 * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.

### Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any
additional terms or conditions.

and in your license headers, if you have them, use the following boilerplate
(based on that used in Rust):

// Copyright 2016 rust-hex developers
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.

It's commonly asked whether license headers are required. I'm not comfortable
making an official recommendation either way, but the Apache license
recommends it in their appendix on how to use the license.

Be sure to add the relevant LICENSE-{MIT,APACHE} files. You can copy these
from the Rust repo for a plain-text
version.

And don't forget to update the license metadata in your Cargo.toml to:

license = "MIT/Apache-2.0"

I'll be going through projects which agree to be relicensed and have approval
by the necessary contributors and doing this changes, so feel free to leave
the heavy lifting to me!

Contributor checkoff

To agree to relicensing, comment with :

I license past and future contributions under the dual MIT/Apache-2.0 license, allowing licensees to chose either at their option.

Or, if you're a contributor, you can check the box in this repo next to your
name. My scripts will pick this exact phrase up and check your checkbox, but
I'll come through and manually review this issue later as well.

• @KokaKiwi

I've just been testing a few things as part of my linux vendoring efforts, and I stumbled upon this:

    Updating crates.io index
   Compiling proc-macro2 v1.0.8
   Compiling unicode-xid v0.2.0
   Compiling syn v1.0.14
   Compiling serde v1.0.104
   Compiling quote v1.0.2
   Compiling serde_derive v1.0.104
   Compiling hex v0.4.2 (/home/kent/.cpanm/work/1582182020.22342/hex-0.4.2)
error[E0433]: failed to resolve: use of undeclared type or module `std`
  --> src/serde.rs:19:5
   |
19 | use std::marker::PhantomData;
   |     ^^^ use of undeclared type or module `std`

error[E0432]: unresolved import `std`
  --> src/serde.rs:18:5
   |
18 | use std::fmt;
   |     ^^^ help: a similar path exists: `serde::std`

error[E0412]: cannot find type `String` in this scope
  --> src/serde.rs:31:37
   |
31 |     let s = data.encode_hex_upper::<String>();
   |                                     ^^^^^^ not found in this scope
   |
help: possible candidates are found in other modules, you can import them into scope
   |
15 | use alloc::prelude::v1::String;
   |
15 | use alloc::string::String;
   |

error[E0412]: cannot find type `String` in this scope
  --> src/serde.rs:45:31
   |
45 |     let s = data.encode_hex::<String>();
   |                               ^^^^^^ not found in this scope
   |
help: possible candidates are found in other modules, you can import them into scope
   |
15 | use alloc::prelude::v1::String;
   |
15 | use alloc::string::String;
   |

error[E0412]: cannot find type `PhantomData` in this scope
  --> src/serde.rs:59:29
   |
59 |     struct HexStrVisitor<T>(PhantomData<T>);
   |                             ^^^^^^^^^^^ not found in this scope
   |
help: possible candidates are found in other modules, you can import them into scope
   |
15 | use core::marker::PhantomData;
   |
15 | use serde::export::PhantomData;
   |

error[E0425]: cannot find value `PhantomData` in this scope
  --> src/serde.rs:87:48
   |
87 |     deserializer.deserialize_str(HexStrVisitor(PhantomData))
   |                                                ^^^^^^^^^^^ not found in this scope
   |
help: possible candidates are found in other modules, you can import them into scope
   |
15 | use core::marker::PhantomData;
   |
15 | use serde::export::PhantomData;
   |

error: aborting due to 6 previous errors

Some errors have detailed explanations: E0412, E0425, E0432, E0433.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `hex`.
warning: build failed, waiting for other jobs to finish...
error: build failed

It may not be important shrug. Though I suspect this can be fixed.

I think it could be useful to run code coverage on this repository, to figure out where tests can be improved :)

@KokaKiwi would have to register on codecov.io or coveralls and the ci would run over the entire repository and list some places, that the tests do not cover. Sometimes it has some mismatches, but it works most of the time.

I would recommend to use https://codecov.io/ in combination with cargo tarpaulin (tarpaulin is way better, than kcov)

For reference, you can look at this repository
https://github.com/sile/hls_m3u8
and this is the .travis.yml
https://github.com/sile/hls_m3u8/blob/master/.travis.yml

The only thing I would change is to add an --ignore-panics flag, maybe you could also add caching to improve compilation speed? (the linked .travis.yml has it too).
I would not run this on every target and only on a single one (stable or nightly, but when you want to include doc examples in the coverage, you have to use nightly)

@KokaKiwi hello! Thank you for the great library! Are you still working on the project? I want to prepare some PR.

I'm using this to save some memory working with a large amount of objects that each have a md5, sha256, sha512, etc...

I was wonder if you were interested in adding some official support for these common hex dumps?

For example, for md5:

#[derive(Debug,Hash,PartialEq,Eq,Default)]
pub struct Md5 {
    hash: [u8;16],
}
impl FromHex for Md5 {
....
}
impl FromHex for Md5 {
....
}

I'm working on getting this to integrate with Serde.