google / cityhash Goto Github PK

I noticed the following code in the rotate function:

static uint64 Rotate(uint64 val, int shift)
{
  // Avoid shifting by 64: doing so yields an undefined result.
  return shift == 0 ? val : ((val >> shift) | (val << (64 - shift)));
}

Wouldn't it be faster to do an additional bitwise-and instead of introducing 
branching? For example:

  return ( (val >> shift) | ( val << ((64 - shift) & 63) ) );

Original issue reported on code.google.com by [email protected] on 13 Feb 2013 at 12:36

What steps will reproduce the problem?

1. Attempt to build x64 version of CityHash using VS2008 results in build error 
C2065: 'ssize_t' undeclared identifier.


Workaround:

Add:

#define ssize_t long

...just after #include statements near beginning of CityHash.cpp. This resolves 
the undefined identifier type.

Original issue reported on code.google.com by [email protected] on 24 May 2011 at 9:16

Please provide any additional information below.

Any chance of an extern "C" interface in header files compatible with C 
programs? C programs don't like typedef std::pair<uint64, uint64> uint128 
unfortunately.

Original issue reported on code.google.com by [email protected] on 8 Feb 2012 at 8:15

Quiero ejercer mis derechos de arco y aser el pago con servicios de blokchain

What steps will reproduce the problem?
1. Build for x64 gcc with -Wsign-compare.
2. Observe warning.

What is the expected output? What do you see instead?
No warnings.

../../third_party/externals/cityhash/src/city.cc:146:21: warning: comparison of 
integers of different signs: 'int' and 'size_t' (aka 'unsigned long') 
[-Wsign-compare]
  for (int i = 0; i < len; i++) {
                  ~ ^ ~~~
1 warning generated.


What version of the product are you using? On what operating system?
Revision 11.

Please provide any additional information below.
Patch attached. With patch the exact same code appears to be generated (with 
Release style compile settings) but no warning diagnostic is emitted.

Original issue reported on code.google.com by [email protected] on 7 Dec 2012 at 5:01

Attachments:

• size_t.diff

UnsignedLongs.join does not treat the first long from its arguments as unsigned.

builder.append(array[0]);
for (int i = 1; i < array.length; i++) {
  builder.append(separator).append(toString(array[i]));
}

where it should be builder.append(toString(array[0])).

Original issue reported on code.google.com by wasserman.louis on 9 Apr 2012 at 10:37

What steps will reproduce the problem?
1. CityHash128() all 144-byte strings with 3 bits set
2. check for collisions

What is the expected output? What do you see instead?

0 collisions are expected.  87 collisions are actually seen.  I saw the first 
collision after about 8 million of the keys.  That's the expected collision 
rate for a 46-bit hash, not a 128-bit hash.

(I had a hash of my own last year, AkronHash, that was also a 128-bit hash 
evolved from MurmurHash.  I quickly withdrew it due to similar quality issues.  
My current attempt, SpookyHash, passes this test.)

Original issue reported on code.google.com by [email protected] on 28 Jul 2011 at 8:00

Quiero ejercer mis derechos de arco y aser el pago con servicios de blokchain

I was trying to install cityhash while met the following error. Is seems the 'PyUnicode_AsUTF8AndSize' is only supported by python3.x.

Need to mention, I was using 'python -m pip install ', because we need to use python2.

You can find the 'PyUnicode_AsUTF8AndSize' in https://docs.python.org/3/c-api/unicode.html

Anyone can help this?

cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Isrc -I/usr/include/python2.7 -c src/cityhash.cpp -o build/temp.linux-x86_64-2.7/src/cityhash.o -O3 -Wno-unused-value -Wno-unused-function
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
src/cityhash.cpp: In function ‘PyObject* __pyx_pf_8cityhash_CityHash32(PyObject*, PyObject*)’:
src/cityhash.cpp:1603:17: error: ‘PyUnicode_AsUTF8AndSize’ was not declared in this scope
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 103, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
src/cityhash.cpp:1603:17: note: suggested alternative: ‘PyUnicode_AsUTF8String’
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 103, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
PyUnicode_AsUTF8String
src/cityhash.cpp: In function ‘PyObject __pyx_pf_8cityhash_2CityHash64(PyObject, PyObject*)’:
src/cityhash.cpp:1826:17: error: ‘PyUnicode_AsUTF8AndSize’ was not declared in this scope
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 136, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
src/cityhash.cpp:1826:17: note: suggested alternative: ‘PyUnicode_AsUTF8String’
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 136, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
PyUnicode_AsUTF8String
src/cityhash.cpp: In function ‘PyObject __pyx_pf_8cityhash_4CityHash64WithSeed(PyObject, PyObject*, uint64)’:
src/cityhash.cpp:2107:17: error: ‘PyUnicode_AsUTF8AndSize’ was not declared in this scope
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 171, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
src/cityhash.cpp:2107:17: note: suggested alternative: ‘PyUnicode_AsUTF8String’
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 171, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
PyUnicode_AsUTF8String
src/cityhash.cpp: In function ‘PyObject __pyx_pf_8cityhash_6CityHash64WithSeeds(PyObject, PyObject*, uint64, uint64)’:
src/cityhash.cpp:2404:17: error: ‘PyUnicode_AsUTF8AndSize’ was not declared in this scope
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 207, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
src/cityhash.cpp:2404:17: note: suggested alternative: ‘PyUnicode_AsUTF8String’
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 207, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
PyUnicode_AsUTF8String
src/cityhash.cpp: In function ‘PyObject __pyx_pf_8cityhash_8CityHash128(PyObject, PyObject*)’:
src/cityhash.cpp:2628:17: error: ‘PyUnicode_AsUTF8AndSize’ was not declared in this scope
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 240, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
src/cityhash.cpp:2628:17: note: suggested alternative: ‘PyUnicode_AsUTF8String’
__pyx_t_2 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_2 == ((char const )NULL))) __PYX_ERR(0, 240, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
PyUnicode_AsUTF8String
src/cityhash.cpp: In function ‘PyObject __pyx_pf_8cityhash_10CityHash128WithSeed(PyObject, PyObject*, PyObject*)’:
src/cityhash.cpp:2951:17: error: ‘PyUnicode_AsUTF8AndSize’ was not declared in this scope
__pyx_t_4 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_4 == ((char const *)NULL))) __PYX_ERR(0, 279, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
src/cityhash.cpp:2951:17: note: suggested alternative: ‘PyUnicode_AsUTF8String’
__pyx_t_4 = PyUnicode_AsUTF8AndSize(__pyx_v_data, (&__pyx_v_encoding_size)); if (unlikely(__pyx_t_4 == ((char const *)NULL))) __PYX_ERR(0, 279, __pyx_L1_error)
^~~~~~~~~~~~~~~~~~~~~~~
PyUnicode_AsUTF8String
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

ERROR: Failed building wheel for cityhash
Failed to build cityhash
ERROR: Could not build wheels for cityhash which use PEP 517 and cannot be installed directly

The warning message will be shown like this.

CMake Warning at CMakeLists.txt:10 (message):
  Setting BUILD_NUMBER to the value less than 5180 will lose compatibility
  with SoftEtherVPN Stable Edition 4.x.  Set to a value greater than or equal
  to 5180 if you are likely to connect from 4.x clients.

  See also:
  https://github.com/SoftEtherVPN/SoftEtherVPN/issues/1392#issuecomment-867348281


-- The C compiler identification is Clang 11.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for sys/auxv.h
-- Looking for sys/auxv.h - found
-- Build date: 24/06/2021
-- Build time: 07:51:50
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "1.7.4")
-- Checking for one of the modules 'libsodium'
-- Found Curses: /usr/lib/libcurses.so
-- Found OpenSSL: /usr/local/lib/libcrypto.so (found version "1.1.1k")
-- Found ZLIB: /usr/lib/libz.so (found version "1.2.11")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- The CXX compiler identification is Clang 11.0.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/meta/github/metalefty/SoftEtherVPN

Originally posted by @metalefty in SoftEtherVPN/SoftEtherVPN#1395 (comment)

Add build support for Solaris.

Original issue reported on code.google.com by [email protected] on 17 Jan 2013 at 3:38

Attachments:

• 001-solaris-byteorder.patch

I'm building some wrappers that'll use CityHash32 and CityHash64 under the hood. It'd be helpful for building tests that my wrappers are correct (e.g. pass the right offsets down) if the README.md has some canonical values for the current version, i.e. just

CityHash32("") = ...
CityHash32("some value") = ...

and same for 64.

Obviously these are unstable and could change, but it'd still be highly helpful. Thanks!

Recently noticed a downstream use of cityhash had trouble with compiling on 
FreeBSD and independently reached the same conclusion as 
https://code.google.com/p/cityhash/issues/detail?id=12.

At the same time, also looked into whether OpenBSD might have a similar issue.  
Believe something like the attached will be needed for OpenBSD, though do not 
have an OpenBSD machine to test on currently.

Original issue reported on code.google.com by [email protected] on 28 Jul 2013 at 4:39

Attachments:

• city.cc.diff

3433874

A comment in city.h says:
// By the way, for some hash functions, given strings a and b, the hash
// of a+b is easily derived from the hashes of a and b.  This property
// doesn't hold for any hash functions in this file.

But I didn't find how it can be done. It would be nice to have this feature. 
For example, it can be used to hash a data stream that becomes available one 
block at a time, but the block sizes are arbitrary, and we need both the hash 
of each block and the hash of the whole stream (invariant of block sizes).

Original issue reported on code.google.com by [email protected] on 26 Mar 2014 at 9:54

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md

Some test vectors to check whether CityHash has been compiled and executes 
correctly or whether another implementation produces the same output would be 
greatly appreciated.

Original issue reported on code.google.com by [email protected] on 13 Apr 2011 at 7:25

this may help:

https://gitorious.org/cityhash_c/cityhash_c/blobs/master/configure.ac

Original issue reported on code.google.com by [email protected] on 4 Aug 2011 at 3:45

Quiero ejercer mis derechos de arco y aser el pago con servicios de blokchain

I'm added patch with fix for build on FreeBSD.

Thank you.

Original issue reported on code.google.com by G.Veniamin on 25 Dec 2012 at 1:01

Attachments:

• patch-src_city.cc

@Hernandez00161

I really appreciated the fact that a version of cityhash can provide 256 bit output, e.g. when using CityHashCrc256(). However, to get that going I had to build with ./configure --enable-sse4.2 and make all check CXXFLAGS="-g -O3 -msse4.2" (as per instructions). As far as my understanding goes, however, SSE4.2 is not supported on a Raspberry Pi 3.

So I was wondering: how can I get a 256 bit output on that platform? Are there workarounds that don't use hardware-accelerated CRC?

The README for CityHash describes CityHashCrc128() and CityHashCrc256(), suggesting they are part of the published API.

However, they are only declared in citycrc.h, which is not installed by make install (it only installs the city.h header). As a result, the *Crc* functions aren't cleanly available to package-based installations of CityHash.

Please include citycrc.h in the installed files list.

Added: tested on macOS. It seems it is installed on Linux...

Do you have any plan for implementing them for 32 platforms ?

Having this stuff only for 64 bit platforms is really limiting, as most 
software has to still support 32 bits.

The 128 and CRC variants could be really useful for 32 bits.

Original issue reported on code.google.com by [email protected] on 28 Jan 2012 at 8:57

Envs:

• win 10
• anaconda
• python 3.6
• pip install cityhash

Error:

  cl: 命令行 error D8021 :无效的数值参数“/Wno-unused-value”
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.21.27702\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

  Failed building wheel for cityhash

It would be nice to have a 32-bit version for use in e.g. hash table 
implementations on 32-bit systems.

Original issue reported on code.google.com by [email protected] on 19 Apr 2011 at 8:05

Greetings cityhash open source developers and contributors,

We’re reaching out because your project is an important part of the open source ecosystem, and we’d like to invite you to integrate with our fuzzing service, OSS-Fuzz. OSS-Fuzz is a free fuzzing infrastructure you can use to identify security vulnerabilities and stability bugs in your project. OSS-Fuzz will:

• Continuously run all the fuzzers you write.
• Alert you when it finds issues.
• Automatically close issues after they’ve been fixed by a commit.

Many widely used open source projects like OpenSSL, FFmpeg, LibreOffice, and ImageMagick are fuzzing via OSS-Fuzz, which helps them find and remediate critical issues.

Even though typical integrations can be done in < 100 LoC, we have a reward program in place which aims to recognize folks who are not just contributing to open source, but are also working hard to make it more secure.

We want to stress that anyone who meets the eligibility criteria and integrates a project with OSS-Fuzz is eligible for a reward.

To help you getting started, we can provide our internal fuzzer for your project that you are welcome to use directly, or to use it as a starting point.

If you're not interested in integrating with OSS-Fuzz, it would be helpful for us to understand why—lack of interest, lack of time, or something else—so we can better support projects like yours in the future.

If we’ve missed your question in our FAQ, feel free to reply or reach out to us at [email protected].

Thanks!

Tommy
OSS-Fuzz Team

 Given the exact same input twice, the hash function should always produce the same output. I cann't understand why cityhash32 and cityhash64 function produce the different output when giving the exact same input string twice.

Original issue reported on code.google.com by [email protected] on 23 Aug 2013 at 2:40

Hi, I need to package this lib for debian as another package depends on it. Could you please publish a release ? This would ease the packaging process.

thanks

What steps will reproduce the problem?
1. char foo[11] = "Hello, World!"; char bar[11] = "Hello, World?";
2. uint128 _foo = CityHash128(foo, 11); uint128 _bar = CityHash128(bar, 11);
3. Print the results. They are the same.

What is the expected output? What do you see instead? I expect a different 
number. I see two identical ones.


What version of the product are you using? On what operating system? Linux 
(Debian/Ubuntu 12.04) 64-bit AMD. Using version 1.03.


Please provide any additional information below.
It may be that this was done on purpose, but if not, I wanted to bring it to 
your attention!

Original issue reported on code.google.com by [email protected] on 11 Jun 2012 at 7:48

From http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-6051

> Google CityHash computes hash values without properly restricting the 
> ability to trigger hash collisions predictably, which allows context-
> dependent attackers to cause a denial of service (CPU consumption) via 
> crafted input to an application that maintains a hash table, as 
> demonstrated by a universal multicollision attack.

Given that the NEWS file does not mention this, is v1.1.0 affected?

Original issue reported on code.google.com by Al3xBio on 3 Dec 2012 at 10:40