Opened 5 years ago

Closed 3 years ago

#1216 closed enhancement (fixed)

Update GMP/jbigi

Reported by: zzz Owned by: zzz
Priority: maintenance Milestone: 0.9.26
Component: api/crypto Version: 0.9.11
Keywords: Cc: orignal, dg, tuna
Parent Tickets:

Description

Last updated May/June? 2011. We currently use 4.3.2 for 32 bit and 5.0.2 for 64 bit. Current version is 5.1.3, and 5.2.0 was due in January 2014 but is not yet released. 5.1.3 is probably safer.

  • Investigate whether to update 32 bit, is it faster than 4.3.2 now? Worth the size increase?
  • Add new architectures
  • Investigate whether to use constant-time mpz_powm_sec
  • Investigate whether to add other functions (I tested multiply, divide, and modInverse - none look like a winner)
  • Add version functions in jbigi.c for the jbigi and GMP versions (I have a patch already)
  • Investigate whether to replace the whole thing with infinity0's GMP-Java
  • Investigate whether to do it in Java (or our own C code) instead, orignal says we can optimize for our ElG prime. Speed test i2pd code vs. mpz_powm.

As before, this is a big effort, that's why we don't do it too often.

ref:

​https://gmplib.org/
​https://gmplib.org/repo/gmp/file/tip/NEWS
​https://github.com/infinity0/gmp-java

Subtickets

#1594: jbigi not available on raspberry pi 2defectclosed

Change History (29)

comment:1 Changed 5 years ago by zzz

  • Cc orignal dg added

<dg> I looked around July 2013 and there were no significant improvements but maybe I was wrong
<dg> or maybe they've improved since then

comment:2 Changed 5 years ago by dg

​https://gmplib.org/repo/gmp/file/tip/NEWS
There were arch recognition improvements as well as general speedups.
Here are some fixes we may gain in either (or both) 64bit and 32bit:

Changes between GMP version 5.1.* and 5.2.0
  * Plain division of large operands is faster and more monotonous in operand
    size.
  * Speedup for Intel Sandy Bridge, Ivy Bridge, Haswell, thanks to rewritten
    and vastly expanded assembly support.
  * Major speedup for ARM, in particular ARM Cortex-A15, thanks to improved
    assembly.
  * Major speedup for SPARC T4/T5 and speedup also for T3, thanks to a lot of
    new assembly.

Changes between GMP version 5.0.3 and 5.0.4
  * Two bugs in multiplication code causing incorrect computation with
    extremely low probability have been fixed.

Changes between GMP version 5.0.0 and 5.0.1
  * Multiplication of large numbers has indirectly been sped up through better
    FFT tuning and processor recognition.  Since many operations depend on
    multiplication, there will be a general speedup.

Changes between GMP version 5.0.* and 5.1.0
  * Major speedup for ARM, in particular ARM Cortex-A9 and A15, thanks to broad
    assembly support.
  * The performance under M$ Windows' 64-bit ABI has been greatly improved
    thanks to complete assembly support.

Changes between GMP version 4.3.X and 5.0.0
  * Multiplication has been overhauled:
    (1) Multiplication of larger same size operands has been improved with
        the addition of two new Toom functions and a new internal function
        mpn_mulmod_bnm1 (computing U * V mod (B^n-1), B being the word base.
        This latter function is used for the largest products, waiting for a
        better Schoenhage-Strassen U * V mod (B^n+1) implementation.
    (2) Likewise for squaring.
    (3) Multiplication of different size operands has been improved with the
        addition of many new Toom function, and by selecting underlying
        functions better from the main multiply functions.
  * Division and mod have been overhauled:
    (1) Plain "schoolbook" division is reimplemented using faster quotient
        approximation.
    (2) Division Q = N/D, R = N mod D where both the quotient and remainder
        are needed now runs in time O(M(log(N))).  This is an improvement of
        a factor log(log(N))
    (3) Division where just the quotient is needed is now O(M(log(Q))) on
        average.
    (4) Modulo operations using Montgomery REDC form now take time O(M(n)).
    (5) Exact division Q = N/D by means of mpz_divexact has been improved
        for all sizes, and now runs in time O(M(log(N))).

  * The function mpz_powm is now faster for all sizes.  Its complexity has
    gone from O(M(n)log(n)m) to O(M(n)m) where n is the size of the modulo
    argument and m is the size of the exponent.  It is also radically
    faster for even modulus, since it now partially factors such modulus
    and performs two smaller modexp operations, then uses CRT.

Perhaps multiplication/division improvements since you last tried them (which GMP version? the one we ship? 32bit or 64bit?) have improved to the point they could be useful?

Last edited 5 years ago by dg (previous) (diff)

comment:3 Changed 5 years ago by zzz

GMP 6 just released.

comment:4 Changed 4 years ago by str4d

  • Milestone changed from 0.9.15 to undecided

comment:5 Changed 4 years ago by zzz

  • Owner set to zzz
  • Status changed from new to accepted

Initial work in branch i2p.i2p.zzz.gmp6 2014-11-30

Name  : changelog
Value : libjbigi updates:
      :    Added:
      :      nativeJbigiVersion()
      :      nativeGMPMajorVersion()
      :      nativeGMPMinorVersion()
      :      nativeGMPPatchVersion()
      :      nativeModInverse()
      :      nativeModPowCT()
      :    Support negative vaules in conversion functions
      :    Support negative base value in modPow()
      :    Throw ArithmeticException for bad arguments rather than crashing
      :    Switch to GMP 6.0.0
      :    New build targets
      : NativeBigInteger changes:
      :    Test improvements
      :    Version reporting
      : Crypto changes:
      :    Use constant time modPow() for signing and decryption
      :    Use native modInverse()

comment:6 Changed 4 years ago by zzz

The plan:

<zzz2> the prop won't be anytime soon at this rate
<zzz2> 2-6 months maybe, depending on progress
<hottuna_> zzz2, what more work needs to be done on zzz.gmp6?
<zzz2> 1) figure out what's a _real_ duplicate and what's not
<zzz2> 2) pick the names for the new targets
<zzz2> 3) add support for the new ones to CPUID
<zzz2> 4) add support for the new ones to NBI
<zzz2> 5, 6, 7, 8, 9) test test test test test
<zzz2> 10) decide which binaries to check in (e.g. probably not any new 32-bit ones)
<zzz2> 11) check in binaries
<zzz2> 12) prop

comment:7 Changed 4 years ago by tuna

1)

$ fdupes -r t/
./libjbigi-linux-core2.so
./libjbigi-linux-corei.so

./libjbigi-freebsd-corei.so
./libjbigi-freebsd-core2.so

./libjbigi-osx-core2.jnilib
./libjbigi-osx-corei.jnili

2) Since corei and core2 are identical lets collapse them under the core2 name.
I vote we support every (x86) GMP-target that produces a unique binary. If it's a duplicate of something else, collapse it under the name of the first target.

comment:8 Changed 4 years ago by tuna

3) CPUID support committed in rev dc946295083bf26424800d4e8073e7ef2e25d955

Added AVX/AVX2/AVX512/TDM feature detection.
Added support for AMD archs with support in GMP.
Added support for Intel archs with support in GMP.

Last edited 4 years ago by tuna (previous) (diff)

comment:9 Changed 4 years ago by tuna

Note to self: Update ​https://geti2p.net/en/misc/jbigi

comment:10 Changed 4 years ago by tuna

4) NBI support committed in rev 779a6dd5df9f586d36b2381cc03c77b8beebf26a

Added support for AMD archs with support in GMP.
Added support for Intel archs with support in GMP.

comment:11 Changed 4 years ago by tuna

I did some quick gmp6 benchmarking, and it would seem that there are minor performance gains to be had. These are the results from a single run, so take them with a grain of salt.

64-bit on Haswell CPU
java:        2103.863ms -> 1x
gmp5 corei:   445.706ms -> 4.72x
gmp6 corei:   433.965ms -> 4.85x
gmp6 corehwl: 340.794ms -> 6.10x

32-bit on Haswell CPU
java:        5167.245ms -> 1x
gmp4 core2:  1422.978ms -> 3.63x
gmp6 core2:  1300.405ms -> 3.97x
Last edited 4 years ago by tuna (previous) (diff)

comment:12 Changed 4 years ago by tuna

Test run on an Raspberry Pi 2 (armcortexa7neon hw suppport).

Conclusions:

  • The armv5/armv7 libs we ship are flat out broken (can't be loaded)
  • NEON support in the lib does not improve performance on gmp6 nor gmpdev (trunk of gmp repo)
  • gmp is going to add (armcortexa5, armcortexa7, armcortexa7neon) and rename (armv7a -> armv7) a few libs. We might aswell use the upcoming naming scheme in what we're writing for NBI now.
  • Some archs are performance incompatible, A7 and A15 for example.
  • From following the gmp mailing list, ARM NEON and A15 improvemens are being worked on. 
    java:                     155783.125ms -> 1x

gmp5 armv5 - error, not valid for this platform (this is a bug)
gmp5 armv6                 11915.303ms -> 13.07x
gmp5 armv7 - error, not valid for this platform (this is a bug)

gmp6 armv5                 12195.651ms -> 12.77x
gmp6 armv6                  7001.033ms -> 22.25x
gmp6 armv7                  6995.054ms -> 22.27x
gmp6 armcortax8             7007.696ms -> 22.23x
gmp6 armcortexa8neon        7040.535ms -> 22.13x
gmp6 armcortexa9            6995.969ms -> 22.27x
gmp6 armcortexa9neon        6996.123ms -> 22.27x
gmp6 armcortexa15           9702.768ms -> 16.06x
gmp6 armcortexa15neon       9693.381ms -> 16.07x

gmpdev armv5               12198.676ms -> 12.77x
gmpdev armv6                6747.837ms -> 23.09x
gmpdev armv7                6747.071ms -> 23.09x
gmpdev armcortexa7          6754.066ms -> 23.07x
gmpdev armcortexa7neon      6751.755ms -> 23.07x
gmpdev armcortexa8          6787.963ms -> 22.94x
gmpdev armcortexa8neon      6796.246ms -> 22.92x
gmpdev armcortexa9          6755.460ms -> 23.06x
gmpdev armcortexa9neon      6780.304ms -> 22.97x
gmpdev armcortexa15         9721.847ms -> 16.02x
gmpdev armcortexa15neon     9712.354ms -> 16.04x
Last edited 4 years ago by tuna (previous) (diff)

comment:13 Changed 4 years ago by plughplover

Add a subticket #1594.

comment:14 Changed 4 years ago by zzz

Tuna says the coding is done and is requesting a review.

To review:

mtn dif -r t:i2p-0.9.17 -r h:i2p.i2p.zzz.gmp6

Need eyes on the added methods, the places where constant time is used, the big NBI fallback rewrite, compatibility old router/new jbigi and vice versa, all the new ARM code, the decision to name ARM binaries based on a GMP version that won't be out for a long time, etc

comment:15 Changed 4 years ago by tuna

What remains to be done:

21) Build binaries

  • FreeBSD: x86 binaries
  • OSX: x86 binaries
  • Linux: x86 and ARM binaries
  • Windows: x86 binaries

22) Test all binaries

23) Check in binaries

24) Propagate into i2p.i2p

comment:16 Changed 4 years ago by tuna

21) FreeBSD:

Binaries hosted here: ​http://www.filedropper.com/libjbigi-freebsd

FreeBSD on Haswell hardware in Virtualbox VM

java:            2366.032ms -> 1x

gmp5 core2_64     399.924ms -> 5.91x
gmp5 corei_64     421.694ms -> 5.61x

gmp6 core2_64     513.252ms -> 4.61x
gmp6 corei_64     411.198ms -> 5.75x
gmp6 coreisbr_64  333.394ms -> 7.10x
gmp6 coreihwl_64  328.180ms -> 7.21x
Last edited 4 years ago by tuna (previous) (diff)

comment:17 Changed 4 years ago by tuna

21) Windows

Binaries ḧosted here: ​http://www.filedropper.com/jbigi-windows

Windows on Haswell hardware in a Virtualbox VM

java              2156.654ms -> 1x

gmp5 core2_64      866.416ms -> 2.49x
gmp5 corei_64      841.125ms -> 2.56x

gmp6 core2_64      447.529ms -> 4.82x
gmp6 corei_64      452.099ms -> 4.77x
gmp6 coreisbr_64   467.581ms -> 4.61x
gmp6 coreihwl_64   331.959ms -> 6.50x
Last edited 4 years ago by tuna (previous) (diff)

comment:18 Changed 4 years ago by tuna

21) OSX

New 32-bit binaries will not be provided since there is no >1.6 JRE available for OSX.

Performance: I'm too lazy to make another table, but preliminary testing suggests the same pattern as on FreeBSD and Windows. Exepect a ~6x speedup on Haswell hardware.

Last edited 4 years ago by tuna (previous) (diff)

comment:19 Changed 4 years ago by zzz

tuna asked for testing of ​http://treze2cubeytypnlfuwmvgmeygolpumgfse3ka76ndudvjs24loa.b32.i2p/new_jbigi.zip yesterday, said it is final.

Putting issues here since comments seem to get completely lost on IRC.

Initial findings:

Here's what's changed and what hasn't:
(diff -rs old new)

Binary files old/jbigi-windows-athlon64_64.dll and new/jbigi-windows-athlon64_64.dll differ
Files old/jbigi-windows-athlon64.dll and new/jbigi-windows-athlon64.dll are identical
Files old/jbigi-windows-athlon.dll and new/jbigi-windows-athlon.dll are identical
Binary files old/jbigi-windows-atom_64.dll and new/jbigi-windows-atom_64.dll differ
Binary files old/jbigi-windows-atom.dll and new/jbigi-windows-atom.dll differ
Only in new: jbigi-windows-bobcat_64.dll
Only in new: jbigi-windows-bobcat.dll
Only in new: jbigi-windows-bulldozer_64.dll
Only in new: jbigi-windows-bulldozer.dll
Binary files old/jbigi-windows-core2_64.dll and new/jbigi-windows-core2_64.dll differ
Binary files old/jbigi-windows-core2.dll and new/jbigi-windows-core2.dll differ
Binary files old/jbigi-windows-corei_64.dll and new/jbigi-windows-corei_64.dll differ
Only in new: jbigi-windows-coreibwl_64.dll
Only in new: jbigi-windows-coreibwl.dll
Binary files old/jbigi-windows-corei.dll and new/jbigi-windows-corei.dll differ
Only in new: jbigi-windows-coreihwl_64.dll
Only in new: jbigi-windows-coreihwl.dll
Only in new: jbigi-windows-coreisbr_64.dll
Only in new: jbigi-windows-excavator_64.dll
Only in new: jbigi-windows-excavator.dll
Binary files old/jbigi-windows-geode.dll and new/jbigi-windows-geode.dll differ
Only in new: jbigi-windows-jaguar.dll
Files old/jbigi-windows-k62.dll and new/jbigi-windows-k62.dll are identical
Files old/jbigi-windows-k63.dll and new/jbigi-windows-k63.dll are identical
Binary files old/jbigi-windows-k6.dll and new/jbigi-windows-k6.dll differ
Files old/jbigi-windows-nano_64.dll and new/jbigi-windows-nano_64.dll are identical
Files old/jbigi-windows-nano.dll and new/jbigi-windows-nano.dll are identical
Files old/jbigi-windows-none_64.dll and new/jbigi-windows-none_64.dll are identical
Files old/jbigi-windows-none.dll and new/jbigi-windows-none.dll are identical
Files old/jbigi-windows-pentium2.dll and new/jbigi-windows-pentium2.dll are identical
Files old/jbigi-windows-pentium3.dll and new/jbigi-windows-pentium3.dll are identical
Files old/jbigi-windows-pentium4_64.dll and new/jbigi-windows-pentium4_64.dll are identical
Files old/jbigi-windows-pentium4.dll and new/jbigi-windows-pentium4.dll are identical
Files old/jbigi-windows-pentium.dll and new/jbigi-windows-pentium.dll are identical
Files old/jbigi-windows-pentiumm.dll and new/jbigi-windows-pentiumm.dll are identical
Files old/jbigi-windows-pentiummmx.dll and new/jbigi-windows-pentiummmx.dll are identical
Only in new: jbigi-windows-piledriver_64.dll
Only in new: jbigi-windows-piledriver.dll
Only in new: jbigi-windows-steamroller_64.dll
Only in new: jbigi-windows-steamroller.dll
Only in new: jbigi-windows-viac32.dll
Binary files old/jbigi-windows-viac3.dll and new/jbigi-windows-viac3.dll differ
Files old/jcpuid-x86_64-windows.dll and new/jcpuid-x86_64-windows.dll are identical
Files old/jcpuid-x86-windows.dll and new/jcpuid-x86-windows.dll are identical
Binary files old/libjbigi-freebsd-athlon64_64.so and new/libjbigi-freebsd-athlon64_64.so differ
Binary files old/libjbigi-freebsd-athlon64.so and new/libjbigi-freebsd-athlon64.so differ
Binary files old/libjbigi-freebsd-athlon.so and new/libjbigi-freebsd-athlon.so differ
Binary files old/libjbigi-freebsd-atom_64.so and new/libjbigi-freebsd-atom_64.so differ
Binary files old/libjbigi-freebsd-atom.so and new/libjbigi-freebsd-atom.so differ
Only in new: libjbigi-freebsd-bobcat_64.so
Only in new: libjbigi-freebsd-bobcat.so
Only in new: libjbigi-freebsd-bulldozer_64.so
Only in new: libjbigi-freebsd-bulldozer.so
Binary files old/libjbigi-freebsd-core2_64.so and new/libjbigi-freebsd-core2_64.so differ
Binary files old/libjbigi-freebsd-core2.so and new/libjbigi-freebsd-core2.so differ
Binary files old/libjbigi-freebsd-corei_64.so and new/libjbigi-freebsd-corei_64.so differ
Only in new: libjbigi-freebsd-coreibwl_64.so
Only in new: libjbigi-freebsd-coreibwl.so
Only in new: libjbigi-freebsd-coreihwl_64.so
Only in new: libjbigi-freebsd-coreihwl.so
Only in new: libjbigi-freebsd-coreisbr_64.so
Only in new: libjbigi-freebsd-coreisbr.so
Binary files old/libjbigi-freebsd-corei.so and new/libjbigi-freebsd-corei.so differ
Only in new: libjbigi-freebsd-excavator_64.so
Only in new: libjbigi-freebsd-excavator.so
Binary files old/libjbigi-freebsd-geode.so and new/libjbigi-freebsd-geode.so differ
Only in new: libjbigi-freebsd-jaguar_64.so
Only in new: libjbigi-freebsd-jaguar.so
Only in new: libjbigi-freebsd-k10_64.so
Only in new: libjbigi-freebsd-k10.so
Binary files old/libjbigi-freebsd-k62.so and new/libjbigi-freebsd-k62.so differ
Only in new: libjbigi-freebsd-k63.so
Binary files old/libjbigi-freebsd-k6.so and new/libjbigi-freebsd-k6.so differ
Binary files old/libjbigi-freebsd-nano_64.so and new/libjbigi-freebsd-nano_64.so differ
Binary files old/libjbigi-freebsd-nano.so and new/libjbigi-freebsd-nano.so differ
Binary files old/libjbigi-freebsd-none_64.so and new/libjbigi-freebsd-none_64.so differ
Binary files old/libjbigi-freebsd-none.so and new/libjbigi-freebsd-none.so differ
Binary files old/libjbigi-freebsd-pentium2.so and new/libjbigi-freebsd-pentium2.so differ
Binary files old/libjbigi-freebsd-pentium3.so and new/libjbigi-freebsd-pentium3.so differ
Binary files old/libjbigi-freebsd-pentium4_64.so and new/libjbigi-freebsd-pentium4_64.so differ
Binary files old/libjbigi-freebsd-pentium4.so and new/libjbigi-freebsd-pentium4.so differ
Binary files old/libjbigi-freebsd-pentiummmx.so and new/libjbigi-freebsd-pentiummmx.so differ
Binary files old/libjbigi-freebsd-pentiumm.so and new/libjbigi-freebsd-pentiumm.so differ
Binary files old/libjbigi-freebsd-pentium.so and new/libjbigi-freebsd-pentium.so differ
Only in new: libjbigi-freebsd-piledriver_64.so
Only in new: libjbigi-freebsd-piledriver.so
Only in new: libjbigi-freebsd-steamroller_64.so
Only in new: libjbigi-freebsd-steamroller.so
Only in new: libjbigi-freebsd-viac32.so
Binary files old/libjbigi-freebsd-viac3.so and new/libjbigi-freebsd-viac3.so differ
Only in new: libjbigi-linux-armcortexa15.so
Only in new: libjbigi-linux-armcortexa8.so
Only in new: libjbigi-linux-armcortexa9.so
Binary files old/libjbigi-linux-armv5.so and new/libjbigi-linux-armv5.so differ
Binary files old/libjbigi-linux-armv6.so and new/libjbigi-linux-armv6.so differ
Binary files old/libjbigi-linux-armv7.so and new/libjbigi-linux-armv7.so differ
Binary files old/libjbigi-linux-athlon64_64.so and new/libjbigi-linux-athlon64_64.so differ
Binary files old/libjbigi-linux-athlon64.so and new/libjbigi-linux-athlon64.so differ
Binary files old/libjbigi-linux-athlon.so and new/libjbigi-linux-athlon.so differ
Binary files old/libjbigi-linux-atom_64.so and new/libjbigi-linux-atom_64.so differ
Binary files old/libjbigi-linux-atom.so and new/libjbigi-linux-atom.so differ
Binary files old/libjbigi-linux-core2_64.so and new/libjbigi-linux-core2_64.so differ
Binary files old/libjbigi-linux-core2.so and new/libjbigi-linux-core2.so differ
Binary files old/libjbigi-linux-corei_64.so and new/libjbigi-linux-corei_64.so differ
Only in new: libjbigi-linux-coreibwl_64.so
Only in new: libjbigi-linux-coreibwl.so
Only in new: libjbigi-linux-coreihwl_64.so
Only in new: libjbigi-linux-coreihwl.so
Only in new: libjbigi-linux-coreisbr_64.so
Only in new: libjbigi-linux-coreisbr.so
Binary files old/libjbigi-linux-corei.so and new/libjbigi-linux-corei.so differ
Binary files old/libjbigi-linux-geode.so and new/libjbigi-linux-geode.so differ
Binary files old/libjbigi-linux-k62.so and new/libjbigi-linux-k62.so differ
Binary files old/libjbigi-linux-k6.so and new/libjbigi-linux-k6.so differ
Binary files old/libjbigi-linux-nano_64.so and new/libjbigi-linux-nano_64.so differ
Binary files old/libjbigi-linux-nano.so and new/libjbigi-linux-nano.so differ
Binary files old/libjbigi-linux-none_64.so and new/libjbigi-linux-none_64.so differ
Binary files old/libjbigi-linux-none.so and new/libjbigi-linux-none.so differ
Binary files old/libjbigi-linux-pentium2.so and new/libjbigi-linux-pentium2.so differ
Binary files old/libjbigi-linux-pentium3.so and new/libjbigi-linux-pentium3.so differ
Binary files old/libjbigi-linux-pentium4_64.so and new/libjbigi-linux-pentium4_64.so differ
Binary files old/libjbigi-linux-pentium4.so and new/libjbigi-linux-pentium4.so differ
Binary files old/libjbigi-linux-pentiummmx.so and new/libjbigi-linux-pentiummmx.so differ
Binary files old/libjbigi-linux-pentiumm.so and new/libjbigi-linux-pentiumm.so differ
Binary files old/libjbigi-linux-pentium.so and new/libjbigi-linux-pentium.so differ
Files old/libjbigi-linux-ppc.so and new/libjbigi-linux-ppc.so are identical
Binary files old/libjbigi-linux-viac3.so and new/libjbigi-linux-viac3.so differ
Binary files old/libjbigi-osx-core2_64.jnilib and new/libjbigi-osx-core2_64.jnilib differ
Files old/libjbigi-osx-core2.jnilib and new/libjbigi-osx-core2.jnilib are identical
Binary files old/libjbigi-osx-corei_64.jnilib and new/libjbigi-osx-corei_64.jnilib differ
Only in new: libjbigi-osx-coreibwl_64.jnilib
Only in new: libjbigi-osx-coreihwl_64.jnilib
Only in old: libjbigi-osx-corei.jnilib
Only in new: libjbigi-osx-coreisbr_64.jnilib
Files old/libjbigi-osx-none_64.jnilib and new/libjbigi-osx-none_64.jnilib are identical
Files old/libjbigi-osx-none.jnilib and new/libjbigi-osx-none.jnilib are identical
Files old/libjcpuid-x86_64-freebsd.so and new/libjcpuid-x86_64-freebsd.so are identical
Files old/libjcpuid-x86_64-kfreebsd.so and new/libjcpuid-x86_64-kfreebsd.so are identical
Files old/libjcpuid-x86_64-linux.so and new/libjcpuid-x86_64-linux.so are identical
Files old/libjcpuid-x86_64-netbsd.so and new/libjcpuid-x86_64-netbsd.so are identical
Files old/libjcpuid-x86_64-openbsd.so and new/libjcpuid-x86_64-openbsd.so are identical
Files old/libjcpuid-x86_64-osx.jnilib and new/libjcpuid-x86_64-osx.jnilib are identical
Files old/libjcpuid-x86_64-solaris.so and new/libjcpuid-x86_64-solaris.so are identical
Files old/libjcpuid-x86-freebsd.so and new/libjcpuid-x86-freebsd.so are identical
Files old/libjcpuid-x86-kfreebsd.so and new/libjcpuid-x86-kfreebsd.so are identical
Files old/libjcpuid-x86-linux.so and new/libjcpuid-x86-linux.so are identical
Files old/libjcpuid-x86-netbsd.so and new/libjcpuid-x86-netbsd.so are identical
Files old/libjcpuid-x86-openbsd.so and new/libjcpuid-x86-openbsd.so are identical
Files old/libjcpuid-x86-osx.jnilib and new/libjcpuid-x86-osx.jnilib are identical
Files old/libjcpuid-x86-solaris.so and new/libjcpuid-x86-solaris.so are identical

File removed from old:

Only in old: libjbigi-osx-corei.jnilib

Is this intentional? You're proposing to delete it? Why not leave the 32 bit in there? Is there no hardware out there for it?
Any others that should be removed? iirc there were some identical files in there, but don't have a list.

Following files are not stripped, so these must be fixed:
(file * | grep not)

libjbigi-freebsd-athlon64_64.so:    ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=33ba5f9e8a33e5ace8dd60d4dcc00a633d483198, not stripped
libjbigi-freebsd-athlon64.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=ee1f51f2ad4efa31c1c1942625b94a9fa2fb0ad2, not stripped
libjbigi-freebsd-athlon.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=755418098aeb517fc73f379f0fad7d993c1b34bc, not stripped
libjbigi-freebsd-atom_64.so:        ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=36d1abaf93ce330d8429f56880aef9e4ee23f727, not stripped
libjbigi-freebsd-atom.so:           ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=49c42fcbf3e8909792aef0425841b37283c74d6a, not stripped
libjbigi-freebsd-bobcat_64.so:      ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=2130505000248a2a94e234bd6c25542a4566f8c9, not stripped
libjbigi-freebsd-bobcat.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=6dd7bce5153610e0307531ca172fdd32d0d18216, not stripped
libjbigi-freebsd-bulldozer_64.so:   ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=e76d95d97edb245f786b1c252b5094284b74f697, not stripped
libjbigi-freebsd-bulldozer.so:      ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=5ec3cd5a8b069c23b85efb035ed456cb883ccf24, not stripped
libjbigi-freebsd-core2_64.so:       ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=c8757c3bcca31e3f98b5f65a27c3a9438285d440, not stripped
libjbigi-freebsd-core2.so:          ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=ec83e863ef1d14be26afe33e18883f390f8d696d, not stripped
libjbigi-freebsd-corei_64.so:       ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=35e05810f92b86e7f445146c2d5dc436949c7ff7, not stripped
libjbigi-freebsd-coreibwl_64.so:    ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=f18eedf184040752824ec12f0dafd722f9f48a70, not stripped
libjbigi-freebsd-coreibwl.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=f5c054a3b292f285326368e3dd2b11a0ee8f06ea, not stripped
libjbigi-freebsd-coreihwl_64.so:    ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=f18eedf184040752824ec12f0dafd722f9f48a70, not stripped
libjbigi-freebsd-coreihwl.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=f5c054a3b292f285326368e3dd2b11a0ee8f06ea, not stripped
libjbigi-freebsd-coreisbr_64.so:    ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=212113ae7f653b4142493612091bd7b56f29978d, not stripped
libjbigi-freebsd-coreisbr.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=f5c054a3b292f285326368e3dd2b11a0ee8f06ea, not stripped
libjbigi-freebsd-corei.so:          ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=78d6aaee59b4d5f07f7e978ceb43de1c445b464e, not stripped
libjbigi-freebsd-excavator_64.so:   ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=b910a2085239daad85e0d9a70ff37f3550eb3cf8, not stripped
libjbigi-freebsd-excavator.so:      ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=1aab7ba56d44c4c83d230ce34430bc5e23b15bb8, not stripped
libjbigi-freebsd-geode.so:          ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=cb4b8a4bbbb6e7e67bdcacd652d327439e40d11f, not stripped
libjbigi-freebsd-jaguar_64.so:      ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=2130505000248a2a94e234bd6c25542a4566f8c9, not stripped
libjbigi-freebsd-jaguar.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=6dd7bce5153610e0307531ca172fdd32d0d18216, not stripped
libjbigi-freebsd-k10_64.so:         ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=e048aea5319a2526f72ab3461f0165abb21a25cc, not stripped
libjbigi-freebsd-k10.so:            ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=22c0b464eaf39310296a122731827deafd239b3d, not stripped
libjbigi-freebsd-k62.so:            ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=2b48e59d1c74299533ee2df223213344e2dd7517, not stripped
libjbigi-freebsd-k63.so:            ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=2b48e59d1c74299533ee2df223213344e2dd7517, not stripped
libjbigi-freebsd-k6.so:             ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=44dd40bc47bb631afdad8d7cd37d29860b5b11d7, not stripped
libjbigi-freebsd-nano_64.so:        ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fa015119fd14213283b5b8f938da52733366915f, not stripped
libjbigi-freebsd-nano.so:           ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=fc84d4f89aa536d98ee94b365b40c6c6400b11b0, not stripped
libjbigi-freebsd-none_64.so:        ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=33ba5f9e8a33e5ace8dd60d4dcc00a633d483198, not stripped
libjbigi-freebsd-none.so:           ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=8cf0e4dde1aebd38d49d9d36997c3b5f84f55cfe, not stripped
libjbigi-freebsd-pentium2.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=9596dfdf1027f082872f9c56ae397b1fb89a4064, not stripped
libjbigi-freebsd-pentium3.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=9596dfdf1027f082872f9c56ae397b1fb89a4064, not stripped
libjbigi-freebsd-pentium4_64.so:    ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d127d5177c71a5d3e3da5a851fea004493436a6d, not stripped
libjbigi-freebsd-pentium4.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=c23f89c3a2109bd95212b643153c323cdafedf8c, not stripped
libjbigi-freebsd-pentiummmx.so:     ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=23cbcf81fe7273b7a6aa8ba3f0cf4fd9699d0fb1, not stripped
libjbigi-freebsd-pentiumm.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=d3be5e5f89359c875887067fc37592f7ee802d1c, not stripped
libjbigi-freebsd-pentium.so:        ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=408eaacacc3aaf1412cf63261122d455f302165e, not stripped
libjbigi-freebsd-piledriver_64.so:  ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=b910a2085239daad85e0d9a70ff37f3550eb3cf8, not stripped
libjbigi-freebsd-piledriver.so:     ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=1aab7ba56d44c4c83d230ce34430bc5e23b15bb8, not stripped
libjbigi-freebsd-steamroller_64.so: ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=b910a2085239daad85e0d9a70ff37f3550eb3cf8, not stripped
libjbigi-freebsd-steamroller.so:    ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=1aab7ba56d44c4c83d230ce34430bc5e23b15bb8, not stripped
libjbigi-freebsd-viac32.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=9596dfdf1027f082872f9c56ae397b1fb89a4064, not stripped
libjbigi-freebsd-viac3.so:          ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=9f5578d13b07a862f6864484056998ff63c255df, not stripped
libjbigi-linux-athlon64_64.so:      ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=c76d093e7e305255f04167b7d48a5b7efd90836c, not stripped
libjbigi-linux-athlon64.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=8469e81cbf3957589108c62f17814e92e8782d62, not stripped
libjbigi-linux-athlon.so:           ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=3dc397d89357daa787ae1e91a8b0c86dd4c777a8, not stripped
libjbigi-linux-atom_64.so:          ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=4b0c8c0e9998e9d3bd0df44cc188a2d7817cc4a5, not stripped
libjbigi-linux-atom.so:             ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=fd765a6259020d66b575ded1f4471bde0234422e, not stripped
libjbigi-linux-core2_64.so:         ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d249c08e0d2d05ca160e30b2a67a81896ef42c14, not stripped
libjbigi-linux-core2.so:            ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=ad825b7944241882bf6cf36f9697ffd1070f4e5f, not stripped
libjbigi-linux-corei_64.so:         ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=96ea1373012068691dfe327adad474bdd85c44b4, not stripped
libjbigi-linux-coreibwl_64.so:      ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=20da2a434e482a7ec0f8d9b879a90f5fe8af249f, not stripped
libjbigi-linux-coreibwl.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=fd915c2de688f3d5d3a8596a7c5c5af14e9f0731, not stripped
libjbigi-linux-coreihwl_64.so:      ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=20da2a434e482a7ec0f8d9b879a90f5fe8af249f, not stripped
libjbigi-linux-coreihwl.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=fd915c2de688f3d5d3a8596a7c5c5af14e9f0731, not stripped
libjbigi-linux-coreisbr_64.so:      ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=4f55b94abb5f89a665c9be022d3fc0d4ee404dbd, not stripped
libjbigi-linux-coreisbr.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=fd915c2de688f3d5d3a8596a7c5c5af14e9f0731, not stripped
libjbigi-linux-corei.so:            ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=8c8c3ee720c19931df954b8b22efa582b7c98af8, not stripped
libjbigi-linux-geode.so:            ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=9535252602ae358663cd5f5e26e496b7eefbed1f, not stripped
libjbigi-linux-k62.so:              ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=f3f20b2654d3579b2f5bc6cffa228fa116509e59, not stripped
libjbigi-linux-k6.so:               ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=aad2533f182894015a8bbf631663a105d80e4668, not stripped
libjbigi-linux-nano_64.so:          ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=3cc95866ae8dbf84aca7f5f2d15c2af6f9705454, not stripped
libjbigi-linux-nano.so:             ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=04283212efd51d9cc03572fbc05323d5f3e5793e, not stripped
libjbigi-linux-none_64.so:          ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=3d084ce06421f6000d36e06dddfbce049fdc93cc, not stripped
libjbigi-linux-none.so:             ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=1d63637530022549b05d95ca16d6a06aa5822d62, not stripped
libjbigi-linux-pentium2.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=9fe63a7583036fc377610f2ed7d7fe8328c0ada3, not stripped
libjbigi-linux-pentium3.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=9fe63a7583036fc377610f2ed7d7fe8328c0ada3, not stripped
libjbigi-linux-pentium4_64.so:      ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=6754bbcefa45be3de922cec0ad61a35de281b75e, not stripped
libjbigi-linux-pentium4.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=f78228433f2832e27d66bec3064ef7ce18cfe410, not stripped
libjbigi-linux-pentiummmx.so:       ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=6a455079779d780c4c9b25de5beb367f4527cf95, not stripped
libjbigi-linux-pentiumm.so:         ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=708402054b49fc6e0fc8344fcc5d77cee7b5c464, not stripped
libjbigi-linux-pentium.so:          ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=979170e2f777a623384fad078ab49261c6a47dee, not stripped
libjbigi-linux-ppc.so:              ELF 32-bit MSB  shared object, PowerPC or cisco 4500, version 1 (SYSV), dynamically linked, BuildID[sha1]=31dca2547d3399a88e4fa15f6383d9c4cddcf758, not stripped
libjbigi-linux-viac3.so:            ELF 32-bit LSB  shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=264ad55c29c7660e9fa147eed2300bbc968ca197, not stripped

There's no libs for newer AMD archs (bulldozer etc.) for linux and osx. Oversight, or was it decided that we don't need anything newer than athlon64 for those?

I see on ​http://zzz.i2p/topics/1960 that you are reporting and fixing isses also:

Issues:

Fixed: windows coreisbr 32 - missing
Unfixed: windows corei 32 - Failed to load the resource jbigi-windows-corei.dll - not a valid library for this platform

When you release a new zip for testing, _please_ put a version number in the name so we know what we're testing.

comment:20 Changed 4 years ago by tuna 

Only in old: libjbigi-osx-corei.jnilib

Was quite unintentional. Must've been removed from my checkout at some point. Added again.

Missing AMD binaries for linux is not intentional, but for OSX it is intentional, because hardware like that has never been shipped.

comment:21 Changed 4 years ago by tuna

New version can be found here ​http://treze2cubeytypnlfuwmvgmeygolpumgfse3ka76ndudvjs24loa.b32.i2p/new_jbigi_v2.zip

Please help me test the changes and the rest of the binaries.

Fixes:

  • Added Linux AMD builds
  • Replaced libjbigi-windows-corei.dll with new version. This one should work. But please test it.
  • Added jbigi-windows-coreisbr.dll
  • Stripped everything except a PPC binary.

comment:22 Changed 4 years ago by zzz

Oh, OK, no AMD macs, got it.

The PPC lib is existing, you didn't build it, OK.

Please review (in trunk) our existing logic on when to re-extract a lib from jbigi.jar, in net.i2p.router.tasks.InstallUpdate?, as this code hasn't been exercised in 4 years.

What's the postmortem on how you built and distributed a broken jbigi-windows-corei.dll for testing? Please give us some confidence in your build/test process.

comment:23 Changed 4 years ago by zzz

the new files in the jar all have the x bit set, these should be cleared

comment:24 Changed 4 years ago by zzz

Pushed some cleanup to the test output and added a -n option to test native only.
This speeds up testing of a bunch of binaries. Unzip jbigi.jar to a subdir, delete all the incompatible stuff, and do (e.g.) :

for i in x86/libjbigi-linux-*64*
do
	echo -------------------------------------------------
	echo $i
	echo -------------------------------------------------
	cp $i libjbigi.so
	java -cp i2p.jar:jbigi.jar -Djava.library.path=. net.i2p.util.NativeBigInteger -n
done

incompatible libs may not load, or may core dump. But this way you can test lots of the binaries. The -n makes the test much faster.

comment:25 Changed 4 years ago by zzz

Please fix, hide, or remove this strange error message if it can't find the CPU type:

ERROR [main        ] net.i2p.util.NativeBigInteger : Couldn't find the compatibility map for arch "null"

comment:26 Changed 4 years ago by zzz

possible fix for modinverse test coprime error pushed, but I still can't see how it could happen more often than 1 in 21061, unless RandomSource? was spitting out all zeros, so need to research that.

comment:27 Changed 4 years ago by dg

  • Cc tuna added
  • Milestone changed from undecided to 0.9.24

Too late for 0.9.23, likely to be 0.9.24.

<&zzz> hottuna, let's target GMP 6 for 0.9.24
<+hottuna> zzz: yeah.
<+hottuna> ẗhat makes sense to me
<+hottuna> ill put in some time tomorrow

comment:28 Changed 3 years ago by zzz

  • Milestone changed from 0.9.24 to 0.9.26

comment:29 Changed 3 years ago by zzz

  • Resolution set to fixed
  • Status changed from accepted to closed

Propped to trunk as 0.9.25-6

-6 notes: ​http://zzz.i2p/topics/2116

Original testing thread: ​http://zzz.i2p/topics/1960

Note: See TracTickets for help on using tickets.