Opened 9 months ago

Last modified 6 weeks ago

#2279 open defect

Debian: Reproducible build

Reported by: zzz Owned by: zzz
Priority: minor Milestone: 0.9.40
Component: package/debian Version: 0.9.35
Keywords: Cc: mhatta
Parent Tickets:

Subtickets (add)

Change History (11)

comment:1 Changed 9 months ago by zzz

Create a build option, remove timestamps from manifests and javadocs
in 49bd4534198c73084f87be7789fccd5c900c2736 to be 0.9.35-2
only the beginning

comment:3 Changed 8 months ago by zzz

Interesting. You were putting strip-nondeterminism into the build.xml files and adding new ant targets.

I was hoping that dh_strip_nondeterminism would just magically get run from rules automatically, but not sure if it is or not. Couldn't find much documentation on how it works. I thought it just "fixed" the jars? Do we really need to explicitly run it standalone (as strip-nondeterminism) in each subproject?

comment:4 Changed 8 months ago by nextloop

There is #reproducible-builds on OFTC IRC network. Those guys are helpful to some extent.

My intention was to get the izPack installer bundle reproducible, but that's where I failed. I could not really get behind that weird packaging format. I had to make all jars reproducible before I throw it to the izPack installer creation tool. I think I had everything up to that step reproducible (at least it did not depend on timestampts anymore).

Never really researched how to get Debian packages reproducible.

What's the submission deadline for the next Debian release? I would like to continue on that, but I can not promise to much.

comment:5 Changed 8 months ago by zzz

Thanks for the clarification.

My conclusion, after researching reproducible builds and the tools available, is that it only makes sense for us to do it in the context of a debian package build. That's where the tools are known to be available. I don't have any appetite for making our izpack installer reproducible.

As to scheduling: I do the launchpad builds a day after each release (next one is .36 in late August) and copy them to our repo deb.i2p2.de. Mhatta is the maintainer for the official Debian sid packages, which get pulled into Ubuntu. He has not yet built .35 and he has not responded to my queries on when he might do so. I'm sure he would be quite helpful to the effort on this ticket if only he had time. In his absence, any contributions you could make would be great.

comment:6 Changed 6 months ago by mhatta

https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/i2p.html

Debian-wise, the only problem seems to be non-deterministic ordering of web.xml (e.g. i2p.​susi.​dns.​jsp.​index_jsp is listed sometimes before, sometimes after i2p.​susi.​dns.​jsp.​subscriptions_jsp).

I'm not sure how it should be fixed in Ant (Gradle has reproducibleFileOrder support).

comment:7 Changed 6 months ago by zzz

  • Milestone changed from 0.9.37 to 0.9.38

ok. The jsp compilation always seems to be in a random order - but the new parallel jsp build in Tomcat (that caused the problem in #2307) may make it even worse - or may be the sole cause of the diff.

The other differences in your link are in individual class files of compiled jsps - not sure if it's the same cause or not.

Will investigate further.

comment:8 Changed 6 months ago by zzz

  • Status changed from new to open

Enhanced the JspC method from #2307 to sort the files, and force single thread. In a1c390d91d9fa73ce206da3b719c26223e334178 0.9.36-7.

Doubt this was the only problem, despite the hopeful comment 6 above. Leaving open to see what diffoscope reports after the .37 release. Not sure why paths to jars are ending up as strings in the jsp class files.

comment:9 Changed 6 months ago by zzz

ok the _jspx_dependants.put(path, long) calls are being added by the taglib processing for susidns. I don't see them in my non-deb builds so it's something the deb version of the tags building are putting in?

According to:
https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/issues/232
quote:
the jspx_dependents are generated to support on-the-fly recompilation. As that won't happen if the servlet is precompiled, it can safely be ignored.

Not sure how to fix. We could edit the generated source in-between the JspC and the javac steps in ant, to rip out the put() lines. Maybe. Need to see why it's not happening locally though. Maybe there's some flag or property we can set to make it stop.

comment:10 Changed 7 weeks ago by zzz

  • Milestone changed from 0.9.38 to 0.9.39

The change to Tomcat 9 (sid/buster/disco) #2364 broke the version detection required for reproducible builds; fixed in 79e718388ed6c775c47029be39b87f0ec66b4f47 to be 0.9.38-4

comment:11 Changed 6 weeks ago by zzz

  • Milestone changed from 0.9.39 to 0.9.40

Milestone changed to .40 as requested by mhatta.

You could, if you wanted, try to take the code changes referenced in comment 4 and stick them in as a patch for a new .38 build, to see how the reproducible build tester bot likes it. But probably not worth the effort, just wait for .39 and see how it goes.

I don't think anybody is yelling at us yet about failing the reproducible build tests, so this is still low-priority in my mind. Unless and until somebody starts pointing fingers at us.

Note: See TracTickets for help on using tickets.