Welcome and good morning
This is the reproducible builds team, talking about
"Stretching out towards trustworthy computing"
[Applause]
We're 4 on stage, but actually this is a team effort.
All these people listed here have contributed to the project at one point.
The 4 of us, that's
Lunar − me
there's Dhole,
Chris Lamb − lamby
and Holger.
But actually, this is DebConf and so a lot more of us have been or are
currently here and so, if you want to thank anybody that is working on this
you need to actually thank all of these folks
'cause, yay.
[Applause]
[Holger] The people in blue are here.
[Lunar] Let's get started.
Quick recap on what we're talking about.
We have software, it's made from source.
Source is readable by humans or at least a good amount of humans.
In this room it's good.
Binary, readable by computer and some tiny fraction of humanity.
Going from source to binary is called build, or like building or compiling
and we're doing free software and free software is awesome because
we can actually run these binaries like we want
We can actually study the software, how it's been made by studying the source
and by studying the source we can assess that it does what it's supposed to do
and not something else that does not
have malware, or trojans or security bugs
So we have the binary that can be used, fine.
We have the source that can be verified.
Problem is that right now, the only way we know that a binary that we get…
We have to trust a website or a Debian repository that says
"Well, this binary has been made with this source"
But there's no way we can actually prove that.
This is actually a problem that has been well explained by
Mike Perry and Seth Schoen at the 31c3 in Hamburg last december.
For example, Seth Schoen made a proof of concept exploit for the Linux kernel
that when GCC was called, the kernel would without modifying anything on the disk
when the kernel detects that GCC is going to read a C file, it will insert some
extra lines of code, and these lines of code can be a very bad thing
in the case of 31c3 talk I was just recalling.
Actually, you can even have developers who are in very good faith, who have
totally secure dev machines, or they thought they have,
who have reviewed all their source code for any bugs
and we would still get totally owned as soon as their computer gets compromised
or one of the build demons from Debian gets compromised for example.
This is not, like, hypothetical threats here we're discussing
A couple of months after Seth and Mike's talk at 31c3,
the Intercept revealed from the Snowden leaks
that at a CIA conference in 2012, one of the talks that happened
was about a project called Strawhorse.
Strawhorse is about modifying Apple XCode, which is the development environment
for MacOS X and iOS applications
and well, they were modifying XCode so it would produce,
without the developer knowing,
binaries with trojans, malware, watermarked binaries, lots of bad things.
So, solution:
enable anyone to reproduce identical binary packages from a given source.
Because if using a source, using the same environment,
multiple people on different computers, on different networks, at different times,
can all get the same thing from the same source
all the same binary, byte for byte,
then there's a good chance that…
Well, everybody could be owned, but let's be more joyful and say that
probably, if everybody gets the same result, there was actually no problem
and everybody is safe.
We call that solution "reproducible builds"
Yay.
[Applause]
Actually, it's not only about security.
For Debian, we have, if you're doing "Multi-arch: same" packages,
well they only have the same bytes if they are built for different architectures,
the files in the package.
Debug packages, you can create at a later time, if you forgot to have debug packages
in the first place,
you can pass the no-strip option later and because the package is reproducible,
you will get the debug symbols that work for software that has been shipped already
We do early detection of FTBFS that way
because if we try pretty quickly to reproduce a build,
then it has to work.
It's useful for build profiles.
We can get smaller .deb deltas,
because from one version to the next we might have the same content.
We can do validation of cross-builds,
Helmut Grohne can talk to you about that.
And also, Niels Thykier told me that
he was very interested in reproducible builds because it would enable him to
test debhelper better, because
if the package builds reproducibly, then he makes a change to debhelper,
then he can rebuild
the same version of a package with a newer debhelper and see what has changed
and this change can be isolated to only what he has worked on debhelper
for example.
And, oh my.
The whole world is watching us.
Since two years or a year and a half ago, everybody I meet in security conference,
in hacker conference, in free software conference is like
"Oh you're working on that, that's awesome."
And, I mean, I've been the one doing quite a lot of talks, and everybody comes to me
and I'm like "Wow wow, this is way bigger",
but we're actually leading the field here.
Yay Debian.
[Applause]
[Holger] So, we are not the only ones leading the field,
Bitcoin and Tor made their software reproducible before us,
Coreboot also succeeded, if you build Coreboot without any payload,
that's 100% reproducible.
FreeBSD has a page on their wiki since 2013
saying there are 5 reproducibility issues in their base system.
We're at the moment trying to confirm this.
On jenkins.debian.net, I've also set up now tests for FreeBSD, NetBSD,
Coreboot and OpenWrt.
So if you go to reproducible.debian.net/
you get that tested.
And there's more in the pipeline.
There are other projects interested as well.
NetBSD also has a variable MKREPRO which you can set
and that builds reproducibly.
Though they think "I'm keeping some timestamps it's fine" and then
filtering them out later".
We disagree.
So this is how Debian looks like, Debian Sid,
but this is a lie.
This is not the truth.
This is just our test setup.
Sid is not like this.
For Sid, it's all orange, there's zero reprodicibility in Sid today.
But we'll talk now and in the following round table,
it's to actually make Sid reproducible.
The current status is
we're working on this in Debian since two years ago.
We have weekly reports about our project now since May
and we've given several talks, especially in the last year
and all these talks, presentation, also other stuff is linked in the wiki.
There's a page with information about Debian, these BSDs,
other Linuxes, upstream softwares all on this wiki.
Since DebConf14, which is merely a year ago,
we've made quite some changes.
We have introduced strip-nondeterminism
which is called by dh at the end of the build of the package
and will normalize some things which Chris will explain later
We have decided on a fixed build path
because the build path is leaked in the binaries and several things
We didn't find a way yet to make the build path arbitrary.
We designed a way to record the build environment
because to rebuild, you need to recreate the build environment.
We set up this Jenkins setup.
We wrote diffoscope which used to be called debbindiff
which shows differences between two packages or two directories or
two filesystems by now.
There's SOURCE_DATE_EPOCH, which is a way that the tools expose
the last modification of the source.
Because the build date, people want to include the build date
because they think this is a meaningful indication:
when a build was done, which software used.
But if the build always recreates the same results
the build date becomes meaningless
and the really interesting thing is the latest modification of the source.
We have written patches for the tools
[Lunar] strip-nondeterminism: is Andrew Ayer in the audience?
Yay! He did it!
It's written in Perl because we didn't want to have a new build dependency
in all Debian packages.
Basically it takes anything and tries to normalize it as much as it can
replacing timestamps or file permissions or removing some issues.
It's working very well on many formats, it's meant to be extensible
so we can actually add more things and it's run by dh at the end of the process, as Holger said.
The .buildinfo is currently a proposal we have not yet totally agreed
but we are generating them as part of the test we have
and basically it's a new control file that will tie the sources, the generated binary
the packages that were used to build this binary and their version.
The idea is that we can use this file to reinstall all the specific versions from snapshot
So we recreate the same build environment then we can just start the build from that source
that was mentioned and see if the binary that has been generated matches.
What it looks like for now, you see there is a source binary, the build path
because currently we don't have any good post-processing tool for buildpaths
in ELF and DWARF binaries, we just decided to specify the build path so when we do
a later rebuild we use that path and be safe.
The source is .dsc, the binary is .deb and a list of packages with the versions.
We currently use the base files version to know which Debian release is to be used
as the basis.
[Holger] The general procedure for testing is: we build the source, we save the results,
we modify the environment and we build it again and compare the results.
That started as a shell script last year which I put on jenkins and then it exploded a bit
and now we have 67 jenkins jobs running on 7 hosts.
Since last week we have 4 armhf small boards where we will be able to test armhf,
but very slowly.
We have two new amd64 build nodes.
The code is now split into Python and bash scripts.
For all the other distro testing there's a lot of bash code now which is mostly
boilerplate and it's 5 lines or something to build FreeBSD and 5 lines to build NetBSD
but there's 100 lines boilercode around so it's really not that much code.
We do test testing, unstable and experimental.
For arm we only start with unstable.
We do like hardware so if you have hardware to donate to us, that would be great,
we need ssh and then root basically.
We are testing Coreboot, OpenWrt and the BSD's, soon I will also set up a Fedora test
I don't want to test all the 20,000 Fedora packages but just 200 or something:
the base system of Fedora to examine how .rpm works
to get really the whole Free Software world reproducible.
This is all run on ProfitBricks hardware since 2002, so thanks to ProfitBricks.
This is the variations we do for Debian.
It's the hostname, username, timezone, locale.
Chris will explain what modifications this causes, variances...
We are not testing at the moment differences in date so the date is always the same
the time is a bit different.
[Lunar] Well almost! Because we cheat with the timezone, we use one timezone that is
GMT-14 and then GMT+12 so it's more than 24 hours appart.
[Holger] On the first of the month we sometimes find new bugs where there's
packages which record the month.
We don't have variations of the CPU type at the moment.
Both time and CPU type variations, we'll have them about one or two weeks
the nodes are being prepared at the moment.
Then we will test all the meaningful variations we could think of.
There will be probably some packages which build different according to the number of
number of CD drives attached or whatever things, but those will be find by you.
[Lunar] We are doing all these tests because we want when you rebuild a package on
your machine that if any this is different from the build deamons in Debian you get
the same results.
We use this to detect this problems early before you actually a false positive that we have
to investigate when someone rebuilds a package on their machine.
To understand the difference that we found from one build to the other.
It started also as a 10 lines shellscript and then it felt okayish
and so Python!
And now it's a lot of code and it actually grew way beyond a Debian package.
We changed the name, it was called debbindiff but it's absolutely not tied to Debian anymore.
It's called diffoscope, thanks to Jocelyn for the name.
Basically what it does: it tries to get to the bottom of what is different between
two archives or directories.
Because it's not useful to compare bytes that are compressed by gzip or xz, that will not
lead you to understand what is different you need to uncompress and look at
uncompressed data, and if the thing actually compressed is a tarball, you might actually
want to compare the files inside the tarball.
If there is a PDF inside this archive, you don't want to compare the bytes of the PDF
you want to compare the text of the PDF.
So this is basically what diffoscope does, it tries to transform anything that is
a container and compare things in this container and if they can be transformed into
a human readable form it will try to do that, and compare these human readable form.
And if it doesn't find any difference but there are still differences from the bin
it will fall back to binary comparison.
Try it, extend it; it's Python, it's modular, it's great.
It already supports squashfs, ISO, .rpm, gettext .mo files files and so many different things.
You can have HTML output like that, so this is what is displayed on many
examples we've shown so far, and also to make it easier for copy paste
and post processing we have the text output.
You can also use it to review packages before uploading them to Debian.
It does fuzzy matching, so even if the directory is different in the archive it will
find it like git does.
It has grown way more beyond just build reproducibly. A useful tool.
[Dhole] In order to solve timestamp issues, we are proposing the SOURCE_DATE_EPOCH variable.
This is because most of the times having the build date embedded in a package
is not useful for the user, because you could take a really old package and build it today
and that day would not be useful.
We are standardizing a replacement for build dates so that tools can use it.
When this value is set, the tool instead of embedding the current date, it will embed
the date taken from SOURCE_DATE_EPOCH which will contain a Unix epoch timestamp.
This is a general solution we are trying to standardize so that not only Debian uses it,
but other Free Software projects and distributions and in the case of Debian,
we set this variable to the latest Debian changelog entry timestamp.
We have already been sending patches to different packages, mostly it's documentation
generation. So here's a list of bugs that we have opened which have been closed
and merged; so it's help2man, Epydoc, Ghostscript, texi2html and Sphinx.
We are both sending these patches to Debian and upstream so all the distributions can
use them, and we have also been sending patches to other packages which are still
open, so we encourage you to take a look at these packages if you are the maintainer
and merge the patch.
[Lunar] Thanks to Daniel Kahn Gillmor and Ximin Luo for pushing this proposal forward.
And also lots of these patches have been written by akira and Dhole as part of their
Google Summer of Code, and you work really great.
[Applause]
[Dhole] The GCC patch is: GCC uses two macros which are _DATE_ and _TIME_
which embed the timestamp and I wrote a patch so that if SOURCE_DATE_EPOCH is set
instead of adding the current time, it takes the time from that variable.
I sent this patch to GCC, it's still there forgotten with many other patches
but hopefully at some point they will realize that this is interesting and they
will merge it.
[Lamby] Hey. Let's very quickly run you through some really simple ways
to fixing packages. The details don't necessarily matter, it's just to give you
of what needs to be changed and basically to point out that it's not rocket science.
So you can just come in and jump in.
For example gzip, it's a very old tool and they decided to add timestamps when
you generate it, but it's an easy fix, you just add -n flag.
Some other things easy to change: some Python stuff had tag_date=True, which
I don't know if you can see it but adds a timestamp to eggs. You just change it to
False to get rid of it.
Static libraries, they are just ar archives so the same format as .deb, and you
can just use binutils or strip-nondeterminism tool.
PNG has timestamps for some reason, you can get rid of them, that's ImageMagick and it's
a bit ugly, but also strip-nondeterminism gets rid of it.
Tarballs are quite interesting, they will by default capture user and group
you just pass --owner=root bla bla bla...
Ordering, this is interesting as well, it will usually use file system ordering
which is completely non-deterministic. So you need to sort with LC_ALL=C.
[Lunar] Think about the locale! Because sorting order varies from local to the next.
[Lamby] They also take timestamps, again you can set --mtime or you can mock around
with find/xargs/touch bla bla...
Lots of other files have timestamps: Erlang files for no reason, even upstream don't
know why they added a timestamp.
We have now a patch for SOURCE_DATE_EPOCH, which I think landed a couple days ago.
Here's an interesting one, not necessarily the current build timestamp, so this is a
timezone dependent date which Ruby loads and then saves incorrectly as your local time.
This gets mangled, so that's patching.
I'm going from changing individual packages to more toolchain things as you can see.
Upstream configure scripts, you can maybe see the top that it just uses hostname
for no reason. Sometimes you can override it in debian/rules just by exporting something
or passing a variable to dh_autobuild or whatever. That's just a little bit more
involved, you have to look at it more carefully.
Perl hash order, lot of Perl uses data Data::Dumper to just output a bunch of stuff which
is just not deterministic. So often just setting Sortkeys, but sometimes it's
a completely different solution.
Header files, so you can maybe see that they are using the timestamp essentially
as a unique identifier, you probably have to start re-writing these something saner
because this is a wrong use of timestamp anyway.
More Makefiles, the deeper they timestamp in the upstream package the more you have
to start patching, so these kind of start sucking a little.
We've made a lot of toolchain changes, some already mentioned, some of them already
merged, see more in this link. Again, details don't matter, just check it out
it isn't crazy, it's just working out what's different.
In terms of the work done we've sent these many patches: two patches a day,
which is not too bad, on average.
[Applause]
[Holger] I can't clap because I sent three or something like that
[Lamby] Holger does three per day.
And this doesn't count other bugs we found in the process of building packages, like
fail to build.
This is blue the ones that are open and orange are done.
You can see that someone went a bit crazy in February filing bugs and eventually they
were being fixed; slowly.
[Holger] And actually we filed more bugs because the fail to build from source bugs
are excluded, I think we filed 300 FTBFS in the last two or three months.
[Lamby] And those include fail to build because of reproducibility things as well
but we haven't split them up.
[Lunar] What's left to be done because Holger said "the graph is a lie".
The main thing that is blocking a lot of work is dpkg. Right now the output of dpkg
will be not deterministic 100% of the time, because of timestamps and at least the
file ordering. We also have a patch that creates these .buildinfo files that we've
shown that works. It's not submitted yet to dpkg because we need to agree on the
format. At least we have ftpmaster or maybe dpkg, well we have a lot of people
and that's what we are going to do the next hour.
debhelper also has a few changes; the make mtimes, debhelper might also not be
best place, maybe we want that in dpkg.
I've been trying to put patches in tar so we can make it easier. It's complicated to
see where's the best place but so far we've been doing our tests with this frame and it works.
[Holger] In our repository we have these packages with these bugs fixed so when
you want to test reproducibility issues on your own machine you need to use the
repository which has these patches applied at the moment.
In pure sid you cannot create reproducible packages.
[Lunar] I heard that the SOURCE_DATE_EPOCH patch is in git already, so it's going to happen.
cdbs also needed to export SOURCE_DATE_EPOCH and we are starting to do more infrastructure
work: josch mainly and akira on sbuild, because we wanted to have this
srebuild script, where you give it a buildinfo and it will do the rebuild and
it needs changes in build daemon for the build path and also a couple of changes in
sbuild itself.
[Holger] And the script is not ready yet, this "Finish" means it uses our repository
at the moment, we need to change it to only use Sid and snapshot.
[Lunar] So there is the buildd issue that we need to discuss
and we also need to see how we could include or not, or somewhere give this buildinfo
control file to the world so they can rebuild the packages, so it's not yet
clear where's the best place to store them.
Because adding 22,000 files, some people get cranky of this idea.
[Holger] It's more than 22,000 files, it's 22,000 source packages multiplied by
10 architectures; but there's a lot of arch builds so that's probably 100,000
.buildinfo files, multiplied by Stretch and Sid, so it's 200,000 files or more on
the file servers and on the mirrors we would like to have it.
That's the same amount of files which are currently there. The mirror operators are
currently not happy, they will not take it, so our current idea is just concatenate
all these files into one file that's 140 MB uncompressed, 40 MB compressed.
That's easier to handle.
And then probably have a service buildinfo.debian.org where you can
download individual buildinfo files if you need them.
[Lunar] And so when we will be done with all that we can maybe add a final patch
it would be to Debian policy, mandating Debian packages be reproducible.
[Applause]
I can say again that the dream of mine is that we would stop uploading .deb when
we upload a package, but instead just upload the hash of the binary, have the buildd
build again this package and only if these two match they can enter the archive.
So we are sure that at least the two machines, the developer machine and the
build deamon agree that they've built the same thing.
[Applause]
[Holger] I share this dream but I think having this in policy is a mass requirement
sadly something only for Stretch + 1, but I'm curious if we had fixed dpkg and
debhelper now, would you think we should upgrade all these wishlist bugs to important now?
[Audience] Yes!
[Holger] We'll talk about this later soon.
[Lunar] But before that we actually have work to do.
[Dhole] In order to fix your package, the first thing you can do is go to
reproducible.debian.net/package, and you can the web interface where you can see
notes on the package, we have tags to identify different issues that make packages
not reproducible, with links to the wiki about how to solve them.
[Holger] When you see this, you want to click on this debbindiff link.
It's still called debbindiff not diffoscope, this will show all the differences,
if there is a note. If the package is unreproducible and there's no note
it will automatically display the debbindiff, and if your package is fine
there's here a sun.
[Dhole] You can also see an entry in the tracker, stating if your package is
reproducible or not.
You can also find information in DDPO and DMD. You can find tips on the wiki it's
ReproducibleBuilds wiki, we are working on a Howto to have detailed steps on different
issues and how to solve them. Lunar gave a talk at CCCamp where there's many issues
really well explained and the solutions for them.
You can also come to our irc channel which is #debian-reproducible and ask for help
or go to the mailing-list.
In order to test locally if your package is reproducible right now we are using a
script that uses pbuilder in a custom configuration, you need to set up our
reproducible repository. In the Howto in the wiki there's the steps on how to set up
the chroot and everything, it's documented in the wiki.
Diffoscope is in unstable and today it's going in Stretch.
We plan to add these scripts to rebuild packages in different settings in debscripts
once dpkg is good, and we welcome you tomorrow to the hacking session from
2 to 7 in Stockholm room.
[Lunar] That's for fixing your packages, please do that. If you want to have even
more fun, then test your own package, join us!
This is the past year of my life, it has been awesome because the team has been
so great, it's been friendly atmosphere, lots of new understanding so many things you didn't
want to learn about that you had to learn about, and basically it feels very good to
be part of this actual changing the world thing. It's just software but it has some
profound effect. I've been told that the work we've been doing is being tossed
around in Cisco and Google and Facebook; all these big dot com companies bla bla,
they actually want to do that as well even though they are not doing Free Software,
which I find weird, but whatever.
So what do we do? We review packages, we have these notes when we actually try to
identify, so when the maintainer comes they don't have to think to much about
the problem and just fix it. We try to identify common trends so when many
packages have the same problem we make an entry and explain and maybe think about fixes
that could apply to the whole archive.
We work on this reproducible.debian.net jenkins setup, the scripts.
We hack on the diffoscope tool, we make strip-nondeterminism better, we propose
changes for the toolchains when there are needs, some need a lot of patches,
most of the bugs we have reported on individual packages have patches.
[Holger] Bugs have patches [Lunar] Yes!
And also we are actually writing some more general documentation from the
understanding of these things we have been having, we are preparing a reproducible
builds Howto to explain to the Free Software world how they can do it so it's about some
of what Chris explained but also more general consideration on what if you're
not Debian and you want your thing reproducible when you distribute as an
independent vendor. So we want to work on reference documentation so the whole world
can actually do that.
We do a lot of talks as you've seen and it's been fun, and with all these
presentations we've made so far it's all in git. And everybody is free to take one
of these slide decks and run with it somewhere, translate it...
Questions?
We have to run with the microphone, because there's no mic anymore.
[Question] I just wanted to make two quick comments: so first of all diffoscope is
really awesome, not only for reproducibility but also for example if you change your
debian/rules in some way and want to see if the package is the same afterwards because
you just cleaned up a bit, that's really awesome for that, so thank you.
And also I think the work you're doing now is something that in 20 years time we're
going to look back towards it and think, well, of course builds should be
reproducible, so thank you very much for your work!
[Applause]
[Question] When reproducibility becomes part of the Debian policy, will there be a
lintian --reproducible?
[Holger] I don't think lintian can detect that because lintian works on the source
package and you need to build the package for this.
[Lamby] Things that could be detected by lintian from a static analysis point of view,
yeah I'm sure, like looking for gzip without -n for example, but that wouldn't
be conclusive from lintian point of view.
[Lunar] One thing that I really wanted to diffoscope at some point - the code is made
the way that it's possible - it's to have hints so when it actually looks up
differences between two packages then you can have an idea, suggest you: hey you need
to remove that timestamps, or you should sort these keys. It's not done yet, but if
anybody wants to do patches it's totally doable.
[Question] Thank you for the work, have you thought about reproducible images?
[Holger] It's on the todo list.
Before images we need reproducible package installation, and then we need reproducible
images like squashfs has some things which are not reproducible, but the package
installation is not reproducible at the moment because apt installs packages in
arbitrary order and then the post-inst create for example users which get
user-ids in the order the packages are installed, so for that to fix either apt
needs a way to install in a deterministic order, but it's on the todo list file.
[Lunar] Pabs started a wiki page a couple of months ago that is called reproducible
install. This is very important if we want tools like Tails to actually be reproducible
so some people will work on that, we do want to work on that.
[Lamby] It's quite a deep problem for example d-i will install different stuff
depending on your hardware, so that's immediately not reproducible.
It'd be great.
[Question] I've been working on a couple of my packages to get them reproducible
build, but I was often wondering if I should fix it in my package or actually
that it should be fixed in higher up and I guess I've been adding some fixes to my
packages which may in the future even not be needed anymore and then it's just
unnecessary code as well.
So how do you see where things should be fixed and how should we as package
maintainers go about with this?
[Holger] There's many things which there's the easy fix to whatever: set the timezone in
debhelper or better in dpkg to UTC, but that will not fix the upstream bugs, so
actually it's better not to fix, set the timezone or other things deterministically
in these tools but rather have them fixed upstream, that's what we want.
Some things we will need to fix them in dpkg to get a meaningful result but
basically we want rather these distributions with just build from source which don't have
debian/rules and they just build with upstream Makefiles, we want the fixes
to land there.
[Lunar] We've been experimenting for two and this is a lot of trials and errors,
trying something, see how it fails, or maybe we can do better than that and
changing. And I know this can be frustrating at some point because you do changes
and they all become unneeded, but in the end this is how we make stuff that matters.
And we move forward, it's not because we're trying to make the big picture at once,
and I know in Debian we sometimes try to do that, so we experiment and learn from it.
[Question] An example that I'm now looking into is actually the documentation is built
for this package by looking in all the files and generating but, for instances the
index file is sorted, but I guess upstream would say: well, if you set some ordering
in your LC parameters you want this page to be order as you want, instead of forcing
it in the sort, so I'm really wondering: should I now upstream this or should
I just fix it in my rules because that's the logical place?
[Lunar] Both. No, there's no good answer, I'm quite a strong proponent on the idea
that if you use a computer you should be able to talk and have the computer talk to
you in the language that you choose, so if people want to have GCC error messages
in German, they should have it.
But local sorting, this is the kind of LC_ALL that can be very local and that
you can do for just one tool, it's fine to do that.
[Question] Do you have ideas on making sources reproducible? Like upstreams
calling make dist, or this infamous autogen.sh files?
[Lunar] I don't think that anybody in the team has looked into that yet, source
files are easy to analyze way more than binary packages so, it would still be great
to have easier ways; you have source tarballs be byte for byte identical,
but it's not as an issue as it is for binaries. If people want to look in that
they should.
[Question] Do you know a way to make git archive build something reproducible?
[Lunar] Well pristine-tar
[Question] Yes, but without it.
[Holger] There's one tool. You want to use a new one? Then write it.
Why not use that tool which does the job?
pristine-tar does it.
[Lunar] This is for source and so that's another issue that what we are actually
currently working on.
[Holger] You're welcome to join the team and extend our scope to sources.
[Lunar] How many questions, two?
Two more questions, two or three.
[Question] So if there is a couple of other environment variables that could be set
in the environment to increase reproducibility, where to put them?
In the rules file? Or in the generic build environment of all packages, or where
should these things be placed?
[Lamby] It'd be nice if upstream fixed it, so if we just change it in debian/rules
that's just only helping us, so often take it upstream, would be the ideal solution.
Are you referring to something else?
[Question] For example many hashmaps have randomized data in the hash function, so if
you have some code that relies on hash order, at least some implementations of
hash functions are leaving them be seeded rather than using something random for
a build thing, but you want the randomness in your hash functions for normal users
because else your hashmaps get open to attacks.
[Lamby] Correct, yes.
[Lunar] In these cases we send patches adding sort everywhere for the keys and
it's solved. For very few cases, for Perl for example you can set and environment
variable and some maintainers prefer to do that. But usually we try to push these
changes upstream, because they are simple enough and they like it.
Actually it makes testing easier to them.
There was one in the back, there.
[Lunar] That's the last question
[Question] Follow up question to what we had here before.
You showed an open bug report against GCC to support SOURCE_DATE_EPOCH to cover
the mdate and mtime timestamps, so I have patches to patch them out in my packages.
Should I remove those patches and if so, when?
[Lunar] Have you seen any more emails from the GCC maintainers?
[Dhole] The mail is forgotten, I guess we should ping it again, and see if they
reply, because what I read from the GCC website is that only the replies from
maintainers are the ones that matter, and I think no maintainer replied to the
message, so we should ping again.
[Question] That was just an example, my question was more general.
At which time should I remove my patches to fix things which were fixed higher up
in the toolchain? Or should I just leave them in there?
[Holger] Once they are in Sid.
[Question] Ok thanks!
[Lunar] Ok, I guess we're out of time.
Thank you for listening.
[Applause]
[Lunar] Fix your packages!