dSYM status report and proposal

[Zachary West]

On Fri, May 29, 2009 at 01:37, Peter Hosey <boredzo at adium.im> wrote:
> Today, I worked on making our build process produce and archive dSYM
> bundles.
>
> For Adium, I got it working, but it had a side effect: Because one of the
> required changes is a switch from copy-phase strip to separate strip, we no
> longer strip Libpurple and related frameworks. That makes the resulting
> Adium actually *bigger* than it would be without the dSYM changes.
>
> Ultimately, putting compiled binaries into the hg repository is not tenable.
> hg fares pretty poorly with binaries, since it can't diff them. Sooner or
> later, we need to get off of that.
>
> The direct solution would be to build Libpurple et al from source. That
> sounds like it would make our build times much, much longer. Moreover, we'd
> have to automate cloning the mtn repository the first time, decide whether
> or not to pull to it every build, require mtn to be installed to build
> Adium—that's all a big ball of suck.
>
> I propose a different solution.
>
> Instead of putting the Libpurple builds into the Adium repo, as we do now,
> let's store them in a directory on the website. We can have our build
> process download the correct build for a given build number, and check the
> download using file size. The build number and file size information would
> be in a small text file, which we would put into the Adium repo in place of
> the build we keep there now.
>
> So:
>
> -Frameworks/libpurple.framework
> +Frameworks/libpurple.buildinfo
> +Utilities/download-from-buildinfo.{py,rb,pl,whatever}
>
> The .buildinfo file contains a URL format, build number, file size in bytes,
> and possibly an MD5 or CRC32 digest.
>
> download-from-buildinfo reads the .buildinfo file, then checks the size and
> hash of the existing file (if any). If these do not match, it needs to
> download, so it interpolates the URL with the build number, then downloads
> that; finally, it checks the size and hash of the newly-downloaded file.
>
> We change the build process to run download-from-build-info before doing
> anything else. Since download-from-build-info checks the size and hash of
> the file, most of the time, it won't download anything.
>
> In addition to the benefit to our repo size, this new process would be
> capable of building and archiving dSYM bundles for the Libpurple frameworks.
> We'd store these with similar URLs to the actual builds.
>
> The next step would be to get these dSYM bundles into our hands. I can
> already hear in my head Zac reminding me how hard it would be to check out a
> tag, look at the build number in the buildinfo file, then download the dSYM
> bundles with that number. I agree—that'd be a hassle.
>
> So what I propose instead is that I extend the build process I already have
> to download the Libpurple dSYM bundles automatically, and add them to the
> same archive that already receives the Adium dSYM bundles. You won't have to
> download anything extra; you'll get all the Adium *and* Libpurple dSYM
> bundles in one archive.
>
> Yes, it's a bit ambitious, but it's doable, and it doesn't have the
> complexities of making our build process dependent on building source every
> time from an mtn repository.
>
>
>

This would be a nice move, and I've been trying to think of ways to
shift the binaries out of the repository. Moving to xibs would also
help on this front.

In either case, the binaries *do* need to be version controlled. It
just makes life easier in case we need to regress without rebuilding,
to test, etc. Also to reduce the complexity of uploading.

We definitely need something to *automate* this process. If any works
is involved in upping these builds (such as manually creating this
buildinfo file) I'll go insane, because libpurple gets updated
decently a lot. Hence why a repository is probably best - it doesn't
require storing SSH credentials in the www's user for people to
upload.

The new repo could hold the new information automatically - much like
the nightly file is generated. Anything to reduce the complexity of
updating a build, because right now it's `hg ci` not $EDITOR
blah.buildinfo ... *looks up a bunch of stuff* *opens Cyberduck and
copies* *tests the python* *builds with the new thing to make sure it
uploaded okay* etc etc. /lazy

I don't care all that much about that particular repo's size, so hg is
probably fine. It can be a website such as binaries.adium.im, and its
contents will just be the raw files.

It doesn't seem all that costly to include the binaries in hg though,
but libpurple updates are a decent chunk of change (maybe 1-2MB each).

Zac