Building OpenMoko 2007.1 from scratch

From Openmoko

Revision as of 01:34, 10 February 2007 by HaraldWelte (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

'Building OpenMoko from scratch Walk-Through

according to Werner Almesberger <werner@openmoko.org>.


This is a guide describing how I set up a running OpenMoko system from scratch.

Do not treat this like a shell script! There are various configuration items you need to set (or skip, as it may be), operations that depend on how your host(s) are set up, and also on the revision of the target platform.

Instead, look at each step, read the instructions, and copy and paste what makes sense for you, and adapt what you disagree with. Links to original and background material in the Wiki are included wherever useful.

Please note that this file isn't actively maintained and is guaranteed to suffer bit rot as I begin to use more and more shortcuts in the build process. Furthermore, naming and other preferences are my own, may have little to do with what is considered common or even "official" style. If you find anything particularly stupid, please let me know

Contents

Roles

The build process may spread over multiple machines. They have the following roles:

  • BUILD - build host, with quick access to the files and CPU power. Must have Internet access.
  • LAB - lab machine connected to the debug board (serial and JTAG) and to USB on the Neo (since this will probably be just a single machine, the roles are not further divided)
  • CARD - machine with a USB-attached SD/MMC card reader

All machines are assumed to share the same filesystem layout. In the title of each of the section below, the respective role is indicated. "(all)" is for settings that apply to all machines, or that - for simplicity - can be applied to all of them.

Directory layout (all)

$OMDIR (/home/moko)	base directory for the whole tree
  openmoko/		files from OpenMoko subversion (SVN) repository
  openembedded/	files from OpenEmbedded (OE) Monotone repository
  sources/		cached downloads of OE
  build/		OE build directory

Environment variables (all)

For simplicity, we just set these environment variables on all hosts involved. If you're not comfortable with this, feel free to weed out the ones you don't need.

Our base directory (configure this for local arrangements):

export OMDIR=/home/moko

The path to the sources/ directory on the buildhost, in SCP path name syntax (replace "werner" with your user name):

export BUILDHOST_SRC=werner@buildhost.openmoko.org:/space/fic/openmoko/sources/

The search path for BitBake files. Note that the order is of vital importance.

export BBPATH=$OMDIR/build:$OMDIR/openmoko/trunk/oe:$OMDIR/openembedded

Permissions (BUILD)

In order to perform the build process, you have to obtain the following permissions:

  • write access to the OpenMoko SVN repository (in principle, it should be possible to simplify this to read access. For further study.)

Prerequisites (BUILD)

There must be at least 7 GB of free space on $OMDIR.

In addition to the traditional development tools (gcc, patch, etc.), the following packages must be installed on the build host:

#   subversion	version control system used by OpenMoko and others
#   quilt	patch management system used by the Linux kernel and others
#   monotone	version control system used by OpenEmbedded. Needs a recent
#		version, e.g., 0.32, although also 0.31 should work.
#   diffstat	the OE build process wants this
#   texi2html	this too
#   git		version control system used by the Linux kernel and others.
#		Do not confuse this with the "GNU Interactive Tools"

Furthermore, the following package can be installed optionally:

#   psycho	Python just-in-time compiler. Speeds up BitBake considerably.
#		Strongly recommended.

Gentoo users can obtain all this with (note that, at the time of writing, Monotone 0.32 isn't available without setting the ~x86 keyword):

echo 'dev-util/monotone ~'`readlink /etc/make.profile | awk -F / '{print $6}'`\
  >>/etc/portage/package.keywords
emerge -u subversion quilt monotone diffstat texi2html dev-util/git psyco

If the local user name does not match the user name with which you access SVN, you can put the following script in your PATH before the regular location of "svn" (usually /usr/bin/).

Here, we assume that $HOME/bin is searched early enough. Replace "werner" with your user name.

cat <<EOF >$HOME/bin/svn
#!/bin/sh
exec `type -path svn` --username=werner "\$@"
EOF
chmod +x $HOME/bin/svn

Prerequisites (LAB)

The following package must be installed on the

  • xc - a simple communications program for the serial port

Gentoo users can obtain this with:

emerge -u xc

Assumptions (LAB, CARD)

There are the following assumptions about hardware setup and devices:

  • the serial console of the Neo phone is connected to /dev/ttyS0 on LAB Debug_Board
  • the JTAG wiggler is connected to /dev/parport0 on LAB. See Debug Board and

Connecting_GTA01Bv2_with_Debug_Board

  • cards inserted in the SD/MMC card reader appear as /dev/uba on CARD and can be mounted on /mnt/tmp (we'll specify the mount point explicitly, so it only has to be there, but we don't need /etc/fstab to reflect this). If in doubt,
mkdir -p /mnt/tmp

OpenEmbedded build: initial downloads (BUILD)

First, we obtain a snapshot of the OpenEmbedded-based tree used by OpenMoko, plus the OE build tool called BitBake.

Obtaining OpenMoko SVN tree

Obtain revision 887 of the OpenMoko tree. This is a "known to be good", version (provided the fixes below are applied). Unfortunately, at some places, "current" versions of upstream packages may get included, thus the build may still fail. If it does, you may wish to inform the authorities.

The checkout should take about 45 minutes over an Internet connection with a round-trip time to svn.openmoko.org of 350 ms.

cd $OMDIR
svn co -r 887 https://svn.openmoko.org/ openmoko

Installing BitBake

Install version 1.6 of BitBake, the build tool of OE. (This is quick.)

svn co http://svn.berlios.de/svnroot/repos/bitbake/branches/bitbake-1.6/ bitbake
cd bitbake
./setup.py install
cd ..

Obtaining OpenEmbedded snapshot

Obtain a snapshot of the Monotone repository of OpenEmbedded, then update it to the latest version, and finally check out our "known to be good" revision. We extract things into $OMDIR/openembedded. OE.mnt.bz2 is about 100 MB.

wget http://www.openembedded.org/snapshots/OE.mtn.bz2
bunzip2 OE.mtn.bz2
mtn --db=OE.mtn pull monotone.openembedded.org org.openembedded.dev
mtn --db=OE.mtn checkout --branch=org.openembedded.dev -r f499733e6db527846e1a48cf70f9862d6b3798ae openembedded
NOTE: For advanced users: if you ever want to update to the latest version of

the repository, you would do a "pull" (see above), followed by:

cd $OMDIR/openembedded && mnt update



Set up the directories for the cache of upstream files and all material related to local builds, then put our configuration file there: OpenMoko#Setting_up_an_OpenMoko_SDK

mkdir -p sources build/conf build/tmp/stamps/armv4t-linux
cat <<EOF >build/conf/local.conf
MACHINE = "fic-gta01"
DISTRO = "openmoko"
BUILD_ARCH = "`uname -m`"
EOF

OpenEmbedded build: fixes (BUILD)

There are unfortunately some problems in the build process. The following fixes work around them:

cd $OMDIR/sources
  • upstream moves old packages away, gratuitously breaking downstreams
wget http://ftp.mozilla.org/pub/mozilla.org/js/older-packages/js-1.5.tar.gz
touch ../build/tmp/stamps/armv4t-linux/js-1.5-r0.do_fetch
  • us2.samba.org mirror has vanished
wget http://us4.samba.org/samba/ftp/stable/samba-3.0.14a.tar.gz
touch ../build/tmp/stamps/armv4t-linux/samba-3.0.14a-r15.do_fetch
  • ghastly patch with CRLF and trailing blanks
perl -pi.orig -e 's/ *$//;s/\r//g' ../openembedded/packages/gcc/gcc-4.1.1/gcc-4.1.1-pr13685-1.patch

OpenEmbedded build (BUILD)

openmoko/trunk/oe/conf/site.conf expects the OpenMoko-specific OE packages in $OMDIR/oe

cd $OMDIR
ln -s openmoko/trunk/oe .

We're now ready to run the build. This will take a while.

cd $OMDIR/build
bitbake openmoko-devel-image

Note that the build will stop several times to ask for SVN access and whether to accept certificates. If you're not quick enough to respond, the underlying session may time out. In this case, just restart "bitbake openmoko-devel-image" and it will pick up from where it left off.

The whole build process involved numerous downloads, takes about 7 hours on an Athlon 64 3200+ (about 1.5h of delays were caused by ftp.debian.org not working properly during this test run), and ends with a message like this:

Build statistics:
  Attempted builds: 4
NOTE: THE INSTRUCTIONS BELOW YIELD A SETUP WITHOUT BAD BLOCK HANDLING}

We're in the process of migrating to a configuration of u-boot and kernel that can skip bad blocks in the NAND Flash. If your NAND Flash has no bad blocks, you can proceed safely, and migrate later.

{note


GTA01Bv2 # nand write.e 0x32000000 0x00634000 0x15AC000

The last argument is the hex number obtained from "bc"

      1. Configure the boot loader ###############################################
  1. Last but not least, we have to set up the boot loader to automatically boot
  2. from Flash. This is in fact easy, since we've already wiped out the
  3. environment, and can thus simply let it restore the default settings.
  1. ---------- We reset to force the boot loader to use the default settings.
  2. GTA01Bv2 # reset
  3. ---------- Wait until the "U-Boot [...]" message, then hit a key.
  4. ---------- It will display
  5. ---------- "*** Warning - bad CRC or NAND, using default environment"
  6. ---------- Save the restored settings in NAND
  7. GTA01Bv2 # saveenv
  8. ---------- Power cycle to boot the Neo (see remarks above)


      1. END #####################################################################
  1. Congratulations ! You've just completed level 1 of the OpenMoko adventure.

</pre>

Personal tools

'Building OpenMoko from scratch Walk-Through

according to Werner Almesberger <werner@openmoko.org>.


This is a guide describing how I set up a running OpenMoko system from scratch.

Do not treat this like a shell script! There are various configuration items you need to set (or skip, as it may be), operations that depend on how your host(s) are set up, and also on the revision of the target platform.

Instead, look at each step, read the instructions, and copy and paste what makes sense for you, and adapt what you disagree with. Links to original and background material in the Wiki are included wherever useful.

Please note that this file isn't actively maintained and is guaranteed to suffer bit rot as I begin to use more and more shortcuts in the build process. Furthermore, naming and other preferences are my own, may have little to do with what is considered common or even "official" style. If you find anything particularly stupid, please let me know

Roles

The build process may spread over multiple machines. They have the following roles:

  • BUILD - build host, with quick access to the files and CPU power. Must have Internet access.
  • LAB - lab machine connected to the debug board (serial and JTAG) and to USB on the Neo (since this will probably be just a single machine, the roles are not further divided)
  • CARD - machine with a USB-attached SD/MMC card reader

All machines are assumed to share the same filesystem layout. In the title of each of the section below, the respective role is indicated. "(all)" is for settings that apply to all machines, or that - for simplicity - can be applied to all of them.

Directory layout (all)

$OMDIR (/home/moko)	base directory for the whole tree
  openmoko/		files from OpenMoko subversion (SVN) repository
  openembedded/	files from OpenEmbedded (OE) Monotone repository
  sources/		cached downloads of OE
  build/		OE build directory

Environment variables (all)

For simplicity, we just set these environment variables on all hosts involved. If you're not comfortable with this, feel free to weed out the ones you don't need.

Our base directory (configure this for local arrangements):

export OMDIR=/home/moko

The path to the sources/ directory on the buildhost, in SCP path name syntax (replace "werner" with your user name):

export BUILDHOST_SRC=werner@buildhost.openmoko.org:/space/fic/openmoko/sources/

The search path for BitBake files. Note that the order is of vital importance.

export BBPATH=$OMDIR/build:$OMDIR/openmoko/trunk/oe:$OMDIR/openembedded

Permissions (BUILD)

In order to perform the build process, you have to obtain the following permissions:

  • write access to the OpenMoko SVN repository (in principle, it should be possible to simplify this to read access. For further study.)

Prerequisites (BUILD)

There must be at least 7 GB of free space on $OMDIR.

In addition to the traditional development tools (gcc, patch, etc.), the following packages must be installed on the build host:

#   subversion	version control system used by OpenMoko and others
#   quilt	patch management system used by the Linux kernel and others
#   monotone	version control system used by OpenEmbedded. Needs a recent
#		version, e.g., 0.32, although also 0.31 should work.
#   diffstat	the OE build process wants this
#   texi2html	this too
#   git		version control system used by the Linux kernel and others.
#		Do not confuse this with the "GNU Interactive Tools"

Furthermore, the following package can be installed optionally:

#   psycho	Python just-in-time compiler. Speeds up BitBake considerably.
#		Strongly recommended.

Gentoo users can obtain all this with (note that, at the time of writing, Monotone 0.32 isn't available without setting the ~x86 keyword):

echo 'dev-util/monotone ~'`readlink /etc/make.profile | awk -F / '{print $6}'`\
  >>/etc/portage/package.keywords
emerge -u subversion quilt monotone diffstat texi2html dev-util/git psyco

If the local user name does not match the user name with which you access SVN, you can put the following script in your PATH before the regular location of "svn" (usually /usr/bin/).

Here, we assume that $HOME/bin is searched early enough. Replace "werner" with your user name.

cat <<EOF >$HOME/bin/svn
#!/bin/sh
exec `type -path svn` --username=werner "\$@"
EOF
chmod +x $HOME/bin/svn

Prerequisites (LAB)

The following package must be installed on the

  • xc - a simple communications program for the serial port

Gentoo users can obtain this with:

emerge -u xc

Assumptions (LAB, CARD)

There are the following assumptions about hardware setup and devices:

  • the serial console of the Neo phone is connected to /dev/ttyS0 on LAB Debug_Board
  • the JTAG wiggler is connected to /dev/parport0 on LAB. See Debug Board and

Connecting_GTA01Bv2_with_Debug_Board

  • cards inserted in the SD/MMC card reader appear as /dev/uba on CARD and can be mounted on /mnt/tmp (we'll specify the mount point explicitly, so it only has to be there, but we don't need /etc/fstab to reflect this). If in doubt,
mkdir -p /mnt/tmp

OpenEmbedded build: initial downloads (BUILD)

First, we obtain a snapshot of the OpenEmbedded-based tree used by OpenMoko, plus the OE build tool called BitBake.

Obtaining OpenMoko SVN tree

Obtain revision 887 of the OpenMoko tree. This is a "known to be good", version (provided the fixes below are applied). Unfortunately, at some places, "current" versions of upstream packages may get included, thus the build may still fail. If it does, you may wish to inform the authorities.

The checkout should take about 45 minutes over an Internet connection with a round-trip time to svn.openmoko.org of 350 ms.

cd $OMDIR
svn co -r 887 https://svn.openmoko.org/ openmoko

Installing BitBake

Install version 1.6 of BitBake, the build tool of OE. (This is quick.)

svn co http://svn.berlios.de/svnroot/repos/bitbake/branches/bitbake-1.6/ bitbake
cd bitbake
./setup.py install
cd ..

Obtaining OpenEmbedded snapshot

Obtain a snapshot of the Monotone repository of OpenEmbedded, then update it to the latest version, and finally check out our "known to be good" revision. We extract things into $OMDIR/openembedded. OE.mnt.bz2 is about 100 MB.

wget http://www.openembedded.org/snapshots/OE.mtn.bz2
bunzip2 OE.mtn.bz2
mtn --db=OE.mtn pull monotone.openembedded.org org.openembedded.dev
mtn --db=OE.mtn checkout --branch=org.openembedded.dev -r f499733e6db527846e1a48cf70f9862d6b3798ae openembedded
NOTE: For advanced users: if you ever want to update to the latest version of

the repository, you would do a "pull" (see above), followed by:

cd $OMDIR/openembedded && mnt update



Set up the directories for the cache of upstream files and all material related to local builds, then put our configuration file there: OpenMoko#Setting_up_an_OpenMoko_SDK

mkdir -p sources build/conf build/tmp/stamps/armv4t-linux
cat <<EOF >build/conf/local.conf
MACHINE = "fic-gta01"
DISTRO = "openmoko"
BUILD_ARCH = "`uname -m`"
EOF

OpenEmbedded build: fixes (BUILD)

There are unfortunately some problems in the build process. The following fixes work around them:

cd $OMDIR/sources
  • upstream moves old packages away, gratuitously breaking downstreams
wget http://ftp.mozilla.org/pub/mozilla.org/js/older-packages/js-1.5.tar.gz
touch ../build/tmp/stamps/armv4t-linux/js-1.5-r0.do_fetch
  • us2.samba.org mirror has vanished
wget http://us4.samba.org/samba/ftp/stable/samba-3.0.14a.tar.gz
touch ../build/tmp/stamps/armv4t-linux/samba-3.0.14a-r15.do_fetch
  • ghastly patch with CRLF and trailing blanks
perl -pi.orig -e 's/ *$//;s/\r//g' ../openembedded/packages/gcc/gcc-4.1.1/gcc-4.1.1-pr13685-1.patch

OpenEmbedded build (BUILD)

openmoko/trunk/oe/conf/site.conf expects the OpenMoko-specific OE packages in $OMDIR/oe

cd $OMDIR
ln -s openmoko/trunk/oe .

We're now ready to run the build. This will take a while.

cd $OMDIR/build
bitbake openmoko-devel-image

Note that the build will stop several times to ask for SVN access and whether to accept certificates. If you're not quick enough to respond, the underlying session may time out. In this case, just restart "bitbake openmoko-devel-image" and it will pick up from where it left off.

The whole build process involved numerous downloads, takes about 7 hours on an Athlon 64 3200+ (about 1.5h of delays were caused by ftp.debian.org not working properly during this test run), and ends with a message like this:

Build statistics:
  Attempted builds: 4
NOTE: THE INSTRUCTIONS BELOW YIELD A SETUP WITHOUT BAD BLOCK HANDLING}

We're in the process of migrating to a configuration of u-boot and kernel that can skip bad blocks in the NAND Flash. If your NAND Flash has no bad blocks, you can proceed safely, and migrate later.

{note


GTA01Bv2 # nand write.e 0x32000000 0x00634000 0x15AC000

The last argument is the hex number obtained from "bc"

      1. Configure the boot loader ###############################################
  1. Last but not least, we have to set up the boot loader to automatically boot
  2. from Flash. This is in fact easy, since we've already wiped out the
  3. environment, and can thus simply let it restore the default settings.
  1. ---------- We reset to force the boot loader to use the default settings.
  2. GTA01Bv2 # reset
  3. ---------- Wait until the "U-Boot [...]" message, then hit a key.
  4. ---------- It will display
  5. ---------- "*** Warning - bad CRC or NAND, using default environment"
  6. ---------- Save the restored settings in NAND
  7. GTA01Bv2 # saveenv
  8. ---------- Power cycle to boot the Neo (see remarks above)


      1. END #####################################################################
  1. Congratulations ! You've just completed level 1 of the OpenMoko adventure.

    </pre>