MokoMakefile

From Openmoko

Revision as of 23:07, 3 September 2007 by RodWhitby (Talk | contribs)

Jump to: navigation, search

Contents

MokoMakefile

MokoMakefile is a fully automated way of setting up an OpenMoko development environment. It is an invaluable tool for getting new developers up and running with a build environment which is configured the same as all the other existing developers. It brings the same repeatability to build environment creation and maintenance as that which OpenEmbedded brings to the main task of actually building embedded software distributions.

Note that MokoMakefile does *not* replace bitbake, or svn, or monotone, or openembedded, or qmake, or anything else. It is a wrapper around all that to make it easy to set up and maintain a development environment that fully complies with the setup instructions published by OpenMoko. Note that you need about 12 GB of available disk space for MokoMakefile to succeed (see below for a tip on how to reduce this to . Please check that your RAM + swap partition space is greater than 512 MB (around 1GB?). Note that the initial build can take 5 hours (on 2GHz core2duo without multiprocessor optimization) to several days.

MokoMakefile is developed by Rod Whitby - it is not an official product of OpenMoko (although I would be happy for them to pick it up and use it internally). If there is any discrepancy between the official OpenMoko build instructions, and the operation of the MokoMakefile, then you should consider the official instructions to be correct.

The MokoMakefile is able to build either OM-2007.1 or OM-2007.2 images. The core team chooses the default, but you can select one or the other at the top of the Makefile.

Installation

With Qemu

This is an easy and good way to install a local version of openmoko on your linux computer(ubuntu). This is mostly the fastest way to get things working (approximatly 15 min)

see MokoMakefile#QEMU

Manually

Here are the steps to use it:

1 - Make sure your build host is set up according to:

  http://www.openembedded.org/wiki/OEandYourDistro

2 - Create your $OMDIR directory (note that you can change ~/moko to any directory you like):

  mkdir ~/moko ; cd ~/moko

3 - Grab MokoMakefile:

  wget http://www.rwhitby.net/files/openmoko/Makefile

If that doesn't work, try

  wget http://svn.nslu2-linux.org/svnroot/mokomakefile/trunk/Makefile
  note: If you want to compile for the old version 2007.1 instead of the new
        version edit the top of the Makefile. Edit the lines at the top to 
        look like this:
            OPENMOKO_GENERATION = 2007.1
            #OPENMOKO_GENERATION = 2007.2
        On kubuntu and Debian Etch I also had to apt-get install help2man

4 - Set up the environment:

  make setup

5 - Start building. Before starting a lengthy make process, check in Tips section about how to make Make multicore aware. You may want to modify the build/conf/local.conf file for your target (emulation/chroot) environment:

  make openmoko-devel-image

This will set up the recommended directory structure as described in Building OpenMoko from scratch, will download all the required software (from the right places with the right versions), and will immediately start building an image.

Once you have done this, you can choose to continue using the MokoMakefile to initiate your subsequent builds, or you can go into the build directory and run bitbake commands manually. The choice is yours.

Updating the environment

For easy maintenance of your build environment the following commands are available.

1 - To update the MokoMakefile to the latest version:

  make update-makefile 

2 - To make sure that any recent changes to the build directory structure have been applied:

  make setup 

3 - To update the OpenMoko repository checkout and the MokoMakefile patches to the latest version:

  make update

A quick way to rebuild a new image with the latest updates:

  make update-makefile && make setup update openmoko-devel-image

Reporting Problems

First, make sure that the problem is reproducible after running

make update-makefile && make setup && make update

then running

make clean-package-<foo>

(where you replace <foo> with the name of the package which is failing)

then running

make openmoko-devel-image

If you can get the error to occur three times in a row after running that sequence of commands (including the update and setup steps) three times, then feel free to report it to rwhitby in #openmoko on IRC.

Known MokoMakefile errors

If you experience the following after changing from OM-2007.1 to OM-2007.2:

Patch bitbake-1.6.6-om3.patch does not apply (enforce with -f)

then type "make clobber-patches" to fix it. There was a period of 24 hours when there was a bug in the MokoMakefile which causes this problem. Once the patches have been clobbered, they will re-download and the problem will not reoccur.

Work-arounds

Work-arounds for temporary or isolated problems should be added to the Discussion page which is associated with this page. As they are fixed, they will be removed from that page.

Tips

  • You can reduce the amount of consumed disk space significantly by adding
  INHERIT += "rm_work"

in your local.conf (e.g. ~/moko/build/conf/local.conf). This will remove the contents of each build/tmp/work/*/<package> directory after the corresponding package builds correctly.

  • If you an encounter an error with monotone similar to the following:
  mtn: misuse: database /home/username/moko/OE.mtn is laid out according to an old schema

Then you need to upgrade OE.mtn Use the following command while in ~/moko:

  # mtn --db OE.mtn db migrate
  • If a certain package does not build due to corrupted download or some such try to remove the sources and rebuild it.
rm sources/<package>*
cd build
. ../setup-env
bitbake -crebuild <package>

after that your build might just work again.

  • For people with multiple CPU's (or dual-core ones) this small patch might be useful to build things faster.

Edit the local.conf and add the following lines:

PARALLEL_MAKE = "-j 4"
BB_NUMBER_THREADS = "4"

Change the PARALLEL_MAKE and BB_NUMBER_THREADS values to something that suits better if it chokes your machine.

  • If you encounter an error related with the qemu-native package and not compiling for the qemu, you can edit the build/conf/local.conf file and add ENABLE_BINARY_LOCALE_GENERATION = "0" line to avoid the error.
  • To prevent building tons of locales, add a line like this to local.conf:
GLIBC_GENERATE_LOCALES = "en_US.UTF-8 nl_NL.UTF-8"
  • To not build any binary locales at all, add this to local.conf:
ENABLE_BINARY_LOCALE_GENERATION = "0"

QEMU

The MokoMakefile now has support for automatically building, flashing, and running the new Neo1973 emulator (which is based on QEMU).

Install in Ubuntu (7.04)

Make sure your build host is set up according to:

 http://www.openembedded.org/wiki/OEandYourDistro

make a new directory:

 mkdir openmoko
 cd openmoko

install the dependencies:

 sudo apt-get install gcc-3.4 lynx netpbm libsdl1.2-dev dosfstools

get the openmokofile:

 wget http://www.rwhitby.net/files/openmoko/Makefile

build it for Qemu:

 make qemu

This will build openmoko for Qemu. Once that is completed, it will automatic run it into the emulator. After you have closed it and want to run openmoko again, you should use the following command:

 make run-qemu

remark: If you want to calibrate your screen, look at http://onlinedev.blogspot.com/2007/08/calibrating-openmoko-v20072.html

Others

Dependencies:

  • gcc 3.4 or 3.3 ('apt-get install gcc-3.4' in Ubuntu 7.04)
  • lynx ('apt-get install lynx' in Ubuntu 7.04)
  • netpbm ('apt-get install netpbm' in Ubuntu 7.04)
  • sdl-devel ('apt-get install libsdl1.2-dev' in Ubuntu .704)
  • mkdosfs ('apt-get install dosfstools' in Debian 4.0)

make qemu” will build qemu-neo1973, download the latest official OpenMoko images, flash the images into the virtual NAND flash, create an empty virtual SD card, and run the emulator (you still need to install the makefile as mentioned above, however). If this is all you want to do, then you don't even need monotone installed on the machine, and you don't need to download the OpenEmbedded monotone database. Just make sure you don't type "make setup" (as that will invoke all the monotone and openembedded stuff).

Other targets are “make download-images” (to download the latest official images), “make flash-qemu-official” (to flash those images) and “make flash-qemu-local” (which will flash your latest locally built images), which can then be followed by “'make run-qemu” or “make run-qemu-snapshot” to run the emulator with the most recently flashed images. You can also use “'make qemu-copy-package-foo” to copy the .ipk file for package foo to the virtual SD card, and later use “'ipkg install /media/mmcblk0/file” from within the emulated Neo1973 to install the package.

You're also invited to read more about how to run OpenMoko under QEMU in the article dedicated to this subject.

Developing with MokoMakefile

NOTE: If using MokoMakefile with OM2007.2 then references to $OMDIR/openmoko should be replaced with $OMDIR/openembedded. Also references to tmp/work/armv4t-linux should be replaced with tmp/work/fic-gta01-angstrom-linux-gnueabi


For the following explanations $OMDIR is the directory where there Makefile puts all the stuff.

To make in-tree changes and have them built and used by qemu:

 cd $OMDIR/openmoko
 quilt new descriptive-patch-name.patch
 quilt add trunk/src/name-of-file-to-change # do this for every file you are about to modify
 ...make the changes...
 quilt refresh # this creates a file in $OMDIR/patches/openmoko-HEAD/ and updates the quilt series file there

Note: Do NOT use absolute paths as this confuses quilt and will get you a diff of the file against /dev/null!

To build the changes and have them used by qemu:

 make build-qemu
 make flash-qemu-local
 make run-qemu

If you want to modify applications instead of the openmoko toolchain, this is what you have to do (example: openmoko-messages):

 cd $OMDIR/build
 . ../setup-env
 bitbake -c unpack openmoko-messages
 cd ../build/tmp/work/armv4t-linux/openmoko-messages-0.0.1+svnnow-r2_2276/openmoko-messages/
 ...make the changes...
 cd -
 bitbake openmoko-messages

Then continue with MokoMakefile usage.

If you want to add an application to your openmoko distribution, do this: All file edits should be done using quilt as described above. That way a patch can easily be submitted to the openmoko project. First, create a directory that will correspond to your package and edit a .bb file in there:

 cd $OMDIR/openmoko/
 quilt new mycoolpackage.patch
 mkdir trunk/oe/packages/mycoolpackage
 quilt add trunk/oe/packages/mycoolpackage/mycoolpackage_1.bb
 quilt edit trunk/oe/packages/mycoolpackage/mycoolpackage_1.bb

The file should have the following content:

 DESCRIPTION = "This is a cool package"
 SECTION = "username/mycoolpackage"
 PV = "1"
 
 inherit autotools
 
 SRC_URI = "http://www.example.com/download/mycoolpackage-1.tar.gz"

Explanation:

  • DESCRIPTION - Just a short text explaining the package
  • SECTION - I have no clue, but I'll use username/mycoolpackage for now
  • PV - Package Version
  • inherit autotools - The package can be compiled by './configure && make && make install' so we tell MokoMakefile to do it this way.
  • SRC_URI = ... - This is the download location of the package source. It's imperative that the tar.gz contains a directory called packagename-packageversion (in this case: mycoolpackage-1) so that MokoMakefile can find it automatically or the build will fail.

This is not all. We also need to tell MokoMakfile that it needs to build and include the package in the image. To do this, do

 $OMDIR/openmoko# quilt edit trunk/oe/packages/tasks/task-openmoko.bb

Here, increase the value PR by one and add mycoolpackage \ (with the backslash!) just before the line reading # update-alternatives \.

Now run

 quilt refresh
 cd ..
 make update openmoko-devel-image

And if everything's alright you should now have an OpenMoko image to flash to your phone or run in qemu as described above.

Hello World application

There is a Hello World! tutorial available too.

Testimonials

MokoMakefile is recommended by 4 out of 4 new developers on #openmoko, with testimonials such as "For some reason last night I couldn't get my manual install of everything to work (bb complained about my bbpath I think) ... but with your makefile, it works great!", "MokoMakefile rocks!", and "Wow this build system is nice - it just seems more polished than my gumstix toolchain buildroot system".

Project page: http://mokomakefile.projects.openmoko.org/

Personal tools

MokoMakefile

MokoMakefile is a fully automated way of setting up an OpenMoko development environment. It is an invaluable tool for getting new developers up and running with a build environment which is configured the same as all the other existing developers. It brings the same repeatability to build environment creation and maintenance as that which OpenEmbedded brings to the main task of actually building embedded software distributions.

Note that MokoMakefile does *not* replace bitbake, or svn, or monotone, or openembedded, or qmake, or anything else. It is a wrapper around all that to make it easy to set up and maintain a development environment that fully complies with the setup instructions published by OpenMoko. Note that you need about 12 GB of available disk space for MokoMakefile to succeed (see below for a tip on how to reduce this to . Please check that your RAM + swap partition space is greater than 512 MB (around 1GB?). Note that the initial build can take 5 hours (on 2GHz core2duo without multiprocessor optimization) to several days.

MokoMakefile is developed by Rod Whitby - it is not an official product of OpenMoko (although I would be happy for them to pick it up and use it internally). If there is any discrepancy between the official OpenMoko build instructions, and the operation of the MokoMakefile, then you should consider the official instructions to be correct.

The MokoMakefile is able to build either OM-2007.1 or OM-2007.2 images. The core team chooses the default, but you can select one or the other at the top of the Makefile.

Installation

With Qemu

This is an easy and good way to install a local version of openmoko on your linux computer(ubuntu). This is mostly the fastest way to get things working (approximatly 15 min)

see MokoMakefile#QEMU

Manually

Here are the steps to use it:

1 - Make sure your build host is set up according to:

  http://www.openembedded.org/wiki/OEandYourDistro

2 - Create your $OMDIR directory (note that you can change ~/moko to any directory you like):

  mkdir ~/moko ; cd ~/moko

3 - Grab MokoMakefile:

  wget http://www.rwhitby.net/files/openmoko/Makefile

If that doesn't work, try

  wget http://svn.nslu2-linux.org/svnroot/mokomakefile/trunk/Makefile
  note: If you want to compile for the old version 2007.1 instead of the new
        version edit the top of the Makefile. Edit the lines at the top to 
        look like this:
            OPENMOKO_GENERATION = 2007.1
            #OPENMOKO_GENERATION = 2007.2
        On kubuntu and Debian Etch I also had to apt-get install help2man

4 - Set up the environment:

  make setup

5 - Start building. Before starting a lengthy make process, check in Tips section about how to make Make multicore aware. You may want to modify the build/conf/local.conf file for your target (emulation/chroot) environment:

  make openmoko-devel-image

This will set up the recommended directory structure as described in Building OpenMoko from scratch, will download all the required software (from the right places with the right versions), and will immediately start building an image.

Once you have done this, you can choose to continue using the MokoMakefile to initiate your subsequent builds, or you can go into the build directory and run bitbake commands manually. The choice is yours.

Updating the environment

For easy maintenance of your build environment the following commands are available.

1 - To update the MokoMakefile to the latest version:

  make update-makefile 

2 - To make sure that any recent changes to the build directory structure have been applied:

  make setup 

3 - To update the OpenMoko repository checkout and the MokoMakefile patches to the latest version:

  make update

A quick way to rebuild a new image with the latest updates:

  make update-makefile && make setup update openmoko-devel-image

Reporting Problems

First, make sure that the problem is reproducible after running

make update-makefile && make setup && make update

then running

make clean-package-<foo>

(where you replace <foo> with the name of the package which is failing)

then running

make openmoko-devel-image

If you can get the error to occur three times in a row after running that sequence of commands (including the update and setup steps) three times, then feel free to report it to rwhitby in #openmoko on IRC.

Known MokoMakefile errors

If you experience the following after changing from OM-2007.1 to OM-2007.2:

Patch bitbake-1.6.6-om3.patch does not apply (enforce with -f)

then type "make clobber-patches" to fix it. There was a period of 24 hours when there was a bug in the MokoMakefile which causes this problem. Once the patches have been clobbered, they will re-download and the problem will not reoccur.

Work-arounds

Work-arounds for temporary or isolated problems should be added to the Discussion page which is associated with this page. As they are fixed, they will be removed from that page.

Tips

  • You can reduce the amount of consumed disk space significantly by adding
  INHERIT += "rm_work"

in your local.conf (e.g. ~/moko/build/conf/local.conf). This will remove the contents of each build/tmp/work/*/<package> directory after the corresponding package builds correctly.

  • If you an encounter an error with monotone similar to the following:
  mtn: misuse: database /home/username/moko/OE.mtn is laid out according to an old schema

Then you need to upgrade OE.mtn Use the following command while in ~/moko:

  # mtn --db OE.mtn db migrate
  • If a certain package does not build due to corrupted download or some such try to remove the sources and rebuild it.
rm sources/<package>*
cd build
. ../setup-env
bitbake -crebuild <package>

after that your build might just work again.

  • For people with multiple CPU's (or dual-core ones) this small patch might be useful to build things faster.

Edit the local.conf and add the following lines:

PARALLEL_MAKE = "-j 4"
BB_NUMBER_THREADS = "4"

Change the PARALLEL_MAKE and BB_NUMBER_THREADS values to something that suits better if it chokes your machine.

  • If you encounter an error related with the qemu-native package and not compiling for the qemu, you can edit the build/conf/local.conf file and add ENABLE_BINARY_LOCALE_GENERATION = "0" line to avoid the error.
  • To prevent building tons of locales, add a line like this to local.conf:
GLIBC_GENERATE_LOCALES = "en_US.UTF-8 nl_NL.UTF-8"
  • To not build any binary locales at all, add this to local.conf:
ENABLE_BINARY_LOCALE_GENERATION = "0"

QEMU

The MokoMakefile now has support for automatically building, flashing, and running the new Neo1973 emulator (which is based on QEMU).

Install in Ubuntu (7.04)

Make sure your build host is set up according to:

 http://www.openembedded.org/wiki/OEandYourDistro

make a new directory:

 mkdir openmoko
 cd openmoko

install the dependencies:

 sudo apt-get install gcc-3.4 lynx netpbm libsdl1.2-dev dosfstools

get the openmokofile:

 wget http://www.rwhitby.net/files/openmoko/Makefile

build it for Qemu:

 make qemu

This will build openmoko for Qemu. Once that is completed, it will automatic run it into the emulator. After you have closed it and want to run openmoko again, you should use the following command:

 make run-qemu

remark: If you want to calibrate your screen, look at http://onlinedev.blogspot.com/2007/08/calibrating-openmoko-v20072.html

Others

Dependencies:

  • gcc 3.4 or 3.3 ('apt-get install gcc-3.4' in Ubuntu 7.04)
  • lynx ('apt-get install lynx' in Ubuntu 7.04)
  • netpbm ('apt-get install netpbm' in Ubuntu 7.04)
  • sdl-devel ('apt-get install libsdl1.2-dev' in Ubuntu .704)
  • mkdosfs ('apt-get install dosfstools' in Debian 4.0)

make qemu” will build qemu-neo1973, download the latest official OpenMoko images, flash the images into the virtual NAND flash, create an empty virtual SD card, and run the emulator (you still need to install the makefile as mentioned above, however). If this is all you want to do, then you don't even need monotone installed on the machine, and you don't need to download the OpenEmbedded monotone database. Just make sure you don't type "make setup" (as that will invoke all the monotone and openembedded stuff).

Other targets are “make download-images” (to download the latest official images), “make flash-qemu-official” (to flash those images) and “make flash-qemu-local” (which will flash your latest locally built images), which can then be followed by “'make run-qemu” or “make run-qemu-snapshot” to run the emulator with the most recently flashed images. You can also use “'make qemu-copy-package-foo” to copy the .ipk file for package foo to the virtual SD card, and later use “'ipkg install /media/mmcblk0/file” from within the emulated Neo1973 to install the package.

You're also invited to read more about how to run OpenMoko under QEMU in the article dedicated to this subject.

Developing with MokoMakefile

NOTE: If using MokoMakefile with OM2007.2 then references to $OMDIR/openmoko should be replaced with $OMDIR/openembedded. Also references to tmp/work/armv4t-linux should be replaced with tmp/work/fic-gta01-angstrom-linux-gnueabi


For the following explanations $OMDIR is the directory where there Makefile puts all the stuff.

To make in-tree changes and have them built and used by qemu:

 cd $OMDIR/openmoko
 quilt new descriptive-patch-name.patch
 quilt add trunk/src/name-of-file-to-change # do this for every file you are about to modify
 ...make the changes...
 quilt refresh # this creates a file in $OMDIR/patches/openmoko-HEAD/ and updates the quilt series file there

Note: Do NOT use absolute paths as this confuses quilt and will get you a diff of the file against /dev/null!

To build the changes and have them used by qemu:

 make build-qemu
 make flash-qemu-local
 make run-qemu

If you want to modify applications instead of the openmoko toolchain, this is what you have to do (example: openmoko-messages):

 cd $OMDIR/build
 . ../setup-env
 bitbake -c unpack openmoko-messages
 cd ../build/tmp/work/armv4t-linux/openmoko-messages-0.0.1+svnnow-r2_2276/openmoko-messages/
 ...make the changes...
 cd -
 bitbake openmoko-messages

Then continue with MokoMakefile usage.

If you want to add an application to your openmoko distribution, do this: All file edits should be done using quilt as described above. That way a patch can easily be submitted to the openmoko project. First, create a directory that will correspond to your package and edit a .bb file in there:

 cd $OMDIR/openmoko/
 quilt new mycoolpackage.patch
 mkdir trunk/oe/packages/mycoolpackage
 quilt add trunk/oe/packages/mycoolpackage/mycoolpackage_1.bb
 quilt edit trunk/oe/packages/mycoolpackage/mycoolpackage_1.bb

The file should have the following content:

 DESCRIPTION = "This is a cool package"
 SECTION = "username/mycoolpackage"
 PV = "1"
 
 inherit autotools
 
 SRC_URI = "http://www.example.com/download/mycoolpackage-1.tar.gz"

Explanation:

  • DESCRIPTION - Just a short text explaining the package
  • SECTION - I have no clue, but I'll use username/mycoolpackage for now
  • PV - Package Version
  • inherit autotools - The package can be compiled by './configure && make && make install' so we tell MokoMakefile to do it this way.
  • SRC_URI = ... - This is the download location of the package source. It's imperative that the tar.gz contains a directory called packagename-packageversion (in this case: mycoolpackage-1) so that MokoMakefile can find it automatically or the build will fail.

This is not all. We also need to tell MokoMakfile that it needs to build and include the package in the image. To do this, do

 $OMDIR/openmoko# quilt edit trunk/oe/packages/tasks/task-openmoko.bb

Here, increase the value PR by one and add mycoolpackage \ (with the backslash!) just before the line reading # update-alternatives \.

Now run

 quilt refresh
 cd ..
 make update openmoko-devel-image

And if everything's alright you should now have an OpenMoko image to flash to your phone or run in qemu as described above.

Hello World application

There is a Hello World! tutorial available too.

Testimonials

MokoMakefile is recommended by 4 out of 4 new developers on #openmoko, with testimonials such as "For some reason last night I couldn't get my manual install of everything to work (bb complained about my bbpath I think) ... but with your makefile, it works great!", "MokoMakefile rocks!", and "Wow this build system is nice - it just seems more polished than my gumstix toolchain buildroot system".

Project page: http://mokomakefile.projects.openmoko.org/