Create an ipk package from sources

From Openmoko

Revision as of 17:11, 28 July 2007 by Squalyl (Talk | contribs)

Jump to: navigation, search

This is a small tutorial to help people that want to write new software for Openmoko, eg. to port existing programs.

Contents

Theory

OpenMoko uses OpenEmbedded, a package and root image build system.

OpenEmbedded provides patches and Bitbake Recipes, that are similar to .spec files to build RPM packages. Bitbake is used to process recipes and obtain packages and FS images. For example, bitbake can fetch sources, apply patches, configure, make and install compiled sources. Other steps are user definable, but this is not the scope of this tutorial.

A standard use of bitbake is to download an existing "vanilla" package, ie "gcc" or "binutils" or "linux kernel", apply platform patches (adding bugfixes, porting fixes, drivers, etc...) then compile and package the resulting binaries.

Software versionning

Versions management is a very wide topic, there are tons of different opinions about it, I'll try to be quick.

Packages versions are important in a distribution building mechanism, and package versions are a complete part of the full package name. They are obviously used to track important changes made to your package, and isolate each version. Here we must be a little aware of basic version management. Current (most advanced) version is named the "trunk". Additions, fixes, etc. are done to the trunk.

Once a certain level of functionality is achieved, a copy of the trunk (a "snapshot") is taken, and named with a version number. As a result, current development continues on the trunk, but users can download well defined "releases", with known bugs and functionnality level. This is called "branching". If version specific bugs are found in version a.b, then we can update this branch independently.

In bitbake, we must specify which release of our software we are going to package. This may not be an habit, especially if you only develop small programs, not maintained by more than one or two person. If your software is named "myprogram" but has no real version number attached, you can just rename your project directory to "myprogram-1.0".

This is a good practice; If you are packaging a software, then it will be used by other people, so when you modify it next time, put it in a folder named "myprogram-1.1" :) This brings clarity to users, ease of management to you, and ability to plan future releases to your team.

Example

Say you have a working software package, ie a file system directory with sources, Makefile, etc... Because this is a common method, I will base my presentation on a project generated using autotools. This implies that the software can be installed somewhere by following the magical steps ./configure && make && make install.


Proposed directory layout

Bitbake recipes are single files describing how to build a package. I recommend to have a recipe for each package version (myprogram-1.0.bb, myprogram-1.1.bb, etc.) all stored in a directory named "myprogram"

The recipe

In the bitbake file, you need to give

  • a description for your package, variable name is DESCRIPTION
  • a package name (PN)
  • a package version (PV)
  • a source URI, I'll give details later
  • a source version, to checkout a particular revision of versionned trees

here's an example:

#package description
DESCRIPTION = "An example application for openmoko, depending on a custom source repository"
SECTION="squalyl/myhello"

#this is optional
LICENSE="GPLv2"

PN="myhello"
PV="1.0"

#use "now" to force svn updates at each build
SRCDATE_${PN}="now"
SRC_URI=svn://repo/path/myhello;proto=https;module=${PN}-${PV}

#for a tarball use:
#SRC_URI=http://server/path/package/${PN}-{$PV}.tar.gz

#include standard rules to build a autoconf/automake package
inherit autotools

Expressions starting by ${} are substituted, this allows you to download a tarball named "mypackage-1.0.tar.gz" for example

Source URIs

bitbake can get the source code for your project from a variety of locations including tarball, subversion and git.

All of this is explained in the bitbake manual, particulary here.

The "build" directory, the one where bitbake will run "configure" et al, MUST have the name template: "PACKAGE_NAME-PACKAGE_VERSION", so

  • tarballs must extract to a subdirectory with this name
  • svn top directory must have this name

It's very important. I spent 3 days blocking on this!

Tell bitbake to build our package

After this, we have to inform bitbake that we have a file for it. I assume a standard Openmoko directory tree with:

$OMDIR
 +- build
 |  +- conf
 |  |  +- local.conf
 |  +- tmp
 +- openmoko
 +- openembedded
 +- sources

etc...

Our personal tree can be everywhere else:

$HOME
 +- $OMDIR
 |  +- ...
 +- MypersonalOpenMokoPackages
    +- packages
    |  +- myhello
    |     +- myhello-1.0.bb
    +- sources
       +- myhello
          +- myhello-1.0
             +- configure
             +- Makefile.am
             +- ...

The only thing to do is edit $OMDIR/build/local.conf to tell bitbake to search our folders for .bb files:

BBFILES+="${HOME}/MypersonalOpenMokoPackages/packages/*/*.bb"

And then, running

$ cd $OMDIR/build
$ bitbake mypackage

will bring all the necessary operations to create a package in $OMDIR/build/tmp/deploy/ipk/armv4t/myhello .....

The following is an example build log:

OE Build Configuration:
BB_VERSION     = "1.6.9"
OE_REVISION    = "<unknown>"
TARGET_ARCH    = "arm"
TARGET_OS      = "linux"
MACHINE        = "fic-gta01"
DISTRO         = "openmoko"
DISTRO_VERSION = ".dev-snapshot-20070723"
TARGET_FPU     = "soft"

NOTE: This BitBake doesn't support revision fetching, falling back to current date
NOTE: preferred version 2.4 of glibc not available
NOTE: preferred version 2.4 of glibc-intermediate not available
NOTE: preferred version 2.4 of glibc not available
NOTE: package myhello-1.0: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_fetch: started
NOTE: Fetch svn://svn.unsads.com/squalyl/om/src/myhello;proto=https;module=myhello-1.0
A    myhello-1.0/configure.ac
A    myhello-1.0/include
A    myhello-1.0/include/config.h.in
A    myhello-1.0/cleanall.sh
A    myhello-1.0/AUTHORS
A    myhello-1.0/bootstrap.sh
A    myhello-1.0/ChangeLog
A    myhello-1.0/src
A    myhello-1.0/src/hello.c
A    myhello-1.0/Makefile.am
A    myhello-1.0/NEWS
A    myhello-1.0/README
Révision 2116 extraite.
NOTE: package myhello-1.0-r0_0_200707231759: task do_fetch: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_unpack: started
NOTE: Unpacking /home/squalyl/OM/sources/myhello-1.0_svn.unsads.com_.squalyl.om.src.myhello__now.tar.gz to /home/squalyl/OM/build/tmp/work/armv4t-linux/myhello-1.0-r0_0_200707231759/
NOTE: package myhello-1.0-r0_0_200707231759: task do_unpack: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_patch: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_patch: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_configure: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_configure: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_compile: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_compile: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_install: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_install: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_package: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_package: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_package_write: started
Packaged contents of myhello-dbg into /home/squalyl/OM/build/tmp/deploy/ipk/armv4t/myhello-dbg_1.0-r0_0_200707231759_armv4t.ipk
Packaged contents of myhello into /home/squalyl/OM/build/tmp/deploy/ipk/armv4t/myhello_1.0-r0_0_200707231759_armv4t.ipk
NOTE: Not creating empty archive for myhello-doc-1.0-r0_0_200707231759
NOTE: Not creating empty archive for myhello-dev-1.0-r0_0_200707231759
NOTE: Not creating empty archive for myhello-locale-1.0-r0_0_200707231759
NOTE: package myhello-1.0-r0_0_200707231759: task do_package_write: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_populate_staging: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_populate_staging: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_build: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_build: completed
NOTE: package myhello-1.0: completed
NOTE: build 200707231952: completed
Build statistics:
  Attempted builds: 1

Get this package included in the rootfs image

bitbake openmoko-devel-image does not include your new package in the rootfs image.

To do this, use:

bitbake openmoko-devel-image -crebuild

This will actually rebuild the rootfs, including your new packages.

Personal tools

This is a small tutorial to help people that want to write new software for Openmoko, eg. to port existing programs.

Theory

OpenMoko uses OpenEmbedded, a package and root image build system.

OpenEmbedded provides patches and Bitbake Recipes, that are similar to .spec files to build RPM packages. Bitbake is used to process recipes and obtain packages and FS images. For example, bitbake can fetch sources, apply patches, configure, make and install compiled sources. Other steps are user definable, but this is not the scope of this tutorial.

A standard use of bitbake is to download an existing "vanilla" package, ie "gcc" or "binutils" or "linux kernel", apply platform patches (adding bugfixes, porting fixes, drivers, etc...) then compile and package the resulting binaries.

Software versionning

Versions management is a very wide topic, there are tons of different opinions about it, I'll try to be quick.

Packages versions are important in a distribution building mechanism, and package versions are a complete part of the full package name. They are obviously used to track important changes made to your package, and isolate each version. Here we must be a little aware of basic version management. Current (most advanced) version is named the "trunk". Additions, fixes, etc. are done to the trunk.

Once a certain level of functionality is achieved, a copy of the trunk (a "snapshot") is taken, and named with a version number. As a result, current development continues on the trunk, but users can download well defined "releases", with known bugs and functionnality level. This is called "branching". If version specific bugs are found in version a.b, then we can update this branch independently.

In bitbake, we must specify which release of our software we are going to package. This may not be an habit, especially if you only develop small programs, not maintained by more than one or two person. If your software is named "myprogram" but has no real version number attached, you can just rename your project directory to "myprogram-1.0".

This is a good practice; If you are packaging a software, then it will be used by other people, so when you modify it next time, put it in a folder named "myprogram-1.1" :) This brings clarity to users, ease of management to you, and ability to plan future releases to your team.

Example

Say you have a working software package, ie a file system directory with sources, Makefile, etc... Because this is a common method, I will base my presentation on a project generated using autotools. This implies that the software can be installed somewhere by following the magical steps ./configure && make && make install.


Proposed directory layout

Bitbake recipes are single files describing how to build a package. I recommend to have a recipe for each package version (myprogram-1.0.bb, myprogram-1.1.bb, etc.) all stored in a directory named "myprogram"

The recipe

In the bitbake file, you need to give

  • a description for your package, variable name is DESCRIPTION
  • a package name (PN)
  • a package version (PV)
  • a source URI, I'll give details later
  • a source version, to checkout a particular revision of versionned trees

here's an example:

#package description
DESCRIPTION = "An example application for openmoko, depending on a custom source repository"
SECTION="squalyl/myhello"

#this is optional
LICENSE="GPLv2"

PN="myhello"
PV="1.0"

#use "now" to force svn updates at each build
SRCDATE_${PN}="now"
SRC_URI=svn://repo/path/myhello;proto=https;module=${PN}-${PV}

#for a tarball use:
#SRC_URI=http://server/path/package/${PN}-{$PV}.tar.gz

#include standard rules to build a autoconf/automake package
inherit autotools

Expressions starting by ${} are substituted, this allows you to download a tarball named "mypackage-1.0.tar.gz" for example

Source URIs

bitbake can get the source code for your project from a variety of locations including tarball, subversion and git.

All of this is explained in the bitbake manual, particulary here.

The "build" directory, the one where bitbake will run "configure" et al, MUST have the name template: "PACKAGE_NAME-PACKAGE_VERSION", so

  • tarballs must extract to a subdirectory with this name
  • svn top directory must have this name

It's very important. I spent 3 days blocking on this!

Tell bitbake to build our package

After this, we have to inform bitbake that we have a file for it. I assume a standard Openmoko directory tree with:

$OMDIR
 +- build
 |  +- conf
 |  |  +- local.conf
 |  +- tmp
 +- openmoko
 +- openembedded
 +- sources

etc...

Our personal tree can be everywhere else:

$HOME
 +- $OMDIR
 |  +- ...
 +- MypersonalOpenMokoPackages
    +- packages
    |  +- myhello
    |     +- myhello-1.0.bb
    +- sources
       +- myhello
          +- myhello-1.0
             +- configure
             +- Makefile.am
             +- ...

The only thing to do is edit $OMDIR/build/local.conf to tell bitbake to search our folders for .bb files:

BBFILES+="${HOME}/MypersonalOpenMokoPackages/packages/*/*.bb"

And then, running

$ cd $OMDIR/build
$ bitbake mypackage

will bring all the necessary operations to create a package in $OMDIR/build/tmp/deploy/ipk/armv4t/myhello .....

The following is an example build log:

OE Build Configuration:
BB_VERSION     = "1.6.9"
OE_REVISION    = "<unknown>"
TARGET_ARCH    = "arm"
TARGET_OS      = "linux"
MACHINE        = "fic-gta01"
DISTRO         = "openmoko"
DISTRO_VERSION = ".dev-snapshot-20070723"
TARGET_FPU     = "soft"

NOTE: This BitBake doesn't support revision fetching, falling back to current date
NOTE: preferred version 2.4 of glibc not available
NOTE: preferred version 2.4 of glibc-intermediate not available
NOTE: preferred version 2.4 of glibc not available
NOTE: package myhello-1.0: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_fetch: started
NOTE: Fetch svn://svn.unsads.com/squalyl/om/src/myhello;proto=https;module=myhello-1.0
A    myhello-1.0/configure.ac
A    myhello-1.0/include
A    myhello-1.0/include/config.h.in
A    myhello-1.0/cleanall.sh
A    myhello-1.0/AUTHORS
A    myhello-1.0/bootstrap.sh
A    myhello-1.0/ChangeLog
A    myhello-1.0/src
A    myhello-1.0/src/hello.c
A    myhello-1.0/Makefile.am
A    myhello-1.0/NEWS
A    myhello-1.0/README
Révision 2116 extraite.
NOTE: package myhello-1.0-r0_0_200707231759: task do_fetch: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_unpack: started
NOTE: Unpacking /home/squalyl/OM/sources/myhello-1.0_svn.unsads.com_.squalyl.om.src.myhello__now.tar.gz to /home/squalyl/OM/build/tmp/work/armv4t-linux/myhello-1.0-r0_0_200707231759/
NOTE: package myhello-1.0-r0_0_200707231759: task do_unpack: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_patch: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_patch: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_configure: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_configure: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_compile: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_compile: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_install: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_install: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_package: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_package: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_package_write: started
Packaged contents of myhello-dbg into /home/squalyl/OM/build/tmp/deploy/ipk/armv4t/myhello-dbg_1.0-r0_0_200707231759_armv4t.ipk
Packaged contents of myhello into /home/squalyl/OM/build/tmp/deploy/ipk/armv4t/myhello_1.0-r0_0_200707231759_armv4t.ipk
NOTE: Not creating empty archive for myhello-doc-1.0-r0_0_200707231759
NOTE: Not creating empty archive for myhello-dev-1.0-r0_0_200707231759
NOTE: Not creating empty archive for myhello-locale-1.0-r0_0_200707231759
NOTE: package myhello-1.0-r0_0_200707231759: task do_package_write: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_populate_staging: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_populate_staging: completed
NOTE: package myhello-1.0-r0_0_200707231759: task do_build: started
NOTE: package myhello-1.0-r0_0_200707231759: task do_build: completed
NOTE: package myhello-1.0: completed
NOTE: build 200707231952: completed
Build statistics:
  Attempted builds: 1

Get this package included in the rootfs image

bitbake openmoko-devel-image does not include your new package in the rootfs image.

To do this, use:

bitbake openmoko-devel-image -crebuild

This will actually rebuild the rootfs, including your new packages.