ELBE Overview: Difference between revisions
Created page with "<div id="header"> =Embedded Linux Build Environment (elbe)= Debian based System to generate root-filesystems for embedded devices. ==General Notes== Generally root-filesystem..." |
No edit summary |
||
Line 1: | Line 1: | ||
<div id="header"> | <div id="header"> | ||
The Embedded Linux Build Environment (elbe) is a Debian based System to generate root-filesystems for embedded devices. | |||
__TOC__ | |||
==General Notes== | == General Notes == | ||
Generally root-filesystem creation faces the following requirements: | Generally root-filesystem creation faces the following requirements: | ||
Line 27: | Line 27: | ||
* Updates, addition or package removal is done via debians package-management (apt), which also solves all the dependency problems. | * Updates, addition or package removal is done via debians package-management (apt), which also solves all the dependency problems. | ||
==Installation== | == Installation == | ||
Elbe is shipped as debian package. This package will pull the necessary dependencies. | Elbe is shipped as debian package. This package will pull the necessary dependencies. | ||
Line 35: | Line 35: | ||
<tt>aptitude install elbe</tt> | <tt>aptitude install elbe</tt> | ||
==Overview== | == Overview == | ||
The Elbe system consists of the Program <tt>elbe</tt>, which is completely implemented in Python. Similar to git, it can be called with several sub commands. For example: | The Elbe system consists of the Program <tt>elbe</tt>, which is completely implemented in Python. Similar to git, it can be called with several sub commands. For example: | ||
Line 68: | Line 68: | ||
===Example XML Files=== | === Example XML Files === | ||
A root-filesystem-project is described via an xml file. | A root-filesystem-project is described via an xml file. | ||
Line 150: | Line 150: | ||
The project-directory must not exist before calling this command. It will contain several scripts, files and a Makefile, which controls the creation of the root-filesystem. | The project-directory must not exist before calling this command. It will contain several scripts, files and a Makefile, which controls the creation of the root-filesystem. | ||
===Creation of build system=== | === Creation of build system === | ||
<div class="paragraph"> | <div class="paragraph"> | ||
Line 170: | Line 170: | ||
</div></div></div></div><div class="sect2"> | </div></div></div></div><div class="sect2"> | ||
===Working inside the virtual machine=== | === Working inside the virtual machine === | ||
<div class="paragraph"> | <div class="paragraph"> | ||
Line 190: | Line 190: | ||
</div></div></div></div></div><div class="sect1"> | </div></div></div></div></div><div class="sect1"> | ||
==Changing the subset that is extracted as the root filesystem== | == Changing the subset that is extracted as the root filesystem == | ||
<div class="sectionbody"><div class="paragraph"> | <div class="sectionbody"><div class="paragraph"> | ||
Line 205: | Line 205: | ||
</div><div class="sect2"> | </div><div class="sect2"> | ||
===Finetuning=== | === Finetuning === | ||
<div class="paragraph"> | <div class="paragraph"> | ||
Line 237: | Line 237: | ||
</div></div></div></div><div class="sect2"> | </div></div></div></div><div class="sect2"> | ||
===Useful directories to trim with finetuning=== | === Useful directories to trim with finetuning === | ||
<div class="paragraph"> | <div class="paragraph"> |
Revision as of 15:03, 24 September 2012
The Embedded Linux Build Environment (elbe) is a Debian based System to generate root-filesystems for embedded devices.
General Notes
Generally root-filesystem creation faces the following requirements:
- Creation of root-filesystem for a specific architecture.
- Generation of a development environment for the specific architecture.
Trying to solve this via cross-compilation of the required packages, we face the following problems:
- Many packages are not designed and tested for cross-compilation.
- Big projects consume a significant amount of time to cross compile.
- Dependencies are not resolved automatically. The Engineer is required to manually find out which package versions of dependencies are required.
- Changes and patches to the packages need to be updated to new version, this process makes an update a very time consuming process.
- One must make sure that the same toolchain is used in the development environment, and root-filesystem.
Elbe takes a different approach at solving these Problems:
- Standard Packages are not self compiled. We use the Debian distributions binary packages.
- Own applications are not cross compiled, but built natively on the target architecture in a virtual machine or emulator.
- The root-filesystem is a subset of of the debian system on the virtual development machine. This implicitly ensures, that the same toolchain is used on development machine and target.
- Updates, addition or package removal is done via debians package-management (apt), which also solves all the dependency problems.
Installation
Elbe is shipped as debian package. This package will pull the necessary dependencies.
Configure /etc/apt/sources.list.d appropriately and install it using the following command:
aptitude install elbe
Overview
The Elbe system consists of the Program elbe, which is completely implemented in Python. Similar to git, it can be called with several sub commands. For example:
elbe <command> elbe create --directory /scratch/example example.xml elbe chg_archive example.xml archive.tar.bz2
An elbe Project consist of an xml file, that describes the project. This description includes the kernel and initrd, which is used to boot the emulator. When creating a project, this kernel and initrd can be downloaded using the elbe dl_kernel command.
Then we use the elbe create command to generate the project directory, which contains the virtual hard disk image and Makefile to create the buildimage and root file system. We change to the project directory, and run make.
This will start the root-filesystem generation process:
- The stock debian installer initrd is modified by elbe, so that it doesn’t ask questions during install. And it hooks the image generation into the install process.
- emulator is run with the specified kernel and initrd.
- inside the emulator, Debian installer will install the distribution without asking questions.
- The final step of the installer is to start the image generation.
- Image generation creates a copy of the buildimage. (We have a set of modes for this copy)
- To this copy the fine-tuning commands are applied.
- And the archive is unpacked into the copy.
- According to what is specified in the xml file. The copy is then baked into a tar archive or filesystem image.
This diagram illustrates the Process:
Inside the emulator, the following happens:
Example XML Files
A root-filesystem-project is described via an xml file.
Elbe ships with a set of example xml files.
This xml file describes a simple system in the default full copy mode. The root filesystem is essentially identical to the build system, and it will be saved as nfsroot.tar.gz
<ns0:RootFileSystem xmlns:ns0="https://www.linutronix.de/projects/Elbe" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" created="2009-05-20T08:50:56" revision="4" xsi:schemaLocation="https://www.linutronix.de/projects/Elbe dbsfed.xsd"> <project> <name>ARMexample</name> <version>08.15</version> <mirror> <primary_host>debian.tu-bs.de</primary_host> <primary_path>/debian</primary_path> <primary_proto>http</primary_proto> </mirror> <noauth /> <suite>squeeze</suite> <buildimage> <arch>armel</arch> <size>20G</size> <mem>256</mem> <interpreter>qemu-system-arm</interpreter> <console>ttyAMA0,115200n1</console> <machine>versatilepb</machine> <NIC> <model>smc91c111</model> <MAC>de:ad:be:ef:be:05</MAC> </NIC> <portforwarding> <forward> <proto>tcp</proto> <buildenv>22</buildenv> <host>5022</host> </forward> </portforwarding> <pkg-list> </pkg-list> </buildimage> </project> <target> <hostname>myARM</hostname> <domain>tec.linutronix.de</domain> <passwd>foo</passwd> <console>ttyS0,115200</console> <package> <tar> <name>nfsroot.tar.gz</name> </tar> </package> <finetuning> </finetuning> <pkg-list> <pkg>build-essential</pkg> <pkg>bash</pkg> <pkg>less</pkg> <pkg>git</pkg> <pkg>debhelper</pkg> <pkg>nfs-common</pkg> <pkg>openssh-server</pkg> </pkg-list> </target> </ns0:RootFileSystem>
This xml file is lacking the kernel and initrd tags, which are necessary for a successful build.
The following command will download these, and include them into the xml file:
elbe dl_kernel example.xml
To generate the project directory from the xml file the following command is used:
elbe create --directory /scratch/example example.xml
The project-directory must not exist before calling this command. It will contain several scripts, files and a Makefile, which controls the creation of the root-filesystem.
Creation of build system
Now change to the project directory and run
make
This will start the debian installer in the virtual machine, which will install the build system according to the xml file description. It also generates the archive nfsroot.tar.gz with a copy of the buildimage.
Working inside the virtual machine
After successful installation, the virtual machine can be started with:
make run
or without graphical terminal with:
make run-con
Changing the subset that is extracted as the root filesystem
Elbe has several methods to select the subset that is packaged as the root-filesystem.
- The finetuning section allows to remove, copy or move files before the archive or image is generated.
- The norecommend tag.
- The initial copy has several modes, which allow to extract very small subsets.
- The embedded archive.
Finetuning
The finetuning section allows to copy, move and delete files in the target root-filesystem. Additionally it is possible to run commands. Here is an example finetuning section:
<finetuning> <rm>/usr/share/doc</rm> <mv path="/var">/var_ro</mv> <cp path=/copy/me>/my/name/on/target</mv> </finetuning>
Useful directories to trim with finetuning
This section provides some useful directories that can safely be removed, and which generate big space savings:
<finetuning> <rm>/usr/share/doc</rm> (1) <rm>var/cache/apt/archives/*</rm> (2) <rm>var/cache/apt/pkgcache.bin</rm> <rm>var/cache/apt/srcpkgcache.bin</rm> <rm>var/lib/apt/lists/*_Release</rm> <rm>var/lib/apt/lists/*_Packages</rm> <rm>var/lib/apt/lists/*_Sources</rm> <rm>boot</rm> (3) <rm>lib/modules/2.6.32-5-versatile/</rm> (4) <rm>var/cache/man/*</rm> (5) <rm>opt/elbe</rm> (6) <rm>var/cache/debconf/*</rm> (7)
</finetuning>
(1) |
The doc file are not necessary. |
(2) |
The apt cache can be downloaded, when necessary. |
(3) |
The boot directory contains the kernel used for the VM. On embedded targets this is generally loaded through u-boot. |
(4) |
The kernel modules for the standard debian kernel. |
(5) |
The man page cache |
(6) |
The full copy mode, will also copy elbe onto the root-filesystem. |
(7) |
debconf cache is also not necessary |
Archive
It’s also possible to include an archive into the xml file, that is unpacked into the target root-filesystem after the finetuning step. This archive must be a bzip2 compressed tar (.tar.bz2) and the following command can be used to embed the archive into the xml file:
elbe chg_archive <xmlfile> archive.tar.bz2
This feature is useful to place custom configuration-files into the final image.
Slimming the root filesystem with the different modes
The copying process has several modes. The mode is configured in the target tag. It needs to be the tag before finetuning.
The following commented example illustrates where the mode is configured.
<target> <hostname>example</hostname> <domain>example.com</domain> <passwd>foo</passwd> <package> <tar>nfsroot.tar.gz</tar> </package> <diet /> (1) <norecommend /> (2) <finetuning \> <pkg-list> <pkg>bash</pkg> </pkg-list> </target>
(1) |
The mode is configured at this place, leaving it out, enables the default mode. |
(2) |
norecommend should be placed here. |
norecommend
norecommend disables installation of recommended packages. This is a frequent cause for big root-filesystems. Installed programs will still work, but some functionality requiring external programs might be disabled.
default
The default mode generates a full copy of the build image. This mode is the easiest to use, but its not possible to generate images which have the debian package management removed.
setsel
The setsel mode can be used to generate images which only contain dpkg, and offers a more fine-grained control on which packages are installed.
The recommended usage is as follows:
- Generate an image in the default mode.
- Run the image and use aptitude to purge unwanted packages.
- Maybe even use dpkg to remove apt and aptitude.
- Then generate the list of selected Packages using dpkg --get-selections > selections.list
- Transfer this file to the host System.
- Use elbe setsel <xmlfile> selections.list to import the pkg-list into the xml file.
- Rebuild using setsel mode.
diet
Diet Mode only copies the files that are referenced in the Package management. It resolves the dependencies so that one does not need to reference library packages.
tighten
Tighten mode only extracts the files referenced by the packages in pkg-list. No dependencies are resolved. This mode is intended for compatibility with old xml files, do not use in new elbe projects.
XML Reference
This section is a step by step rundown of the arm example.
The xml file starts with the following header, which should always be the same:
<ns0:RootFileSystem xmlns:ns0="https://www.linutronix.de/projects/Elbe" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" created="2009-05-20T08:50:56" revision="4" xsi:schemaLocation="https://www.linutronix.de/projects/Elbe dbsfed.xsd">
After this Header follows the project description.
<project> <name>ARMexample</name> <version>08.15</version> <mirror> <primary_host>debian.tu-bs.de</primary_host> (1) <primary_path>/debian</primary_path> <primary_proto>http</primary_proto> <url-list> (2) <url> <binary>http://192.168.1.2/custom/pkgs ./</binary> </url> </url-list> </mirror> <noauth /> (3) <suite>squeeze</suite> (4)
(1) |
The mirror tag contains primary_host, primary_path and primary_proto. It describes the debian mirror used. |
(2) |
The url-list can contain additional binary package sources. |
(3) |
The noauth tag can be used to disable authentication, when custom unsigned packages, are supposed to be installed. |
(4) |
The suite also controls which kernel and initrd is downloaded with the elbe dl_kernel` command. It also selects the distribution codename used from the primary mirror. |
The following section describes the virtual machine in which the debian installer will run, and where we want to build our custom packages in.
<buildimage> <arch>armel</arch> (1) <size>20G</size> (2) <mem>256</mem> (3) <interpreter>qemu-system-arm</interpreter> (4) <console>ttyAMA0,115200n1</console> (5) <machine>versatilepb</machine> (6) <NIC> <model>smc91c111</model> <MAC>de:ad:be:ef:be:05</MAC> </NIC> <portforwarding> (7) <forward> <proto>tcp</proto> <buildenv>22</buildenv> <host>5022</host> </forward> </portforwarding> <pkg-list> (8) </pkg-list> </buildimage> </project>
(1) |
arch is the debian architecture of the target. |
(2) |
size is the size of the hard-disk image. |
(3) |
mem controls the amount of RAM that is allocated to the virtual machine. |
(4) |
The interpreter is either qemu-system-<arch> or qemu-kvm. Usage of kvm is usually only possible, when the target is x86. |
(5) |
The console line needs special attention, because it is different on different emulation targets. |
(6) |
The machine tag contains the -machine parameter, that is provided to qemu. |
(7) |
portforwarding of network server ports from the virtual machine to the host. This example forwards the ssh port to port 5022 on the host. |
(8) |
This is the package list of buildimage. |
<target> <hostname>myARM</hostname> (1) <domain>tec.linutronix.de</domain> (2) <passwd>foo</passwd> (3) <console>ttyS0,115200</console> <package> <tar> <name>nfsroot.tar.gz</name> (4) </tar> </package> <finetuning> (5) </finetuning> <pkg-list> (6) <pkg>build-essential</pkg> <pkg>bash</pkg> <pkg>less</pkg> <pkg>git</pkg> <pkg>debhelper</pkg> <pkg>nfs-common</pkg> <pkg>openssh-server</pkg> </pkg-list> </target>
(1) |
The hostname |
(2) |
and domainname |
(3) |
This is the root password of the machine. |
(4) |
This describes, that the resulting rootfilesystem, shall be generated as nfsroot.tar.gz |
(5) |
Empty finetuning section. Format of this section was already explained above. |
(6) |
The Packages list resides here. |
Building ubi images
Elbe also has the ability to generate ubi images, including different partitions and mountpoints. Lets look at an example target section.
<target> <hostname>myARM</hostname> <domain>tec.linutronix.de</domain> <passwd>foo</passwd> <console>ttyS0,115200</console> <images> <mtd> <name>linux.img</name> (1) <nr>0</nr> <size>60MB</size> <ubivg> <label>nand</label> <miniosize>2048</miniosize> (2) <maxlogicaleraseblockcount>1533</maxlogicaleraseblockcount> <logicaleraseblocksize>126976</logicaleraseblocksize> <physicaleraseblocksize>128KiB</physicaleraseblocksize> <ubi> (3) <type>static</type> <label>kernel</label> <id>0</id> <size>4MiB</size> <binary>/boot/vmlinuz-2.6.33.9-rt31</binary> </ubi> <ubi> <type>dynamic</type> <label>rfs</label> <id>1</id> <size>26MiB</size> </ubi> <ubi> <type>dynamic</type> <label>data</label> <id>2</id> <size>30MiB</size> </ubi> </ubivg> </mtd> </images> <fstab> (4) <bylabel> (5) <label>data</label> <mountpoint>/opt</mountpoint> <fs> <type>ubifs</type> <mkfs>-x lzo</mkfs> </fs> <options>rw</options> </bylabel> <bylabel> <label>rfs</label> <mountpoint>/</mountpoint> <fs> <type>ubifs</type> <mkfs>-x lzo</mkfs> </fs> <options>ro</options> </bylabel> <bydev> (6)
</font>'''proc'''<font color="#0000FF">
<mountpoint>/proc</mountpoint> <fs> <type>proc</type> </fs> </bydev> <bydev>
</font>'''sysfs'''<font color="#0000FF">
<mountpoint>/sys</mountpoint> <fs> <type>sysfs</type> </fs> </bydev> <bydev>
</font>'''tmpfs'''<font color="#0000FF">
<mountpoint>/tmp</mountpoint> <fs> <type>tmpfs</type> </fs> <options>size=2M</options> </bydev> <bydev>
</font>'''tmpfs'''<font color="#0000FF">
<mountpoint>/var/log</mountpoint> <fs> <type>tmpfs</type> </fs> <options>size=4M</options> </bydev> <bydev>
</font>'''tmpfs'''<font color="#0000FF">
<mountpoint>/var/run</mountpoint> <fs> <type>tmpfs</type> </fs> <options>size=2M</options> </bydev> </fstab> <finetuning> </finetuning> <pkg-list> <pkg>dash</pkg> </pkg-list> </target>
(1) |
This specifies an mtd image with 60MB size whose filename is linux.img |
(2) |
Parameters for the ubi tools, describing the NAND geometry. |
(3) |
Specification of ubi volumes. |
(4) |
Because we are dealing with a root-filesystem split into more than one partition, we need to specify the fstab, so that everything can be copied into the right place. |
(5) |
Normal partition entries are by label. |
(6) |
Since we now create the target fstab ourselves, we also need to include the entries for /proc, /sys etc. These can be generated with bydev nodes. |