Mokopedia

From Openmoko

(Difference between revisions)
Jump to: navigation, search
(Development Status)
(wiki)
 
(21 intermediate revisions by 9 users not shown)
Line 1: Line 1:
{{todo|Support this project by adding ideas and comments to the discussion page and edit this article}}
+
[[Image:Mokopedia_main.png|thumb|150px|''Mokopedia running on Openmoko'']]
  
Mokopedia makes wikipedia accessible on your openmoko phone even without being online because all articles will reside in a compressed filesystem (squashFS) on your SD Card.
+
{{application|Mokopedia}}
  
You will have to install lighttpd to access a local perl cgi script with the web browser of your choice. This script will offer you a text box to easily search your stored articles.
+
{{ApplicationBox|
 +
Name=[[Mokopedia]]|
 +
Description=Read Wikipedia offline from small optimized dumps|
 +
Screenshot=Mokopedia_main.png|
 +
Homepage=http://mokopedia.mister-muffin.de/|
 +
TestedOn=-|
 +
PackageName=no package yet
 +
}}
  
==Feature Ideas==
+
This project aims to provide software to create and use small size wikipedia dumps for embedded Linux devices.
Please feel free to add anything that comes to your mind.
+
*it will have no pictures - not enough room on any storage media
+
*tex forumlas could be generated with a local LaTeX install but this may be too big for the phone
+
*all wikipedia versions except the english one will fit on a 1GB SDCard (tested with the second largest wikipedia (german)) - compressability tests with en-wiki in progress
+
*support for stripped down wikipedia versions (100MB, 500MB)
+
  
==Development Status==
+
All Wikipedia pages will be stripped down html files compressed in a squashfs lzma image.
  
*I finished writing the perl and bash scripts for optimizing your wikipedia html dump. To obtain the dumps goto http://static.wikipedia.org. The scripts are in svn on the mokopedia project site http://projects.openmoko.org/plugins/scmsvn/viewcvs.php/?root=mokopedia
+
For more info visit the official website: http://mokopedia.mister-muffin.de/
*lighttpd is working on openmoko as a light webserver without too much overhead
+
*cgi-script for searching and accessing articles is finished
+
You can download the current code at http://projects.openmoko.org/projects/mokopedia/
+
You can view a little demo here: http://www.rabenfrost.net/mokopedia_demo.ogg
+
  
==Requirements==
+
Join our irc channel on freenode: #mokopedia.
===On the target (NEO)===
+
*lighttpd
+
*changes in /etc/lighttpd.conf
+
<pre>enable server-module "mod_cgi", "mod_rewrite", "mod_redirect", "mod_access"</pre>
+
<pre>#### CGI module
+
cgi.assign                  = ( ".pl"  => "/usr/bin/perl",
+
                                ".py"  => "/usr/bin/python",
+
                                ".cgi" => "/usr/bin/perl" )
+
</pre>
+
<pre>#### url handling modules (rewrite, redirect, access)
+
url.rewrite-once        = ( "^/wiki/(.*)"      => "/wiki/search.py?q=$1" )
+
</pre>
+
*python-core
+
*squashfs-tools (optional)
+
*kernel-module-squashfs
+
*kernel-module-loop
+
*[[Web Browser]]
+
  
===On the host===
 
To build the squashFS image on your PC instead of downloading ready-to-use-images you additionally need mksquashfs-tools
 
  
==Credits==
+
It's the goal to make ALL the articles (except images) of Wikipedia (including formatting, linking, tables, formulas) fit on very little space.
A big thank to the following people:
+
* kahuna for bringing me back to the idea of mokopedia as served html to a browser of your choice
+
* fgau for trying out everything I wrote on his neo
+
* thomasg for suggesting the use of a compressed file system
+
* Marcel_M for his great repository with squashfs and loop kernel modules http://celtune.morb-design.com/ipk/fic-gta01/
+
* everyone on #neo1973-germany
+
All you guys helped me a lot!!
+
  
[[Category:Add-on data]]
+
Tests show that the German Wikipedia will be around 1.3GB and the English Wikipedia around 4GB so with flash storage prices dropping I will not bother removing certain articles - the dumps will include everything.
[[Category:Ideas]]
+
 
[[Category:Unimplemented]]
+
Some installation instructions can be found at
 +
http://rabenfrost.net/celtune/1-NEWS-WARNING-INFO (english) or
 +
http://seeseekey.net/blog/index.php?/archives/93-Mokopedia-einrichten.html (german).
 +
 
 +
 
 +
= Development status =
 +
 
 +
Some people are already running the new mokopedia on their freerunners but i wont announce any official info until enwiki isnt done so up to now there are only dewiki and others that are possible to use.
 +
 
 +
I you are interested to get it running on you freerunner join our irc channel and i will explain the few steps you have to take.
 +
 
 +
In the near feature there will be ipkgs for everything.
 +
 
 +
= To everybody that was familiar with the old implementation =
 +
 
 +
There is no lighty needed anymore - i now implement the server in a simple python script.
 +
 
 +
Now we have FULLTEXT search - yeah you hear right. a fulltext search through 6.8 GB of dewiki is possible in mere 20 seconds on the neo1973.
 +
 
 +
Now there are also a lot of things already implemented as ipkg - only the server is left but will follow soon.
 +
 
 +
[[Category:Applications]]

Latest revision as of 16:31, 19 July 2009

Mokopedia running on Openmoko

Mokopedia is one of the applications that runs on the Openmoko Phones. For a list of all applications, visit Applications

Mokopedia main.png Mokopedia

Read Wikipedia offline from small optimized dumps


Homepage: http://mokopedia.mister-muffin.de/
Package: no package yet
Tested on: -


This project aims to provide software to create and use small size wikipedia dumps for embedded Linux devices.

All Wikipedia pages will be stripped down html files compressed in a squashfs lzma image.

For more info visit the official website: http://mokopedia.mister-muffin.de/

Join our irc channel on freenode: #mokopedia.


It's the goal to make ALL the articles (except images) of Wikipedia (including formatting, linking, tables, formulas) fit on very little space.

Tests show that the German Wikipedia will be around 1.3GB and the English Wikipedia around 4GB so with flash storage prices dropping I will not bother removing certain articles - the dumps will include everything.

Some installation instructions can be found at http://rabenfrost.net/celtune/1-NEWS-WARNING-INFO (english) or http://seeseekey.net/blog/index.php?/archives/93-Mokopedia-einrichten.html (german).


[edit] Development status

Some people are already running the new mokopedia on their freerunners but i wont announce any official info until enwiki isnt done so up to now there are only dewiki and others that are possible to use.

I you are interested to get it running on you freerunner join our irc channel and i will explain the few steps you have to take.

In the near feature there will be ipkgs for everything.

[edit] To everybody that was familiar with the old implementation

There is no lighty needed anymore - i now implement the server in a simple python script.

Now we have FULLTEXT search - yeah you hear right. a fulltext search through 6.8 GB of dewiki is possible in mere 20 seconds on the neo1973.

Now there are also a lot of things already implemented as ipkg - only the server is left but will follow soon.

Personal tools
TODO: Support this project by adding ideas and comments to the discussion page and edit this article (See: To-Do List)

Mokopedia makes wikipedia accessible on your openmoko phone even without being online because all articles will reside in a compressed filesystem (squashFS) on your SD Card.

You will have to install lighttpd to access a local perl cgi script with the web browser of your choice. This script will offer you a text box to easily search your stored articles.

Feature Ideas

Please feel free to add anything that comes to your mind.

  • it will have no pictures - not enough room on any storage media
  • tex forumlas could be generated with a local LaTeX install but this may be too big for the phone
  • all wikipedia versions except the english one will fit on a 1GB SDCard (tested with the second largest wikipedia (german)) - compressability tests with en-wiki in progress
  • support for stripped down wikipedia versions (100MB, 500MB)

Development Status

You can download the current code at http://projects.openmoko.org/projects/mokopedia/ You can view a little demo here: http://www.rabenfrost.net/mokopedia_demo.ogg

Requirements

On the target (NEO)

  • lighttpd
  • changes in /etc/lighttpd.conf
enable server-module "mod_cgi", "mod_rewrite", "mod_redirect", "mod_access"
#### CGI module
cgi.assign                  = ( ".pl"  => "/usr/bin/perl",
                                ".py"  => "/usr/bin/python",
                                ".cgi" => "/usr/bin/perl" )
#### url handling modules (rewrite, redirect, access)
url.rewrite-once        = ( "^/wiki/(.*)"      => "/wiki/search.py?q=$1" )
  • python-core
  • squashfs-tools (optional)
  • kernel-module-squashfs
  • kernel-module-loop
  • Web Browser

On the host

To build the squashFS image on your PC instead of downloading ready-to-use-images you additionally need mksquashfs-tools

Credits

A big thank to the following people:

  • kahuna for bringing me back to the idea of mokopedia as served html to a browser of your choice
  • fgau for trying out everything I wrote on his neo
  • thomasg for suggesting the use of a compressed file system
  • Marcel_M for his great repository with squashfs and loop kernel modules http://celtune.morb-design.com/ipk/fic-gta01/
  • everyone on #neo1973-germany

All you guys helped me a lot!!