<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Jan-Piet Mens &#187; CLI</title>
	<atom:link href="http://blog.fupps.com/category/cli/feed/" rel="self" type="application/rss+xml" />
	<link>http://blog.fupps.com</link>
	<description>my animals and other friends</description>
	<lastBuildDate>Sat, 31 Jul 2010 14:34:04 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.8.6</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>On telephones and little black books or directories</title>
		<link>http://blog.fupps.com/2010/07/29/on-telephones-and-little-black-books-or-directories/</link>
		<comments>http://blog.fupps.com/2010/07/29/on-telephones-and-little-black-books-or-directories/#comments</comments>
		<pubDate>Thu, 29 Jul 2010 12:09:56 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[CLI]]></category>
		<category><![CDATA[Database]]></category>
		<category><![CDATA[LDAP]]></category>
		<category><![CDATA[Mobile]]></category>
		<category><![CDATA[Voip]]></category>
		<category><![CDATA[directory]]></category>
		<category><![CDATA[telephony]]></category>
		<category><![CDATA[white-pages]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3502</guid>
		<description><![CDATA[Remember this? For those too young to remember, it is a rotary dial phone, and I assume many young people wouldn&#039;t even know how to use it. To dial a number, you looked up the phone number in a telephone directory (or your little black book), picked up the handset, pushed your finger into the [...]]]></description>
			<content:encoded><![CDATA[<p>Remember this? For those too young to remember, it is a <a href="http://en.wikipedia.org/wiki/Rotary_dial">rotary dial</a> phone, and I assume many young people wouldn&#039;t even know how to use it. To dial a number, you looked up the phone number in a telephone directory (or your little black book), picked up the handset, pushed your finger into the hole corresponding to the digit you wanted to dial, and moved said finger clockwise until it bumped against the metal thingy. After pulling out your finger, the spring-loaded rotary dialer moved back to its original position, and you then proceeded to dial the next digit until you&#039;d done them all. Simple.</p>
<p><a href="http://www.flickr.com/photos/knirsch_knarz/2212110215/"><img src="http://blog.fupps.com/wp-content/media/2010/ZZ44C5BCFA.jpg" width="500" height="334" alt="" /></a></p>
<p>To place a call you needed a telephone number, of course, just like you need today. Nothing has changed in that respect. I remember knowing most of my friends numbers by heart. Those I didn&#039;t know, I had in my little black book, or they were scribbled on a piece of paper somewhere. My friends had one telephone &#8212; the apparatus in their (parents&#039;) homes &#8212; and thus exactly one phone number.</p>
<p>Thirty years later the situation is quite different. Nobody I know has only one number. Everybody and his uncle has at least an additional mobile phone. And an office number. And perhaps a SIP number. And the greatest problem of all is that I cannot for the life of me remember any of these phone numbers any more, which is due mainly to the fact that numbers change when people change mobile operators (some don&#039;t care to retain their existing cell phone number), and numbers change when people relocate. (There are <a href="http://de.wikipedia.org/wiki/Persönliche_Rufnummer">vanity numbers in Germany</a>, but these are quite expensive.)</p>
<p>What has also changed, at least for me, is the little black book. Aside from the printout I keep in case of disaster, I store my phone numbers electronically. In Apple&#039;s Addressbook app which syncs over nicely to iPhones and devices which have <a href="http://en.wikipedia.org/wiki/ISync">iSync</a> support.</p>
<p>And what about other phones in the house? If I look around me, I find we have quite a few phones in this household. One PBX with three DECT phones connected to it. An IP phone in my office. Two iPhones. One BlackBerry. (And an innumerable number of older mobile phones in boxes in the cellar, but those don&#039;t count because they aren&#039;t used any longer.) All these devices also have built-in directories: the DECT models have their own but they can list numbers on the PBX, though a bit cumbersome. The IP phone has a telephone book. And LDAP access. And XML. </p>
<p>If I weren&#039;t so stingy, I&#039;d get a cell phone flat rate and just use that. But I am stingy so I tend to use a land line phone to call land lines, so I have to choose the outgoing phone. As such I need my directory on all phones I use.</p>
<p>I&#039;ve solved the problem by throwing large amounts of software at the hardware. Basically it&#039;s a mixture of getting my Addressbook into LDAP (<a href="http://blog.fupps.com/2009/09/11/abxldap-gets-better-and-better/">ABxLDAP</a>), massaging that <a href="http://blog.fupps.com/2010/06/25/upload-phonebook-to-a-fritzbox-from-the-command-line/">into my Fritz!Box</a>, and sucking some of the directory entries out to create a directory for the IP phone. Does it work? Yes. Is it simple? No. Why is it such a mess? Because there are far too many standards: standards for directory systems (LDAP), standards for SIP phones (<a href="http://en.wikipedia.org/wiki/XML_Configuration_Access_Protocol">XCAP</a>), standards for DECT phones, and so on, and so forth.</p>
<p>Who knows: some day I may revert to the little black book and &#8230;</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ0D5D16A4.jpg" width="320" height="480" alt="" /></p>
<p> the rotary dialer. On the iPhone.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/07/29/on-telephones-and-little-black-books-or-directories/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>The Samhain file integrity monitor in iX 08/2010</title>
		<link>http://blog.fupps.com/2010/07/08/the-samhain-file-integrity-monitor-in-ix-082010/</link>
		<comments>http://blog.fupps.com/2010/07/08/the-samhain-file-integrity-monitor-in-ix-082010/#comments</comments>
		<pubDate>Thu, 08 Jul 2010 07:52:58 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[CLI]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[MacOSX]]></category>
		<category><![CDATA[Security]]></category>
		<category><![CDATA[samhain]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3493</guid>
		<description><![CDATA[Martin introduced me to the Samhain file integrity monitor a while back &#8212; an impressive bit of software with which I&#039;ve spent some quality time. I was much impressed with the features of the software, and I thought I&#039;d spread the word a bit, so I wrote a short introductory article to it. The article [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://binblog.info/">Martin</a> introduced me to the <a href="http://www.la-samhna.de/samhain/">Samhain file integrity monitor</a> a while back &#8212; an impressive bit of software with which I&#039;ve spent some quality time. I was much impressed with the features of the software, and I thought I&#039;d spread the word a bit, so I wrote a short introductory article to it. The article will appear on page 127 in <a href="http://ix.de/">iX</a> 8/2010.</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ4420591C.jpg" width="228" height="319" alt="" /></p>
<p>Thanks, Martin, and thank you Samhain for lending me a copy of <a href="http://www.la-samhna.de/beltane/index.html">Beltane II</a>. Oh, and thanks to <a href="http://ix.de">iX</a>&#039; <b>excellent</b> layouter for putting my photo up there. <img src='http://blog.fupps.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/07/08/the-samhain-file-integrity-monitor-in-ix-082010/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Upload phonebook to a Fritz!Box from the command line</title>
		<link>http://blog.fupps.com/2010/06/25/upload-phonebook-to-a-fritzbox-from-the-command-line/</link>
		<comments>http://blog.fupps.com/2010/06/25/upload-phonebook-to-a-fritzbox-from-the-command-line/#comments</comments>
		<pubDate>Fri, 25 Jun 2010 07:52:54 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[CLI]]></category>
		<category><![CDATA[Hardware]]></category>
		<category><![CDATA[Software]]></category>
		<category><![CDATA[Voip]]></category>
		<category><![CDATA[fritzbox]]></category>
		<category><![CDATA[phonebook]]></category>
		<category><![CDATA[telefonbuch]]></category>
		<category><![CDATA[telephony]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3488</guid>
		<description><![CDATA[A FRITZ!Box Fon WLAN 7270 has a built-in telephone directory which is maintained via a Web interface, from which I can create, edit, remove and backup telephone directory entries used by attached DECT phones.

That is all fine and dandy, but I want to be able to populate that directory from existing an existing database, and [...]]]></description>
			<content:encoded><![CDATA[<p>A <a href="http://www.avm.de/de/Produkte/FRITZBox/FRITZ_Box_Fon_WLAN_7270/">FRITZ!Box Fon WLAN 7270</a> has a built-in telephone directory which is maintained via a Web interface, from which I can create, edit, remove and backup telephone directory entries used by attached DECT phones.</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ407CD647.png" width="500" height="354" alt="" /></p>
<p>That is all fine and dandy, but I want to be able to populate that directory from existing an existing database, and I don&#039;t want to fumble with import stuff from the Web interface.</p>
<p>It turns out that is easier said than done, and it took a while to find out how to do that. I finally managed, thanks to work done by Christoph Linder on a Thunderbird add-on called <a href="https://addons.mozilla.org/en-US/thunderbird/addon/75914/">Thunder!Box</a>.</p>
<blockquote><p>Thunder!Box allows transferring contacts from your Thunderbird-Addressbook to the phonebook of a connected AVM FRITZ!Box FON router and back. You may also use your address book to initiate calls.</p></blockquote>
<p>A Fritz!Box has a very special <a href="http://sourceforge.net/apps/mediawiki/thunderbox/index.php?title=Tech:Docs">authentication handshake</a>, which makes the implementation rather secure, but tricky to access. (AVM have published a <a href="http://www.avm.de/de/Extern/Technical_Note_Session_ID.pdf">technical note</a> on this, in German.)</p>
<p>Anyways, to cut a long story short, I&#039;ve implemented what I call <tt>fritzuploader</tt> &#8212; a small command-line tool which uploads an XML file containing a telephone directory to overwrite that in the Fritz!Box&#039; flash.</p>
<p>I&#039;ve been using this quite a bit and haven&#039;t experienced any problems with it, but note it might quite well fry your Fritz!Box. Further note, that you should probably use this a bit sparingly, as the number of writes to <a href="http://en.wikipedia.org/wiki/Flash_memory">flash memory</a> is finite.</p>
<p>The <a href="http://github.com/jpmens/fritzuploader">Github repository for fritzuploader</a> contains a sample <tt>phonebook.csv</tt> you can use as a starting point, the tiny <tt>csv2xml</tt> converter to create the <tt>phonebook.xml</tt> file which is then finally uploaded to the Fritz!Box with <tt>fritzuploader</tt>. At this point the current telephone directory on the Fritz!Box is overwritten, so you&#039;ll want to back that up first via the Web interface.</p>
<p>The resulting import looks like this:</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ56EF4174.png" width="500" height="354" alt="" /></p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ1A2E19FF.png" width="500" height="354" alt="" /></p>
<p>I&#039;d be interested to hear if you are successful with other AVM Fritz!Box models.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/06/25/upload-phonebook-to-a-fritzbox-from-the-command-line/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Truly scrumptious bookmarks in CouchDB</title>
		<link>http://blog.fupps.com/2010/05/25/truly-scrumptious-bookmarks-in-couchdb/</link>
		<comments>http://blog.fupps.com/2010/05/25/truly-scrumptious-bookmarks-in-couchdb/#comments</comments>
		<pubDate>Tue, 25 May 2010 18:39:06 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[Backup]]></category>
		<category><![CDATA[CLI]]></category>
		<category><![CDATA[Database]]></category>
		<category><![CDATA[NoSQL]]></category>
		<category><![CDATA[bookmarks]]></category>
		<category><![CDATA[CouchDB]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3470</guid>
		<description><![CDATA[Services in the cloud are great, and I make some us of them when I find them appropriate or particularly useful. One of the cloud services I&#039;ve used with some frequency is the del.icio.us bookmarking service, because I can access the bookmarks from any location. Having my data in &#034;the cloud&#034; means it isn&#039;t really [...]]]></description>
			<content:encoded><![CDATA[<p>Services in the cloud are great, and I make some us of them when I find them appropriate or particularly useful. One of the cloud services I&#039;ve used with some frequency is the del.icio.us bookmarking service, because I can access the bookmarks from any location. Having my data in &#034;the cloud&#034; means it isn&#039;t really mine, so I attempt to retain a copy of it to be on the safe side. I certainly don&#039;t know when a cloud service will decide to go &#034;puff&#034; and take my data along with it. In order to accomplish this, I&#039;m implementing a trivial bookmarking application a la del.icio.us, with a CouchDB back-end. Now, before I continue a disclaimer: I&#039;m not a developer, so there are plenty of things I won&#039;t get right, such as my lousy CSS, poor JavaScript, etc. Feel free to provide a better solution, but I hope you&#039;ll find this as interesting as I do.</p>
<p>Let&#039;s first have a look at the final result. You can follow along by installing a  version of <a href="http://github.com/couchapp/couchapp">couchapp</a> and <a href="http://couchdb.apache.org/">CouchDB</a> of course. Then go to the <a href="http://github.com/jpmens/scrumptious">program&#039;s Github repository</a>, clone the app, push it into CouchDB and launch the URL:</p>
<pre class="brush: bash;">
git clone http://github.com/jpmens/scrumptious.git
cd scrumptious
couchapp push http://localhost:5984/scrumptious
</pre>
<p>By pointing your Web browser at the URL given you by <code>couchapp</code>, you&#039;ll be redirected to a <code>show</code> function, from which you drag the generated bookmarklet URL to your browser bar: (I have to &#034;generate&#034; the bookmarklet for you because it contains the URL to your CouchDB instance; the show function determines the full path and creates the bookmarklet containing that.)</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ16E78FA0.png" width="500" height="360" alt="" /></p>
<p>Go ahead and drag the link into your bookmarks bar: mine is shown as the little recycle symbol. When you&#039;ve done that, click on view all bookmarks, which will take you to the paginated list of the bookmarks stored in your CouchDB. (Yes, there are three there already. <img src='http://blog.fupps.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ5F17FE8E.png" width="500" height="360" alt="" /></p>
<p>Clicking on the greater symbol takes you to a page describing the bookmark. Click on the bookmark title to take you to the URL saved with the bookmark. Within the description page, click on the &#034;delete bookmark&#034; to kill the document.</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ2131F0D4.png" width="500" height="360" alt="" /></p>
<p>When visiting a site you wish to bookmark, press on the bookmarklet you saved in your bookmark bar when on the page. The text you select on the page becomes the bookmark&#039;s description, and you&#039;ll be propted to enter a list of (white space or comma-separated) tags for the page.</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ7625C9D4.png" width="500" height="360" alt="" /></p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ5BD70F42.png" width="400" height="194" alt="" /></p>
<p>The bookmark is saved into CouchDB with a shortish <code>_id</code> (swiped from Jan&#039;s <a href="http://github.com/janl/io">io</a>), whereupon you get a confirmation page. (I&#039;d have liked to have the update handler return a 302 redirection to the browser, but <a href="https://issues.apache.org/jira/browse/COUCHDB-648">that isn&#039;t possible with CouchDB 0.11</a>.)</p>
<p>The resulting bookmark looks like this:</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ6AABE099.png" width="500" height="360" alt="" /></p>
<p>One URL you may like to see is produced by a couchapp list: <code>/_design/app/_list/bookmarksfile/all</code> produces a <code>bookmarks.html</code> file you can import into some Web browsers. Now that we&#039;ve seen the app at work, let me discuss its innards a bit. </p>
<h3>Standalone applications</h3>
<p>Couchapps are <a href="http://jchrisa.net/drl/_design/sofa/_show/post/standalone_applications_with_co">standalone applications</a>. They consist of JavaScript and JSON documents stored in a CouchDB design document:</p>
<blockquote>
<p>As a proof of concept, look no further than CouchDB&#039;s built-in administrative interface. Futon, as it&#039;s called, is just a collection of HTML, CSS, and Javascript files, and it is a fully functional database browser and JSON document editor, as well as the runner for CouchDB&#039;s functional test suite. The test suite is written in Javascript and runs from the browser. What else could more strongly indicate that Ajax and CouchDB go hand in hand?</p>
</blockquote>
<p>I find that there is little documentation about couchapps, which is fair enough if you believe in &#034;read the source&#034; strategies, so I&#039;ve created this trivial couchapp, and I&#039;m going to point out the things I find most interesting, or situations where I had a hard time understanding what happens in couchapps.</p>
<h3>del.icio.us</h3>
<p>Del.icio.us is a service which lets me submit, describe, and tag URLs I find interesting. I can share these with the rest of the world or mark URLs as being private. They have a sexy Web interface with which I can create and manage my bookmarks, and there exist dozens of bookmarklets and clients for managing them. They also have an API.</p>
<p>The <a href="http://delicious.com/help/api">del.icio.us API</a> allows me to submit an authenticated HTTP GET request with URL-encoded parameters containing the data for the bookmark I want to add to my collection. As an example, I can point any HTTP client (e.g. my Web browser) at a URL like</p>
<pre><code>https://api.del.icio.us/v1/posts/add?
  url=...&amp;description=...&amp;tags=....
</code></pre>
<p>and create a new bookmark.</p>
<p>I thought it would be trivial to implement the del.icio.us API on top of a CouchDB database, which would have had the great advantage that I could use any of the dozens of del.icio.us clients to drop bookmarks into my CouchDB. Unfortunately, the guys over at del.icio.us decided to implement a not-so-REST-y interface to their service; to add a new bookmark, del.icio.us uses an HTTP GET request instead of a POST or a PUT request.</p>
<p>Without some sort of middleware which converts a GET request to a POST, there is no way to add a document to a CouchDB database &#8212; it must be either a POST or a PUT request. I can call an update handler, but update handlers in CouchDB support POST and PUT only, and <a href="http://jan.prima.de/">Jan Lehnardt</a> quickly pointed out in no uncertain terms that I was barking up the wrong tree, trying to GET into an update handler. He was right in pointing out on <code>#couchdb</code> that I was trying to circumvent REST principles. (And I&#039;m grateful to him for having put me right.)</p>
<p>So, in order to POST to a CouchDB, I had to create a browser bookmarklet which does a POST request.  There is a huge amount of information on <a href="http://betterexplained.com/articles/how-to-make-a-bookmarklet-for-your-web-application/">how to make a bookmarklet</a>, but I finally swiped code from <a href="http://ptaff.ca/smart/?lang=en_CA">Smart Bookmarks and Bookmarklets</a>, which provided just what I was looking for: one click and the JavaScript submits a POST request to my update handler.</p>
<h3>Tools used</h3>
<p>Editing JavaScript via the Futon interface is a no go; you&#039;ll go crazy attempting it. The only sane method is via the <a href="http://github.com/couchapp/couchapp">couchapp</a> program which does a quite a bit of magic on the fly. What does <a href="http://github.com/couchapp/couchapp">couchapp</a> actually do? Well, these are some of the things I&#039;ve found so far.</p>
<ul>
<li>
<p>Couchapp provides helper macros for getting code into your show and list functions. If you look at the file <code>list/ls.js</code>, you&#039;ll see a JavaScript comment that looks like this:</p>
<pre><code>// !json templates.index
</code></pre>
<p>That is parsed by the <code>couchapp</code> utility when you invoke a <code>push</code> and the content of the file <code>templates/index.html</code> is inserted into our <code>ls.js</code> JavaScript, much like an <code>#include</code> in C, for example. Go ahead and view the <code>app</code> design document with Futon. The file&#039;s content becomes a JSON object called <code>templates</code> with an name <code>index</code> and as value, the content of the file. Neat.</p>
</li>
<li>
<p>A similar macro facility called <code>!code</code> inserts JavaScript code from the specified path, which is relative to the top-level couchapp directory. (See example in <code>updates/post.js</code>.) When <a href="http://github.com/couchapp/couchapp">couchapp</a> is run, it replaces the line with the content of the JavaScript file.</p>
</li>
<li>
<p>Files in the <code>_attachments</code> directory are dumped into your <a href="http://couchdb.apache.org/">CouchDB</a> database as attachments. You&#039;ll note I&#039;ve stored the CSS for the HTML in such an attachment, for example.</p>
</li>
<li>
<p>I can have <code>couchapp</code> add documents to the CouchDB database; see three examples in the <code>_docs</code> directory. (This is why you get three bookmarks in your database when you push my simple app into your CouchDB.)</p>
</li>
<li>
<p>JavaScript files in the <code>lib/</code> file-system hierarchy are added to the design document, from which you can load them by requiring them, as in <code>var mustache = require("lib/mustache");</code>. Omit the <code>.js</code> extension when doing so, or CouchDB complains about infinite recursion or something similar.</p>
</li>
</ul>
<p>Couchapp contains a lot more magic: invoke `couchapp generate whatever` and explore the files created for you in the file system!</p>
<p>You&#039;ll note I make some use of <a href="http://github.com/janl/mustache.js">mustache.js</a> &#8212; an indespensible templating system which thankfully allows me to keep code and HTML separate from eachother. Jan recently <a href="http://blog.couch.io/post/622014913/mustache-js">wrote a bit more about mustache.js</a>, and the <a href="http://github.com/janl/mustache.js">project&#039;s README</a> contains the rest of what you need to get started.</p>
<h3>Import from del.icio.us</h3>
<p>You can import your existing del.icio.us bookmarks rather easily by grabbing an XML stream containing your own bookmarks and dumping that into a file <tt>mine.xml</tt>:</p>
<pre class="brush: bash;">
curl -sf -u username:password -o mine.xml https://api.del.icio.us/v1/posts/all
</pre>
<p>Let&#039;s massage that XML with the following small program (`delicious2couchdb.pl`) by &#034;translating&#034; the XML entities to corresponding JSON objects ready for CouchDB:</p>
<pre class="brush: perl;">
#!/usr/bin/perl

use strict;
use XML::Simple;
use JSON;

my $json = new JSON;
my $ref = XMLin('mine.xml', ForceArray =&gt; 1);
my @docs;

foreach my $post (@{$ref-&gt;{post}}) {
	my $book = {
		type	=&gt; 'bookmark',
		url	=&gt; $post-&gt;{href},
		date	=&gt; $post-&gt;{time},
		tags	=&gt; [ split(/\s+/, $post-&gt;{tag}) ],
		title	=&gt; $post-&gt;{description},
		description	=&gt; $post-&gt;{extended},
		hash	=&gt; $post-&gt;{hash},		# Keep for reference
		imported =&gt; JSON::true,
		};
	push(@docs, $book);

}

my $all = {
	docs =&gt; [ @docs ],
};
print $json-&gt;encode($all);
</pre>
<p>The format produced by <tt>delicious2couchdb.pl</tt> is ready to be POSTed into CouchDB&#039;s <a href="http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API">_bulk_docs API</a>:</p>
<pre class="brush: bash;">
./delicious2couchdb.pl |
  curl -X POST http://localhost:5984/scrumptious/_bulk_docs -d@-
</pre>
<p>The funny-looking <tt>@-</tt> is for curl: data is read from the named file, and as the file is a single dash, curl reads the data from standard input.</p>
<h3>Further reading</h3>
<p>There is a lot more to discover here, and I&#039;d like to point out some neat Couch Apps:</p>
<ul>
<li>Jan Lehnardt&#039;s <a href="http://github.com/janl/io">io</a> (mentioned above already) is a CouchDB-based URL shortener.</li>
<li>J Chris A. created <a href="">sofa</a>, a CouchDB-based blog, which is <a href="http://jchrisa.net/drl/_design/sofa/_list/index/recent-posts?descending=true&amp;limit=5">what he uses</a>.</li>
<li><a href="http://github.com/jchris/taskr">Taskr</a> is a task-tracking app by J Chris A.</li>
<li>More at <a href="http://couchapp.org/">CouchApp.org</a></li>
</ul>
<p>The chapter on <a href="http://books.couchdb.org/relax/example-app/standalone-applications">Standalone Applications</a> in <a href="http://books.couchdb.org/relax/">CouchDB: The Definitive Guide</a> talks a bit about Couch Apps, and I don&#039;t want you to miss my <a href="http://blog.fupps.com/2010/04/20/the-antepenultimate-couchdb-reference-card/">CouchDB Reference Card</a></p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/05/25/truly-scrumptious-bookmarks-in-couchdb/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Update notifications in CouchDB: tweeting urgent documents</title>
		<link>http://blog.fupps.com/2010/05/10/update-notifications-in-couchdb-tweeting-urgent-documents/</link>
		<comments>http://blog.fupps.com/2010/05/10/update-notifications-in-couchdb-tweeting-urgent-documents/#comments</comments>
		<pubDate>Mon, 10 May 2010 08:40:25 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[CLI]]></category>
		<category><![CDATA[Database]]></category>
		<category><![CDATA[NoSQL]]></category>
		<category><![CDATA[CouchDB]]></category>
		<category><![CDATA[node.js]]></category>
		<category><![CDATA[twitter]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3455</guid>
		<description><![CDATA[Update notifications sent by a database allow me to react to changes submitted to it in almost real-time without having to continuously (or periodically) poll said database. Note, that update notifications are not the same thing as database triggers as known in the relational database world: update notifications are submitted after a database modification has [...]]]></description>
			<content:encoded><![CDATA[<p>Update notifications sent by a database allow me to react to changes submitted to it in almost real-time without having to continuously (or periodically) poll said database. Note, that update notifications are not the same thing as <a href="http://en.wikipedia.org/wiki/Database_trigger">database triggers</a> as known in the relational database world: update notifications are submitted after a database modification has occurred &#8212; there is no way to avoid the update. These notifications are typically consumed by an external process which then somehow reacts to the update. <a href="http://couchdb.apache.org/">CouchDB</a> has support for two types of update notifications (of which I&#039;m aware):</p>
<ol>
<li>Update notifications powered by an external process, and</li>
<li>Change notifications using the changes API.</li>
</ol>
<h3>External process</h3>
<p><a href="http://couchdb.apache.org/">CouchDB</a> uses <a href="http://wiki.apache.org/couchdb/ExternalProcesses">external processes</a> which read an update notification on <em>stdin</em> whenever a database is modified. To configure one or more such processes, edit your CouchDB&#039;s <code>local.ini</code> to read</p>
<pre><code>[update_notification]
jpnoti=/usr/bin/couchjp
</code></pre>
<p>This process receives a JSON string as a notification which includes the name of the database that was just modified.</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/notif-proc.png" width="320" height="106" alt="" /></p>
<p>For example, whenever my <em>twitter</em> database changes, the <code>couchjp</code> program receives the following line on <em>stdin</em>:</p>
<pre><code>{"type":"updated","db":"twitter"}
</code></pre>
<p>Your program could then retrieve the database name and go and look for what has changed. Or for example, the program could automatically <a href="http://blog.idearise.com/2008/09/08/couchdb-081/">launch replication</a>, if you don&#039;t want to use <code>continuous_replication</code> as exists in <a href="http://couchdb.apache.org/">CouchDB</a> 0.11.</p>
<h3>Change notifications</h3>
<p><a href="http://couchdb.apache.org/">CouchDB</a> has the so-called <tt>_changes</tt> database-API which your application can use to get <a href="http://en.wikipedia.org/wiki/Comet_(programming)">Comet-style</a> notification of <a href="http://books.couchdb.org/relax/reference/change-notifications">changes</a>, be it by your application polling the database to query if anything has changed since a particular time or by CouchDB providing your application with a continuous stream of changes. I&#039;m not going to delve into this API much, because it is <a href="http://books.couchdb.org/relax/reference/change-notifications">very well documented</a> in the book <a href="http://books.couchdb.org/relax/">CouchDB: The Definitive Guide</a>. In essence, you use an HTTP <tt>GET</tt> request to pull in a list of changed documents.</p>
<p>What I am going to dig into though, is how to tap that API with a JavaScript interface created by Mikeal Rogers on top of <a href="http://en.wikipedia.org/wiki/Nodejs">node.js</a>. Now, if you are like I, when you hear JavaScript, you probably think of those pesky <code>alert('hello');</code> or <code>document.write('whatever');</code> on Web pages here and there. Well, times change: JavaScript has become a <a href="http://en.wikipedia.org/wiki/Server-side_JavaScript">server-side language</a>, and there is little it cannot do, as <a href="http://nodejs.org/">node.js</a> very impressively demonstrates. (Disclaimer: I am neither a JavaScript nor a <em>node.js</em> specialist, so I may be doing something incorrectly here.)</p>
<p>Enter <a href="http://nodejs.org/">node.js</a>.</p>
<h3>Node.js</h3>
<p><a href="http://nodejs.org/">node.js</a> is an evented I/O framework built by Ryan Dahl on Google&#039;s <a href="http://en.wikipedia.org/wiki/V8_(JavaScript_engine)">V8 JavaScript engine</a>. It provides a very efficient system to create scalable and efficient applications without having to understand or use threading, processes, etc. In an event-based model you register what should happen, and <a href="http://nodejs.org/">node.js</a> takes care of calling the callback function (you define) when the specific event happens. People who are used to programming JavaScript in a Web browser application (with, for example, jQuery), know this model well.</p>
<p>The event-based programming model is fundamental to JavaScript programming, and <a href="http://nodejs.org/">node.js</a> leverages this model to programs that can handle massively concurrent events with little memory and processing power.</p>
<p>You&#039;ll have to install node.js, but doing so is <a href="http://nodejs.org/#build">easy</a>. Once you have that running, come back and continue here.</p>
<h3>node.couch.js</h3>
<p><a href="http://www.mikealrogers.com">Mikeal Rogers</a> created what he calls a <a href="http://www.mikealrogers.com/archives/726">generic change consumer</a> that you can point at a running <a href="http://couchdb.apache.org/">CouchDB</a> instance. This change consumer, written in <a href="http://nodejs.org/">node.js</a> periodically checks all CouchDB databases for existence of a <code>changes</code> field in the design documents of databases. If such a <code>changes</code> field is found, it must contain valid <a href="http://nodejs.org/">node.js</a> code, which is pulled in by the consumer and executed on the machine running <code>node</code>. </p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/notif-node.png" width="450" height="221" alt="" /></p>
<p>What this means, is that once you have a consumer running, you can use Futon or CouchApp to create and modify code which is stored in your database design document, and that code is automatically read by the consumer and is triggerred by any PUT, POST or DELETE to documents in your database.</p>
<h3>Prerequisites</h3>
<p>To follow what I&#039;m doing here, you&#039;ll need a few things:</p>
<ol>
<li>A running instance of <a href="http://couchdb.apache.org/">CouchDB</a> of course.</li>
<li>A working <a href="http://nodejs.org/">node.js</a> engine. Instructions for obtaining the source and building it are on the node.js site. (Google&#039;s V8 JavaScript engine is contained in <a href="http://nodejs.org/">node.js</a>&#039; source.)</li>
<li>A copy of <a href="http://github.com/mikeal/node.couch.js">node.couch.js</a>. Actually, we need only the two files in <code>changes/lib</code>.</li>
<li>A CouchDB database you can play with, and a design document, named whatever you like.</li>
</ol>
<p>You then launch <em>node</em> with the <tt>service.js</tt> script and point it to the URL of your CouchDB instance. It is important to note that <tt>node</tt> need not run on the same machine as your CouchDB instance: thanks to CouchDB&#039;s RESTful HTTP interface, it can obviously be queried from afar. Furthermore, all JavaScript and/or Node.js dependencies must be accessible on the machine running <tt>node</tt> (i.e. the client machine). I&#039;ve wrapped the lot into a small executable I&#039;ve called <tt>couchlisten.js</tt>:</p>
<pre class="brush: bash;">
#!/usr/bin/env node

var service = require('node.couch.js/changes/lib/service');
var couchdbUrl = 'http://192.168.1.20:5984';

service.start(couchdbUrl);
</pre>
<p>So, what happens when I launch that program? <tt>node</tt> is executed and loads the required modules. The principal module is <tt>service</tt>, which then</p>
<ol>
<li>Reads a list of all databases at the specified URI.</li>
<li>For each of the design documents, <tt>service</tt> tries to load the JavaScript contained in each of the <code>changes</code> fields. If the JavaScript code in the <code>changes</code> field contains an error, <code>node</code> will bail out and stop.</li>
<li>That JavaScript code sets up listeners to wait for document updates to actually occur.</li>
<li><code>service.js</code> sets up an event to rescan CouchDB databases and design documents every 60 seconds.</li>
</ol>
<p>Let me create a first design document called <code>_design/app</code> which contains the following JavaScript in the <code>changes</code> field:</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ4427C682.jpg" width="479" height="263" alt="" /></p>
<p>I&#039;ll now start <code>node</code> on another window, and keep an eye on that, while we continue working here. </p>
<p>I&#039;m now going to add a document to my database.</p>
<pre><code>$ curl -X PUT http://home:5984/db/jane -d '{"name":"Jane","type":"card","sn":"Jolie"}'
{"ok":true,"id":"jane","rev":"1-73ecb8a37d617c04e52c982cd4de5e11"}
</code></pre>
<p>At the same instant, our node listener picks up the change and executes our JavaScript program, so we see on <code>node</code>&#039;s console:</p>
<pre><code>{"seq":2,"id":"jane","changes":[{"rev":"1-73ecb8a37d617c04e52c982cd4de5e11"}],"doc":{"_id":"jane","_rev":"1-73ecb8a37d617c04e52c982cd4de5e11","name":"Jane","type":"card","sn":"Jolie"}}
Hey: Jane
</code></pre>
<p>The first line with the JSON string is printed by the <code>JSON.stringify()</code> method &#8212; useful for visualizing what our JavaScript has to work with. The second line (<code>Hey: Jane</code>) is printed by the second <code>sys.puts()</code>.</p>
<p>Instantaneously.</p>
<p>Shall I delete that document? Ok, here goes:</p>
<pre><code>curl -X DELETE http://home:5984/db/jane?rev=1-73ecb8a37d617c04e52c982cd4de5e11
{"ok":true,"id":"jane","rev":"2-1a6dfcfc0899864047813a9e7ae38b1f"}
</code></pre>
<p>whereupon our <code>node</code> program displays</p>
<pre><code>{"seq":3,"id":"jane","changes":[{"rev":"2-1a6dfcfc0899864047813a9e7ae38b1f"}],"deleted":true,"doc":{"_id":"jane","_rev":"2-1a6dfcfc0899864047813a9e7ae38b1f","_deleted":true}}
</code></pre>
<p>Take particular note of the <code>_deleted</code> field, which allows me to differenciate whether I&#039;m handling a real update or whether I should perhaps ignore this update notification as it is a deletion.</p>
<p>So far we&#039;ve managed to create a minuscule JavaScript program contained in a CouchDB design document which is read by our node.js instance and executed whenever a document is added to (or deleted from) our CouchDB database.</p>
<p>Taking this a step further, let&#039;s see if we can actually do something with the updates.</p>
<h3>Urgent tweets</h3>
<p>Suppose I have a database into which messages (i.e. documents) are posted that may have a particular priority. I want to ensure that documents tagged with an &#034;urgent&#034; priority are notified to my Twitter account as soon as they are added to CouchDB.</p>
<p>The <em>changes</em> script in the design document now contains this JavaScript code, which relies on <a href="http://github.com/ryanmcgrath/twitscript">twitscript</a> (a node.js module) to do the actual tweeting. In order to avoid update notifications for design documents, I check whether the document&#039;s ID is one of those.</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ557AD9E0.jpg" width="489" height="474" alt="" /></p>
<p>I don&#039;t have to restart <tt>couchlisten.js</tt>, as it will pick up the modified design document momentarily.</pp>
<pp>Let me show you what the result looks like in the following screen cast, which consists of three windows:</p>
<ol>
<li>The top window is where I&#039;ll be submitting requests to CouchDB, using <a href="http://blog.fupps.com/2010/04/26/resty/">resty</a>.</li>
<li>The window in the center shows the output of our local <a href="http://nodejs.org/">node.js</a> program, which I&#039;ve called <code>couchlisten.js</code>.</li>
<li>In the bottom window, I have a small <code>node.js</code> program based on <a href="http://github.com/kompozer/twitter-node">twitter-node</a>. This program &#034;follows&#034; the Twitter user who is submitting the &#034;urgent&#034; Tweets.</li>
</ol>
<p><object id="scPlayer" width="545" height="654"><param name="movie" value="http://content.screencast.com/users/jpmens/folders/Jing/media/0119aacd-d713-4242-a02a-2581845e9e3f/jingswfplayer.swf"></param><param name="quality" value="high"></param><param name="bgcolor" value="#FFFFFF"></param><param name="flashVars" value="thumb=http://content.screencast.com/users/jpmens/folders/Jing/media/0119aacd-d713-4242-a02a-2581845e9e3f/FirstFrame.jpg&#038;containerwidth=545&#038;containerheight=654&#038;content=http://content.screencast.com/users/jpmens/folders/Jing/media/0119aacd-d713-4242-a02a-2581845e9e3f/00000006.swf"></param><param name="allowFullScreen" value="true"></param><param name="scale" value="showall"></param><param name="allowScriptAccess" value="always"></param><param name="base" value="http://content.screencast.com/users/jpmens/folders/Jing/media/0119aacd-d713-4242-a02a-2581845e9e3f/"></param>  <embed src="http://content.screencast.com/users/jpmens/folders/Jing/media/0119aacd-d713-4242-a02a-2581845e9e3f/jingswfplayer.swf" quality="high" bgcolor="#FFFFFF" width="545" height="654" type="application/x-shockwave-flash" allowScriptAccess="always" flashVars="thumb=http://content.screencast.com/users/jpmens/folders/Jing/media/0119aacd-d713-4242-a02a-2581845e9e3f/FirstFrame.jpg&#038;containerwidth=545&#038;containerheight=654&#038;content=http://content.screencast.com/users/jpmens/folders/Jing/media/0119aacd-d713-4242-a02a-2581845e9e3f/00000006.swf" allowFullScreen="true" base="http://content.screencast.com/users/jpmens/folders/Jing/media/0119aacd-d713-4242-a02a-2581845e9e3f/" scale="showall"></embed></object></p>
<p>And the result is quite pleasing:</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ3B2E1884.jpg" width="302" height="169" alt="" style='border:1px solid;' /></p>
<h3>And then?</h3>
<p><a href="http://couchdb.apache.org/">CouchDB</a>&#039;s changes API together with <a href="http://nodejs.org/">node.js</a> and a bit of JavaScript open the potential to create exciting applications that react in real time to database modifications. I can use these update notifications for all sorts of interesting tasks:</p>
<ul>
<li>I can use these update notifications to tally how many updates to a database have occurred, and then, initiate a backup to a remote site. (One that doesn&#039;t have CouchDB &#8212; I&#039;d otherwise use replication for that, wouldn&#039;t I? <img src='http://blog.fupps.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' />  )</li>
<li>If I use CouchDB as a writing platform, I can post via <a href="http://www.xmlrpc.com/">XML-RPC</a> to my WordPress, Blogger, MT, etc. blog as soon as a document changes.</li>
<li>If I think &#034;DNS&#034; database, with a solution like my <a href="http://blog.fupps.com/2010/05/05/powerdns-and-a-couchdb-backend/">CouchDB pipe back-end to PowerDNS</a>, I can use an update notification to send an alert e-mail to the domain&#039;s hostmaster, so that (s)he can verify the zone.</li>
</ul>
<p>There is a lot to learn here, and I&#039;ve only just started. For example, there is the <a href="CommonJS effort sets JavaScript on path for world domination">CommonJS</a> initiative which wants to create standardized modules for use in JavaScript. The good news? <a href="">CouchDB</a> supports this as well. There&#039;s also a lot more (for me) to learn about <a href="http://nodejs.org/">node.js</a>, including the <a href="http://wiki.github.com/ry/node/modules">dozens of modules</a> for it; these include Web servers, frameworks, database interfaces, templating systems, parsers, etc.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/05/10/update-notifications-in-couchdb-tweeting-urgent-documents/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Experience live replication in CouchDB: the Pro Git book</title>
		<link>http://blog.fupps.com/2010/05/04/experience-live-replication-in-couchdb-the-pro-git-book/</link>
		<comments>http://blog.fupps.com/2010/05/04/experience-live-replication-in-couchdb-the-pro-git-book/#comments</comments>
		<pubDate>Tue, 04 May 2010 07:21:55 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[Books]]></category>
		<category><![CDATA[CLI]]></category>
		<category><![CDATA[Database]]></category>
		<category><![CDATA[MacOSX]]></category>
		<category><![CDATA[NoSQL]]></category>
		<category><![CDATA[CouchDB]]></category>
		<category><![CDATA[Replication]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3447</guid>
		<description><![CDATA[Do you want to quickly experience CouchDB&#039;s powerful replication? Jan Lehnardt, one of the people behind CouchDB, announced yesterday that he&#039;d made a copy of the very good book Pro Git available as a CouchDB Application (a CouchApp). There are two ways you can go about getting that:

Follow his instructions on getting the app with [...]]]></description>
			<content:encoded><![CDATA[<p>Do you want to quickly experience <a href="http://couchdb.apache.org/">CouchDB</a>&#039;s powerful replication? <a href="http://jan.prima.de/">Jan Lehnardt</a>, one of the people behind CouchDB, <a href="http://twitter.com/janl/status/13308777128">announced</a> yesterday that he&#039;d made a copy of the <a href="http://blog.fupps.com/2009/09/21/on-my-bookshelf-pro-git/">very good book</a> Pro Git available as a CouchDB Application (a CouchApp). There are two ways you can go about getting that:</p>
<ol>
<li>Follow his <a href="http://github.com/janl/progit/tree/master/couchapp/">instructions</a> on getting the app with <code>git</code>, for which you&#039;ll need an installed <a href="http://github.com/couchapp/couchapp">couchapp</a> program.</li>
<li>Continue reading. (Hint: this is much more sexy and more in line with showing off what CouchDB is capable of.)</li>
</ol>
<p>What I&#039;m going to show you here, is how to replicate the content of the <code>progit</code> database Jan created. The result will be a copy of the Pro Git book on your machine:</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ0C59510F.jpg" width="500" height="483" alt="" /></p>
<p> I&#039;m assuming that you</p>
<ul>
<li>have a running copy of CouchDB on a machine. (On a Mac, I recommend Jan&#039;s <a href="http://janl.github.com/couchdbx/">one-click CouchDB package</a>; download, double-click, relax. <img src='http://blog.fupps.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' />  )</li>
<li>your CouchDB is running on address <code>127.0.0.1</code> and port <code>5984</code> (the defaults).</li>
</ul>
<p>Send off a replication command to your CouchDB to &#034;pull&#034; the content of the database.</p>
<pre class="brush: bash;">
curl -X POST http://127.0.0.1:5984/_replicate -d \
      '{&quot;source&quot;:&quot;http://progit.couchone.com/progit&quot;,\
        &quot;target&quot;:&quot;progit&quot;,\
        &quot;create_target&quot;:true}'
</pre>
<p>This instructs your CouchDB to connect to the <em>source</em> URL and start replicating the <em>progit</em> database. Your CouchDB will create the <em>target</em> database, also called <em>progit</em>. (Note: the <tt>create_target</tt> option is new for CouchDB 0.11; if you have an older version omit the option and pre-create the database in Futon.) </p>
<p>This can take a bit of time. When replication finishes, you should see the results, which will look something like this (reformatted for clarity):</p>
<pre class="brush: plain;">
{
   &quot;ok&quot; : true,
   &quot;history&quot; : [
      {
         &quot;docs_read&quot; : 10,
         &quot;session_id&quot; : &quot;9c6ec637a9686e3078de963fa6774c56&quot;,
         &quot;recorded_seq&quot; : 10,
         &quot;end_last_seq&quot; : 10,
         &quot;doc_write_failures&quot; : 0,
         &quot;start_time&quot; : &quot;Tue, 04 May 2010 06:54:22 GMT&quot;,
         &quot;start_last_seq&quot; : 0,
         &quot;end_time&quot; : &quot;Tue, 04 May 2010 07:04:50 GMT&quot;,
         &quot;missing_checked&quot; : 0,
         &quot;docs_written&quot; : 10,
         &quot;missing_found&quot; : 10
      }
   ],
   &quot;session_id&quot; : &quot;9c6ec637a9686e3078de963fa6774c56&quot;,
   &quot;source_last_seq&quot; : 10
}
</pre>
<p>Now point your Web browser at your own CouchDB instance (which is probably running on <code>127.0.0.1</code>), with a URL of</p>
<pre><code>http://127.0.0.1:5984/progit/_design/chacon/_show/chapter/01-chapter1
</code></pre>
<p>which should get you a page that looks like what you saw above.</p>
<p>After browsing around a bit, I recommend you use Futon to access your CouchDB instance (<code>http://127.0.0.1:5984/_utils/index.html</code>) and look at what is there. Fascinating.</p>
<p>And, if you didn&#039;t feel like doing all that and simply want to see what it looked like, you <a href="http://progit.couchone.com/progit/_design/chacon/_show/chapter/01-chapter1">can do so here</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/05/04/experience-live-replication-in-couchdb-the-pro-git-book/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>A trickle from the Twitter Firehose</title>
		<link>http://blog.fupps.com/2010/04/27/a-trickle-from-the-twitter-firehose/</link>
		<comments>http://blog.fupps.com/2010/04/27/a-trickle-from-the-twitter-firehose/#comments</comments>
		<pubDate>Tue, 27 Apr 2010 10:17:05 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[CLI]]></category>
		<category><![CDATA[Database]]></category>
		<category><![CDATA[data]]></category>
		<category><![CDATA[twitter]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3434</guid>
		<description><![CDATA[For experimenting, I wanted a source of data. Lots of data. For my application it would be trivial to generate such data, but I wanted lots of randomness in it, so I tapped into a bit of the Twitter firehose. This firehose is the full feed of tweets (somewhere about 50 million (!) tweets per [...]]]></description>
			<content:encoded><![CDATA[<p>For experimenting, I wanted a source of data. Lots of data. For my application it would be trivial to generate such data, but I wanted lots of randomness in it, so I tapped into a bit of the Twitter firehose. This firehose is the full feed of tweets (somewhere about <a href="http://blog.twitter.com/2010/02/measuring-tweets.html">50 million (!) tweets per day</a>) to which only a <a href="http://blog.twitter.com/2010/03/enabling-rush-of-innovation.html">limited number of clients</a> have access. </p>
<p>As part of the firehose, there is an API called the <a href="http://apiwiki.twitter.com/Streaming-API-Documentation">Streaming API</a> which allows anybody access to a fraction of the stream, so I tapped into that. (I jokingly <a href="http://twitter.com/jpmens/status/12890848067">said yesterday</a> that I may be responsible for some of the <a href="http://en.wikipedia.org/wiki/Twitter#Outages">fail whales</a> of the day.)</p>
<p>The stream we mere mortals have access to provides me with about 50.000 statuses (tweets) per hour, which adds up after a day or two. Here is a shot of the statuses streaming past:</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ07255962.jpg" width="400" height="361" alt="" /></p>
<p>The individual tweets are provided in <a href="http://www.json.org/">JSON</a> format, which suits me well, and I&#039;ll turn the tap off in <del>an hour</del> a day or two.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/04/27/a-trickle-from-the-twitter-firehose/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Resty: making curl access to CouchDB easy</title>
		<link>http://blog.fupps.com/2010/04/26/resty/</link>
		<comments>http://blog.fupps.com/2010/04/26/resty/#comments</comments>
		<pubDate>Mon, 26 Apr 2010 12:57:06 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[CLI]]></category>
		<category><![CDATA[NoSQL]]></category>
		<category><![CDATA[CouchDB]]></category>
		<category><![CDATA[curl]]></category>
		<category><![CDATA[REST]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3432</guid>
		<description><![CDATA[Resty is a marvellous little shell script which simplifies command-line access to REST resources (e.g. CouchDB) tremendously.
Instead of very long curl command-lines, you&#039;ll typically use the GET, PUT, POST, DELETE commands &#8212; functions imported to your shell from Resty.
I&#039;ve made a short screencast to demonstrate:
  
That last command I showed you might be Resty&#039;s [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://github.com/micha/resty">Resty</a> is a marvellous little shell script which simplifies command-line access to REST resources (e.g. <a href="http://couchdb.apache.org/">CouchDB</a>) tremendously.</p>
<p>Instead of very long <a href="http://curl.haxx.se/">curl</a> command-lines, you&#039;ll typically use the GET, PUT, POST, DELETE commands &#8212; functions imported to your shell from <a href="http://github.com/micha/resty">Resty</a>.</p>
<p>I&#039;ve made a short screencast to demonstrate:</p>
<p><object id="scPlayer" width="573" height="523"><param name="movie" value="http://content.screencast.com/users/jpmens/folders/Jing/media/7a3cd8ca-de2a-4670-bf35-944cf34978e2/jingswfplayer.swf"></param><param name="quality" value="high"></param><param name="bgcolor" value="#FFFFFF"></param><param name="flashVars" value="thumb=http://content.screencast.com/users/jpmens/folders/Jing/media/7a3cd8ca-de2a-4670-bf35-944cf34978e2/FirstFrame.jpg&#038;containerwidth=573&#038;containerheight=523&#038;content=http://content.screencast.com/users/jpmens/folders/Jing/media/7a3cd8ca-de2a-4670-bf35-944cf34978e2/00000005.swf"></param><param name="allowFullScreen" value="true"></param><param name="scale" value="showall"></param><param name="allowScriptAccess" value="always"></param><param name="base" value="http://content.screencast.com/users/jpmens/folders/Jing/media/7a3cd8ca-de2a-4670-bf35-944cf34978e2/"></param>  <embed src="http://content.screencast.com/users/jpmens/folders/Jing/media/7a3cd8ca-de2a-4670-bf35-944cf34978e2/jingswfplayer.swf" quality="high" bgcolor="#FFFFFF" width="573" height="523" type="application/x-shockwave-flash" allowScriptAccess="always" flashVars="thumb=http://content.screencast.com/users/jpmens/folders/Jing/media/7a3cd8ca-de2a-4670-bf35-944cf34978e2/FirstFrame.jpg&#038;containerwidth=573&#038;containerheight=523&#038;content=http://content.screencast.com/users/jpmens/folders/Jing/media/7a3cd8ca-de2a-4670-bf35-944cf34978e2/00000005.swf" allowFullScreen="true" base="http://content.screencast.com/users/jpmens/folders/Jing/media/7a3cd8ca-de2a-4670-bf35-944cf34978e2/" scale="showall"></embed></object></p>
<p>That last command I showed you might be <a href="http://github.com/micha/resty">Resty</a>&#039;s best, so I&#039;ll give it to you again, so that it sinks in:</p>
<pre class="brush: bash;">
GET /jane | PUT -V
</pre>
<p>Highly recommended.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/04/26/resty/feed/</wfw:commentRss>
		<slash:comments>4</slash:comments>
		</item>
		<item>
		<title>The Antepenultimate CouchDB Reference Card (cheat sheet)</title>
		<link>http://blog.fupps.com/2010/04/20/the-antepenultimate-couchdb-reference-card/</link>
		<comments>http://blog.fupps.com/2010/04/20/the-antepenultimate-couchdb-reference-card/#comments</comments>
		<pubDate>Tue, 20 Apr 2010 11:35:15 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[Apache]]></category>
		<category><![CDATA[Books]]></category>
		<category><![CDATA[CLI]]></category>
		<category><![CDATA[Database]]></category>
		<category><![CDATA[NoSQL]]></category>
		<category><![CDATA[cheat-sheet]]></category>
		<category><![CDATA[CouchDB]]></category>
		<category><![CDATA[reference]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3424</guid>
		<description><![CDATA[After starting off with CouchDB I thought I&#039;d create a reference card (a.k.a. cheat sheet) for myself, a bit like the one I created for proxy configuration files. As I&#039;m rather satisfied with the result, I thought I&#039;d share this PDF with you.

Go ahead and get The Antepenultimate CouchDB Reference Card. If you have suggestions [...]]]></description>
			<content:encoded><![CDATA[<p>After <a href="http://blog.fupps.com/2010/04/09/couchdb-and-a-bit-of-lotus-notes/">starting off</a> with <a href="http://couchdb.apache.org/">CouchDB</a> I thought I&#039;d create a reference card (a.k.a. cheat sheet) for myself, a bit like the one <a href="http://blog.fupps.com/2009/10/02/proxy-autoconfiguration-pac-reference-card-cheat-sheet/">I created for proxy configuration files</a>. As I&#039;m rather satisfied with the result, I thought I&#039;d share this PDF with you.</p>
<p><a href="http://mens.de/:/couchdbref"><img src="http://blog.fupps.com/wp-content/media/2010/ZZ7FD1B679.jpg" width="500" height="346" alt="" /></a></p>
<p>Go ahead and get <a href="http://mens.de/:/couchdbref">The Antepenultimate CouchDB Reference Card</a>. If you have suggestions on how to improve it, I&#039;d by happy to hear in the comments.</p>
<p><b>Update 2010-04-22</b>: added getting documents by keys, more map/reduce, a bit of CouchApp, and a section I call &#034;More relaxation&#034; with pointers to related tools.<br />
<b>Update 2010-04-24</b>: small fixes, added links<br />
<b>Update 2010-04-26</b>: added replication filters and URL rewriting.<br />
<b>Update 2010-04-29</b>: tweaked attachments section, and added <a href="http://github.com/jpmens/pudo">pudo</a>.<br />
<b>Update 2010-05-11</b>: added update handlers.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/04/20/the-antepenultimate-couchdb-reference-card/feed/</wfw:commentRss>
		<slash:comments>4</slash:comments>
		</item>
		<item>
		<title>CouchDB (and a bit of Lotus Notes)</title>
		<link>http://blog.fupps.com/2010/04/09/couchdb-and-a-bit-of-lotus-notes/</link>
		<comments>http://blog.fupps.com/2010/04/09/couchdb-and-a-bit-of-lotus-notes/#comments</comments>
		<pubDate>Fri, 09 Apr 2010 10:53:43 +0000</pubDate>
		<dc:creator>Jan-Piet Mens</dc:creator>
				<category><![CDATA[CLI]]></category>
		<category><![CDATA[Database]]></category>
		<category><![CDATA[DomiNotes]]></category>
		<category><![CDATA[NoSQL]]></category>
		<category><![CDATA[CouchDB]]></category>

		<guid isPermaLink="false">http://blog.fupps.com/?p=3411</guid>
		<description><![CDATA[CouchDB is a document-oriented database, a non-relational datastore, one which belongs to the group of NoSQL databases. (More on NoSQL here.) Contrary to a relational database (RDBMS) in which data is contained in tables, rows, and columns, CouchDB contains a collection of JSON documents in a modified B-Tree file structure, each of which is indexed [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://couchdb.apache.org/">CouchDB</a> is a <a href="http://en.wikipedia.org/wiki/Document-oriented_database">document-oriented</a> database, a non-relational datastore, one which belongs to the group of <a href="http://en.wikipedia.org/wiki/NoSQL">NoSQL</a> databases. (More on <a href="http://www.roadtofailure.com/2010/04/08/nosql-why-its-so-damn-sticky/">NoSQL here</a>.) Contrary to a relational database (RDBMS) in which data is contained in tables, rows, and columns, CouchDB contains a collection of <a href="http://www.json.org/">JSON</a> documents in a modified B-Tree file structure, each of which is indexed by a single unique key. (The key is often a <a href="http://en.wikipedia.org/wiki/Globally_Unique_Identifier">GUUID</a>, but it need not be.)</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ78EAA48F.jpg" width="400" height="164" alt="" /></p>
<p>CouchDB documents can have any number of files attached to them, including their MIME type, and each of these documents contains a revision number which CouchDB uses to support multiversion concurrency control (<a href="http://en.wikipedia.org/wiki/Multiversion_concurrency_control">MVCC</a>).</p>
<p><a href="http://www.designboom.com/weblog/cat/8/view/9492/quinze-milan-eastpak-built-to-resist.html"><img src="http://blog.fupps.com/wp-content/media/2010/ZZ73A89A6C.jpg" width="450" height="274" alt="" /></a></p>
<p>CouchDB is accessed via HTTP only; there are no application-specific binary protocols used, making it an ideal system to be put behind load-balancers and proxies. As such, one of the tools of the trade with which to quickly access CouchDB is <a href="http://curl.haxx.se/">curl</a>.</p>
<p>For example, in order to create a database in CouchDB and add the document shown above, I do the following:</p>
<pre class="brush: bash;">
H=http://127.0.0.1:5984/
curl -X PUT $H/demo
{&quot;ok&quot;:true}

curl -X PUT $H/demo/1 -d &quot;`cat /tmp/jane.json`&quot;
{&quot;ok&quot;:true,&quot;id&quot;:&quot;1&quot;,&quot;rev&quot;:&quot;1-2867008f6fa2e8e834745efc4da65b01&quot;}
</pre>
<p>The first curl invocation creates a database called <tt>demo</tt>, and the second creates a new document with a key <tt>1</tt> reading the content of the data from the specified JSON text. CouchDB responds with a JSON structure indicating errors or success.</p>
<p>CouchDB&#039;s Futon utility, also accessible from a Web browser, shows the result and lets me modify the document, attach a file, etc. (CouchDB&#039;s tagline, <em>relax</em>, is neatly carried on in the name <a href="http://en.wikipedia.org/wiki/Futon">futon</a>.)</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ6E4223B5.jpg" width="500" height="398" alt="" /></p>
<p>According to its author, <a href="http://damienkatz.net/2010/02/migrating_notesdomino_to_couch.html">CouchDB is largely inspired</a> by the <a href="http://en.wikipedia.org/wiki/IBM_Lotus_Notes">Lotus Notes</a> back-end</a> (i.e. Lotus Domino), and I see a number of parallels:</p>
<ul>
<li>CouchDB detects and flags document conflicts when a document is updated on more than one server. However, contrary to Notes, it cannot merge individual portions of a document &#8212; that is an exercise left to the application.</li>
<li>CouchDB and Lotus Notes are predestined for offline use with subsequent replication when online.</li>
<li>I can set up CouchDB on a number of nodes (systems) and have a single master replicate to any number of slave nodes (on demand or continuously), and I can have an application write to any number of those nodes, effectively creating a multi-master system, just as with Lotus Notes. (The Lotus Notes client&#039;s automatic failover to a different cluster node I would implement with a proxy, just as I&#039;d have to do with a Notes Web application.) As with Lotus Domino, a cluster of CouchDB nodes need not be implemented on identical hardware or operating system &#8212; a great plus.</li>
<li>CouchDB integrates a <a href="http://couchdb.apache.org/docs/overview.html">view model</a> for aggregating and reporting on documents in a database. Views are built dynamically, and they are defined withing special design documents. This is quite similar to they way I view Notes views.</li>
</ul>
<p>There are, however, a number of things that <a href="http://www.ibm.com/developerworks/lotus/library/ls-NDHistory/">Lotus Notes</a> includes out of the box, which CouchDB doesn&#039;t:</p>
<ul>
<li>Notes has a very high and very granular level of security. (Encrypted connections, encrypted databases, reader and author fields, etc.)</li>
<li>Due to its &#034;all-in-a-box&#034; model, Lotus Notes integrates e-mail, calendaring and directory services, which CouchDB doesn&#039;t do, of course, as it is a database layer only.</li>
</ul>
<p>Whether these are required in Web applications or not, is debatable.</p>
<p>CouchDB uses JavaScript as its internal language, although others are possible, where Lotus Notes uses @Formula, LotusScript, Java, and the C API.</p>
<p>Where CouchDB certainly excels compared to Lotus Notes/Domino is upon storing a large number of documents. I remember creating a Notes application some years ago (on 7.x), with a database containing a couple of hundred thousand small documents (about 200 &#8212; 400 bytes per document) and two or three views. Whenever that database was opened on a Notes client, the whole system (client and server) crawled. Now, this is a bit like comparing apples and pine trees, but a CouchDB database with ten times the number of documents in the same application doesn&#039;t make the server sweat.</p>
<p>CouchDB is supported by a <a href="http://wiki.apache.org/couchdb/Basics">large number of programming languages</a> (including <a href="http://wiki.apache.org/couchdb/Getting_started_with_LotusScript">LotusScript</a>). For PHP developers out there, IBM has nice document called <a href="http://www.ibm.com/developerworks/opensource/library/os-php-couchdb/index.html">CouchDB basics for PHP developers</a> on offer.</p>
<p><img src="http://blog.fupps.com/wp-content/media/2010/ZZ6C87555C.jpg" width="350" height="392" alt="" /></p>
<p>If I work with Lotus Notes, is it useful to know (or even to <a href="http://damienkatz.net/2010/02/migrating_notesdomino_to_couch.html">migrate certain applications to</a>) CouchDB? The answer is yes. There are some resemblances, which make my life easy in a transition phase. As an Open Source product, CouchDB has no licensing fees (and support is available from <a href="http://www.couch.io/">CouchIO</a> &#8212; the makers of CouchDB). I get to use whichever programming language I desire, and I&#039;m not restricted by the &#034;IDE&#034; that the Domino Designer is. Complete applications called <a href="http://github.com/jchris/couchapp/tree/master">CouchApps</a> can be store in and served by CouchDB, </p>
<p><a href="http://damienkatz.net/">Damien Katz</a>, the man who rewrote the Lotus Notes @Formula language, is the author of CouchDB. He gives an <a href="http://www.infoq.com/interviews/CouchDB-Damien-Katz">interview</a> which is worthwhile listening to if you have the time. I recommend reading <a href="http://books.couchdb.org/relax/">CouchDB: The Definitive Guide</a>: I read it cover to cover and liked it, although you need a bit of imagination here and there. A good resource is also Matt Woodward&#039;s <a href="http://blog.mattwoodward.com/massive-couchdb-brain-dump">Massive CouchDB Brain Dump</a> &#8212; a large collection of bullet points.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.fupps.com/2010/04/09/couchdb-and-a-bit-of-lotus-notes/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
	</channel>
</rss>
