Thursday, 20 December 2012

Imagemagick

 The version of Imagemagick on velvet did not support the -path option for mogrify
# mogrify
Version: ImageMagick 6.2.4 07/28/09 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2005 ImageMagick Studio LLC
Upgraded using:

How to install ImageMagick and Imagick on Debian Linux 
Also needed:
  • http://stackoverflow.com/questions/11515118/imagemagick-not-converting-jpegs
  • http://www.hardwaretutorials.com/2012/05/installing-png-support-for-debian/
Result:
/usr/local/bin/mogrify | more
Version: ImageMagick 6.8.1-0 2012-12-20 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2013 ImageMagick Studio LLC

Monday, 19 November 2012

XQuery templating engines and TXQ

"...separating logic from presentation" is good thing; and you don`t need much presentation before code can start looking a bit messy. Here is an example of mine at about the limit of what I feel comfortable with.

Landscape

I have been using Node.js a bit recently and it has an impressive list of  templating engines. One of the things I like about  Node is the module ecosystem; where vast numbers of different  approaches compete in  Darwinian fashion for "developer mind share". For reasons that I can't quite remember, I ended up using EJS with custom delimiters of single { and }. It looks a bit like XQuery.

When it comes to XQuery there are not so many choices :-). Maybe most are mentioned in this thread: eXist MVC and separating XHTML templates from XQuery code More recently eXistdb has Exist html templates. This looks to be standard XQuery 3.0 and provide a lot of high level functionality, but very much linked into the eXistdb environment.

What the world needs now is...

.. another XQuery templating system. So from my experience with EJS and express.js.  Some features that I liked are:
  • A render method to take a view and a map type object to produce a page
  • A location, a collection or folder, for templates (views) completely separated from the code.
  • A quick syntax to wrap common page features  around rendered results.
  • A block/partial/repeat type concept for list/tables.
 Looking for the simplest thing that might possibly  work..Noting that:
  1. A XML document is a valid XQuery program.
  2. If we put some {$var} type expressions in it and get XQuery to evaluate it  in the right context-  we are almost there.
  3. BaseX has a (non-standard) function xquery:invoke that has just the right signature.
The start..
declare function txq:render($template,$map){
xquery:invoke($template,$map)
}
view raw render.xq hosted with ❤ by GitHub
With this view1.xml template
<div>{$greeting} {$who}</div>
view raw view1.xml hosted with ❤ by GitHub
Invoking..
txq:render("view1.xml",map{"greeting":="hello","who":="world"})
(: result :)
<div>hello world</div>
view raw simple.xq hosted with ❤ by GitHub

Layout pages use the convention that a layout template has a $body field into which the sub template will be inserted
(:~
: template function with wrapping layout
: @param layout
: @return updated doc from map
:)
declare function render($template as xs:string,$map as map(*),
$layout as xs:string){
let $content:=render($template,$map)
let $map:=map:new(($map,map{"body":=$content}))
return render($layout,$map)
};
view raw render2.xq hosted with ❤ by GitHub
Example layout template.
<html>
<head><title>{$title}</title>
<script src="...."></script>
</head>
<body>
<div>..nav bar..</div>
{$body}
</body>
</html>
view raw layout.xml hosted with ❤ by GitHub
Invoking
let $map:=map{"greeting":="hello","who":="world","title":="full page"}
return txq:render("view1.xml", $map, "layout.xml")
(: result :)
<html>
<head><title>full page</title>
<script src="...."></script>
</head>
<body>
<div>..nav bar..</div>
<div>hello world</div>
</body>
</html>
view raw page.xq hosted with ❤ by GitHub

Often there is a need to render repeated items from some node collection. A solution to this repeatedly invoke a sub-template, injecting a map with an item set to each node in turn. For example our map has a "actions" value. An XML node with a variable number of <generate> sub nodes that we wish to render.
<div>
<h2>{$heading} - {count($actions/generate)}</h2>
<p>Actions are processes that generate a new item from an existing item.</p>
<div>
{$partial("action1.xml","action",$actions/generate )}
</div>
</div>
view raw actions.xml hosted with ❤ by GitHub
The actions1.xml partial template.
<div>
<h3>{$action/@id/string()}</h3>
<span class="label">{$action/@in/string()}</span> &#x2192; <span class="label">{$action/@out/string()}</span>
<p>{$action/describe}</p>
<pre>{$action/xquery}</pre>
</div>
view raw actions1.xml hosted with ❤ by GitHub

A slight addition to the original render function makes this easy. Below shows the the full TXQ module.
(:~
: A(nother) templating Engine for XQuery (BaseX 7.5 specific)
: specials:
: partial(file,name,sequence)
:
: @author andy bunce
: @since sept 2012
:)
module namespace txq = 'apb.txq';
declare default function namespace 'apb.txq';
import module namespace xquery = "http://basex.org/modules/xquery";
(:~
: template function
: @param template url to fill
: @param map name and value to apply
: @return updated doc from map
:)
declare function render($template as xs:string,$map as map(*)){
let $map:=map:new(($map,map{"partial":=partial(?,?,?,$map,$template)}))
return xquery:invoke($template,$map)
};
(:~
: template function with wrapping layout
: @param layout
: @return updated doc from map
:)
declare function render($template as xs:string,$map as map(*),$layout as xs:string){
let $content:=render($template,$map)
let $map:=map:new(($map,map{"body":=$content}))
return render($layout,$map)
};
(:~
: partial template function: evaluate part for each value in sequence
: @return updated doc from map
:)
declare function partial($part as xs:string,$name,$seq,$map,$base){
for $s in $seq
let $map:=map:new(($map,map{$name:=$s}))
return render(fn:resolve-uri($part,$base),$map)
};
view raw txq.xqm hosted with ❤ by GitHub

Saturday, 10 November 2012

Software updates

NAS updates

  • RAIDiator on readynas to  4.1.10  need to add index.xml to /etc/frontview/apache/httpd.conf
  • RAIDiator on velvet to 4.2.22  
Hoping 4.2.22 will fix the Readynas pro locking up after a couple of weeks.

PaaS

My openshift BaseX instances seem broken by their upgrades. Version rhc 1.0.4 is giving permission denied for server. this?

Wednesday, 12 September 2012

Gadgets: tl-wr703n

I bought one of these : totocity Chinese Handy Wi-Fi Wireless Router 150M Wifi Wi-Fi for USB 3G LAN WAN Converter with the idea of creating a MIFI wireless network to use with my tablet.
To be honest I thought it had a 3G modem, but in fact it can only make use of one via USB - never mind I have a ZTE modem and it looks like a fun device.
£21 for a tiny UNIX server with LAN + WIFI. More information here. Hardware hacks here

 

 Step 1 

"I wouldn't call this a router for the average home user."
I replaced the Chinese interface with open-wrt see instructions at Installing OpenWrt on the TL-WR703N not too hard really.

http://wiki.openwrt.org/toh/tp-link/tl-wr703n

Step 2

Getting the modem to work..

install packages :

  • kmod-usb2, 
  • kmod-usb-option, 
  • comgt, 
  • usb-modeswitch, 
  • usb-modeswitch-data  

to be continued...the boot log

Sep 12 21:04:03 OpenWrt syslog.info syslogd started: BusyBox v1.19.4
Sep 12 21:04:03 OpenWrt kern.notice kernel: klogd started: BusyBox v1.19.4 (2012-08-28 23:48:59 EDT)
Sep 12 21:04:03 OpenWrt kern.notice kernel: [ 0.000000] Linux version 3.3.8 (openwrt@callisto.lab.bufferbloat.net) (gcc version 4.6.3 20120201 (prerelease) (Linaro GCC 4.6-2012.02) ) #1 Thu Aug 30 06:36:56 EDT 2012
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 0.000000] MyLoader: sysp=3dd7cb87, boardp=f0717c55, parts=4852ecbe
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] bootconsole [early0] enabled
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] CPU revision is: 00019374 (MIPS 24Kc)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] SoC: Atheros AR9330 rev 1
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Clocks: CPU:400.000MHz, DDR:400.000MHz, AHB:200.000MHz, Ref:25.000MHz
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Determined physical RAM map:
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] memory: 02000000 @ 00000000 (usable)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Initrd not found or empty - disabling initrd
Sep 12 21:04:03 OpenWrt kern.warn kernel: [ 0.000000] Zone PFN ranges:
Sep 12 21:04:03 OpenWrt kern.warn kernel: [ 0.000000] Normal 0x00000000 -> 0x00002000
Sep 12 21:04:03 OpenWrt kern.warn kernel: [ 0.000000] Movable zone start PFN for each node
Sep 12 21:04:03 OpenWrt kern.warn kernel: [ 0.000000] Early memory PFN ranges
Sep 12 21:04:03 OpenWrt kern.warn kernel: [ 0.000000] 0: 0x00000000 -> 0x00002000
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 0.000000] On node 0 totalpages: 8192
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 0.000000] free_area_init_node: node 0, pgdat 8031a620, node_mem_map 81000000
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 0.000000] Normal zone: 64 pages used for memmap
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 0.000000] Normal zone: 0 pages reserved
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 0.000000] Normal zone: 8128 pages, LIFO batch:0
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 0.000000] pcpu-alloc: [0] 0
Sep 12 21:04:03 OpenWrt kern.warn kernel: [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Sep 12 21:04:03 OpenWrt kern.notice kernel: [ 0.000000] Kernel command line: board=TL-WR703N console=ttyATH0,115200 rootfstype=squashfs,jffs2 noinitrd
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] PID hash table entries: 128 (order: -3, 512 bytes)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Sep 12 21:04:03 OpenWrt kern.warn kernel: [ 0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
Sep 12 21:04:03 OpenWrt kern.warn kernel: [ 0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Writing ErrCtl register=00000000
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Readback ErrCtl register=00000000
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Memory: 28952k/32768k available (2215k kernel code, 3816k reserved, 579k data, 208k init, 0k highmem)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] SLUB: Genslabs=9, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] NR_IRQS:51
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.000000] Calibrating delay loop... 265.42 BogoMIPS (lpj=1327104)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.080000] pid_max: default: 32768 minimum: 301
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.080000] Mount-cache hash table entries: 512
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.090000] NET: Registered protocol family 16
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.090000] gpiochip_add: registered GPIOs 0 to 29 on device: ath79
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.100000] MIPS: machine is TP-LINK TL-WR703N v1
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.340000] bio: create slab <bio-0> at 0
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.350000] Switching to clocksource MIPS
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.350000] NET: Registered protocol family 2
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.360000] IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.360000] TCP established hash table entries: 1024 (order: 1, 8192 bytes)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 0.360000] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 8.540000] usbcore: registered new device driver usb
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 9.020000] ath: EEPROM regdomain: 0x0
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 9.020000] ath: EEPROM indicates default country code should be used
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 9.020000] ath: doing EEPROM country->regdmn map search
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 9.020000] ath: country maps to regdmn code: 0x3a
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 9.020000] ath: Country alpha2 being used: US
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 9.020000] ath: Regpair used: 0x3a
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 9.030000] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
Sep 12 21:04:03 OpenWrt kern.debug kernel: [ 9.030000] Registered led device: ath9k-phy0
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.030000] ieee80211 phy0: Atheros AR9330 Rev:1 mem=0xb8100000, irq=2
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.040000] cfg80211: Calling CRDA for country: US
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.040000] cfg80211: Regulatory domain changed to country: US
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.050000] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.060000] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (300 mBi, 2700 mBm)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.060000] cfg80211: (5170000 KHz - 5250000 KHz @ 40000 KHz), (300 mBi, 1700 mBm)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.070000] cfg80211: (5250000 KHz - 5330000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.080000] cfg80211: (5490000 KHz - 5600000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.090000] cfg80211: (5650000 KHz - 5710000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.090000] cfg80211: (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 3000 mBm)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.160000] PPP generic driver version 2.4.2
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.300000] ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.420000] NET: Registered protocol family 24
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.440000] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.440000] ehci-platform ehci-platform: Generic Platform EHCI Controller
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.450000] ehci-platform ehci-platform: new USB bus registered, assigned bus number 1
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.490000] ehci-platform ehci-platform: irq 3, io mem 0x1b000000
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.510000] ehci-platform ehci-platform: USB 2.0 started, EHCI 1.00
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.510000] hub 1-0:1.0: USB hub found
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.510000] hub 1-0:1.0: 1 port detected
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.540000] nf_conntrack version 0.5.0 (455 buckets, 1820 max)
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.790000] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Sep 12 21:04:03 OpenWrt kern.info kernel: [ 9.900000] usb 1-1: new high-speed USB device number 2 using ehci-platform
Sep 12 21:04:05 OpenWrt kern.info kernel: [ 13.050000] device eth0 entered promiscuous mode
Sep 12 21:04:05 OpenWrt daemon.notice netifd: Interface 'lan' is now up
Sep 12 21:04:05 OpenWrt daemon.notice netifd: Interface 'loopback' is now up
Sep 12 21:04:05 OpenWrt kern.info kernel: [ 13.360000] eth0: link up (100Mbps/Full duplex)
Sep 12 21:04:05 OpenWrt kern.info kernel: [ 13.360000] br-lan: port 1(eth0) entered forwarding state
Sep 12 21:04:05 OpenWrt kern.info kernel: [ 13.370000] br-lan: port 1(eth0) entered forwarding state
Sep 12 21:04:06 OpenWrt user.notice usb-modeswitch: 1-0:1.0: Manufacturer=Linux_3.3.8_ehci_hcd Product=Generic_Platform_EHCI_Controller Serial=ehci-platform
Sep 12 21:04:06 OpenWrt user.notice ifup: Enabling Router Solicitations on lan (br-lan)
Sep 12 21:04:06 OpenWrt user.notice usb-modeswitch: 1-1:1.0: Manufacturer=ZTE,Incorporated Product=ZTE_CDMA_Technologies_MSM Serial=1234567890ABCDEF
Sep 12 21:04:06 OpenWrt user.notice usb-modeswitch: 1-1:1.0: Selecting /etc/usb_modeswitch.d/19d2:2000 for mode switching
Sep 12 21:04:06 OpenWrt syslog.notice usb_modeswitch: switching device 19d2:2000 on 001/002
Sep 12 21:04:07 OpenWrt kern.info kernel: [ 15.370000] br-lan: port 1(eth0) entered forwarding state
Sep 12 21:04:08 OpenWrt user.notice ifup: Enabling Router Solicitations on loopback (lo)
Sep 12 21:04:08 OpenWrt kern.info kernel: [ 16.570000] usb 1-1: USB disconnect, device number 2
Sep 12 21:04:09 OpenWrt kern.info kernel: [ 16.850000] usb 1-1: new high-speed USB device number 3 using ehci-platform
Sep 12 21:04:11 OpenWrt user.info sysinit: Loading defaults
Sep 12 21:04:12 OpenWrt user.info sysinit: Loading synflood protection
Sep 12 21:04:12 OpenWrt user.info sysinit: Adding custom chains
Sep 12 21:04:12 OpenWrt user.info sysinit: Loading zones
Sep 12 21:04:13 OpenWrt user.info sysinit: Loading forwardings
Sep 12 21:04:13 OpenWrt user.info sysinit: Loading rules
Sep 12 21:04:13 OpenWrt user.info sysinit: Loading redirects
Sep 12 21:04:13 OpenWrt user.info sysinit: Loading includes
Sep 12 21:04:13 OpenWrt user.info sysinit: Optimizing conntrack
Sep 12 21:04:13 OpenWrt user.info sysinit: Loading interfaces
Sep 12 21:04:14 OpenWrt user.info firewall: adding lan (br-lan) to zone lan
Sep 12 21:04:14 OpenWrt authpriv.info dropbear[1074]: Running in background
Sep 12 21:04:16 OpenWrt user.notice dnsmasq: DNS rebinding protection is active, will discard upstream RFC1918 responses!
Sep 12 21:04:16 OpenWrt user.notice dnsmasq: Allowing 127.0.0.0/8 responses
Sep 12 21:04:16 OpenWrt user.notice dnsmasq: found already running DHCP-server on interface 'br-lan' refusing to start, use 'option force 1' to override
Sep 12 21:04:16 OpenWrt daemon.info dnsmasq[1121]: started, version 2.62 cachesize 150
Sep 12 21:04:16 OpenWrt daemon.info dnsmasq[1121]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack
Sep 12 21:04:16 OpenWrt daemon.info dnsmasq[1121]: using local addresses only for domain lan
Sep 12 21:04:16 OpenWrt daemon.info dnsmasq[1121]: reading /tmp/resolv.conf.auto
Sep 12 21:04:16 OpenWrt daemon.info dnsmasq[1121]: using nameserver 192.168.1.1#53
Sep 12 21:04:16 OpenWrt daemon.info dnsmasq[1121]: using local addresses only for domain lan
Sep 12 21:04:16 OpenWrt daemon.info dnsmasq[1121]: read /etc/hosts - 1 addresses
Sep 12 21:04:58 OpenWrt user.notice usb-modeswitch: 1-1:1.0: Manufacturer=ZTE,Incorporated Product=ZTE_CDMA_Technologies_MSM Serial=1234567890ABCDEF
Sep 12 21:04:59 OpenWrt user.notice usb-modeswitch: 1-1:1.1: Manufacturer=ZTE,Incorporated Product=ZTE_CDMA_Technologies_MSM Serial=1234567890ABCDEF
Sep 12 21:04:59 OpenWrt user.notice usb-modeswitch: 1-1:1.2: Manufacturer=ZTE,Incorporated Product=ZTE_CDMA_Technologies_MSM Serial=1234567890ABCDEF
Sep 12 21:04:59 OpenWrt user.notice usb-modeswitch: 1-1:1.3: Manufacturer=ZTE,Incorporated Product=ZTE_CDMA_Technologies_MSM Serial=1234567890ABCDEF
view raw gistfile1.txt hosted with ❤ by GitHub

Monday, 3 September 2012

Drawing trees with XQuery and SVG

Looking for a simple way to draw tree structures, I came across this old xml.com article Automated Tree Drawing: XSLT and SVG  by Jirka Kosek. This uses some EXSLT extensions. I am sure it could have been rewritten in XSLT2 but I thought it would be  interesting to try it in XQuery ( and add a RESTXQ web interface). Demo and  source.



Thursday, 26 July 2012

Updates, including Node 0.8

Upgraded ..
  • Node.js on velvet to 0.8.4
  • BaseX on velvet to 7.4 beta
  • RAIDiator on readynas to  4.1.9  (required add index.xml to /etc/frontview/apache/httpd.conf
  • RAIDiator on velvet to 4.2.21 

Saturday, 23 June 2012

Extracting metadata using Apache Tika

Looking at using Apache Tika to pull metadata from photos and mp3 files. It is a single jar that runs as a command line, GUI or http server.

MP3


java -jar ~andy/tika-app-1.1.jar 12\ Time\ Of\ The\ Season.mp3
<?xml version="1.0" encoding="UTF-8"?><html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="xmpDM:audioCompressor" content="MP3"/>
<meta name="xmpDM:releaseDate" content="1968"/>
<meta name="Content-Length" content="5632909"/>
<meta name="xmpDM:album" content="Odessey &amp; Oracle"/>
<meta name="xmpDM:artist" content="Zombies"/>
<meta name="Author" content="Zombies"/>
<meta name="xmpDM:genre" content=""/>
<meta name="xmpDM:logComment" content=""/>
<meta name="Content-Type" content="audio/mpeg"/>
<meta name="resourceName" content="12 Time Of The Season.mp3"/>
<title>Time Of The Season</title>
</head>
<body><h1>Time Of The Season</h1>
<p>Zombies</p>
<p>Odessey &amp; Oracle, track 12</p>
<p>1968</p>
</body></html>
view raw gistfile1.xml hosted with ❤ by GitHub

JPEG

java -jar ~andy/tika-app-1.1.jar skating.jpg
<?xml version="1.0" encoding="UTF-8"?><html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="Software" content="Picasa"/>
<meta name="GPS Altitude Ref" content="Sea level"/>
<meta name="subject" content="greenwich"/>
<meta name="subject" content="ice"/>
<meta name="subject" content="skate"/>
<meta name="Content-Length" content="96154"/>
<meta name="Exif Version" content="2.20"/>
<meta name="date" content="2012-06-23T20:10:18"/>
<meta name="Component 1" content="Y component: Quantization table 0, Sampling factors 2 horiz/2 vert"/>
<meta name="tiff:ImageLength" content="768"/>
<meta name="Component 2" content="Cb component: Quantization table 1, Sampling factors 1 horiz/1 vert"/>
<meta name="GPS Latitude" content="51&quot;28'58.75122"/>
<meta name="Component 3" content="Cr component: Quantization table 1, Sampling factors 1 horiz/1 vert"/>
<meta name="GPS Latitude Ref" content="N"/>
<meta name="description" content="solange, ann and oliver ice skating collage"/>
<meta name="tiff:ImageWidth" content="1024"/>
<meta name="Image Width" content="1024 pixels"/>
<meta name="resourceName" content="skating.jpg"/>
<meta name="Keywords" content="greenwich"/>
<meta name="Keywords" content="ice"/>
<meta name="Keywords" content="skate"/>
<meta name="GPS Longitude Ref" content="W"/>
<meta name="GPS Longitude" content="0&quot;0'25.3368"/>
<meta name="Caption/Abstract" content="solange, ann and oliver ice skating collage"/>
<meta name="tiff:Software" content="Picasa"/>
<meta name="Number of Components" content="3"/>
<meta name="Image Height" content="768 pixels"/>
<meta name="Data Precision" content="8 bits"/>
<meta name="tiff:BitsPerSample" content="8"/>
<meta name="geo:lat" content="51.48299"/>
<meta name="Last-Modified" content="2012-06-23T20:10:18"/>
<meta name="Unknown tag (0xa420)" content="45925b38462f517f73a2bab9760ae5ac"/>
<meta name="Date/Time" content="2012:06:23 20:10:18"/>
<meta name="Directory Version" content="4"/>
<meta name="geo:long" content="-0.00704"/>
<meta name="GPS Version ID" content="2 2 0 0"/>
<meta name="Content-Type" content="image/jpeg"/>
<title/>
</head>
view raw gistfile1.xml hosted with ❤ by GitHub
Looking good.

Monday, 11 June 2012

Upgrading to dojo 1.7

Starting to upgrade the places website to dojo 1.7  migration-17
And trying to get to grips with the build system 

Issues

Web based code editors

Looking for a nice web based code edit component. I wanted XQuery and  Sparql  in addition to the usual suspects. The choice seems to be between Codemirror and Ace.

Codemirror supports both Sparql and XQuery out of the box.

Ace has very nice XQuery support initially developed for the eXide feature of eXist by Wolfgang Meier and then enhanced, and pushed back to the main  Ace project, by William Candillon. Ace currently has no official Sparql mode, but I spotted Callimachus  has the code  for a slightly earlier Ace release.

For me the deciding factor is that ACE is built with  AMD support, whereas Codemirror is not. There is a project to convert/wrap Codemirror for use with Dojo, which means using AMD,  but it looks like a lot of work.
Related projects to watch:   Treehugger and Dojo Widget for Ace

Tuesday, 5 June 2012

BaseX on Debian

Current install strategy:
  • unzip to /usr/local/share/basex-x.y 
  • link basex to basex-x.y
velvet:/usr/bin# 
ln -s /usr/local/share/basex/bin/basexclient
ln -s /usr/local/share/basex/bin/basexserver
ln -s /usr/local/share/basex/bin/basexhttp
ln -s /usr/local/share/basex/bin/xquery 
 
TODO init.d

Monday, 4 June 2012

Marklogic 5

I picked up a MarkLogic USB drive with 5.0.2 at XML Prague 2012. I installed this on Ubuntu but I wanted to get it to run on my Readynas Pro 2. This wish resulted in a factory reset on my first try, while attempting to install alien.

Older and wiser I tried again using chroot with a Debian Sarge image. The source file from the USB was MarkLogic-5.0-2.i686.rpm. Instructions based on
http://developer.marklogic.com/download#comment-354202122

To install

After doing debootstrap to create /c/home/squeeze
chroot /c/home/squeeze 

sudo apt-get install alien
sudo alien --to-deb --verbose MarkLogic-5.0-2.i686.rpm
sudo dpkg -i marklogic_5.0-3_i386.deb

To run

mount proc /c/home/squeeze/proc/  -t proc
cp /etc/resolv.conf /c/home/squeeze/etc
cp /etc/hosts /c/home/squeeze/etc
chroot /c/home/squeeze
/etc/init.d/MarkLogic start

The result 

 

Sunday, 3 June 2012

BaseX event handling

Trying to update BaseX-node to handle events.
It works the 1st time:

21:19:01.858 [127.0.0.1:38155] LOGIN admin OK
21:19:01.861 [127.0.0.1:38156] LOGIN test1 OK
21:19:02.003 [127.0.0.1:38155] CREATE EVENT messenger
21:19:02.005 [127.0.0.1:38155] OK 143.0 ms
21:19:02.027 [127.0.0.1:38156] OK 13.41 ms
21:19:02.247 [127.0.0.1:38156] QUERY(0) for $i in 1 to 1000000 where $i=3 return $i OK 219.51 ms
21:19:02.248 [127.0.0.1:38156] QUERY(0) OK 0.5 ms
21:19:02.252 [127.0.0.1:38155] QUERY(0) db:event('messenger', 'Hello World!') OK 230.45 ms
21:19:02.254 [127.0.0.1:38155] QUERY(0) OK 0.76 ms
21:19:02.584 [127.0.0.1:38155] EXEC(0) OK 329.22 ms
21:19:02.943 [127.0.0.1:38156] EXEC(0) OK 694.17 ms
21:19:02.945 [127.0.0.1:38156] OK 0.34 ms
21:19:03.004 [127.0.0.1:38155] DROP EVENT messenger
21:19:03.005 [127.0.0.1:38155] OK 2.18 ms
21:19:03.006 [127.0.0.1:38155] EXIT
21:19:03.011 [127.0.0.1:38155] OK 6.03 ms
21:19:03.012 [127.0.0.1:38155] LOGOUT admin OK
21:19:03.014 [127.0.0.1:38156] EXIT
21:19:03.019 [127.0.0.1:38156] OK 6.57 ms
21:19:03.020 [127.0.0.1:38156] LOGOUT test1 OK

Then it hangs..

21:19:16.106 [127.0.0.1:38158] LOGIN admin OK
21:19:16.112 [127.0.0.1:38158] CREATE EVENT messenger
21:19:16.115 [127.0.0.1:38158] OK 27.79 ms
21:19:16.151 [127.0.0.1:38159] LOGIN test1 OK
21:19:16.192 [127.0.0.1:38158] QUERY(0) db:event('messenger', 'Hello World!') OK 0.98 ms
21:19:16.200 [127.0.0.1:38158] QUERY(0) OK 7.4 ms
21:19:16.207 [127.0.0.1:38159] OK 0.65 ms
21:19:16.220 [127.0.0.1:38159] QUERY(0) for $i in 1 to 1000000 where $i=3 return $i OK 7.77 ms
21:19:16.231 [127.0.0.1:38159] QUERY(0) OK 4.3 ms
21:19:16.253 [127.0.0.1:38158] EXEC(0) Error: null
21:19:16.732 [127.0.0.1:38159] EXEC(0) OK 459.42 ms
21:19:20.036 [127.0.0.1:38158] LOGOUT admin OK
21:19:20.042 [127.0.0.1:38159] LOGOUT test1 OK
What is that error?

Sunday, 27 May 2012

fuseki

Installed Fuseki 0.2.2 snapshot from here onto velvet. (/c/home/jena/fuseki-0.2.2/fuseki-server)


mkdir dataDir
 ./fuseki-server --update --loc=dataDir /myDataset


11:10:37 INFO  Server :: TDB dataset: directory=dataDir
11:10:38 INFO  Server :: Dataset path = /myDataset
11:10:38 INFO  Server :: Fuseki 0.2.2-SNAPSHOT 20120526-0501
11:10:38 INFO  Server :: Jetty 7.x.y-SNAPSHOT
11:10:38 INFO  Server :: Started 2012/05/27 11:10:38 GMT on port 3030

One goal is  Jena SKOS inferencing

Sunday, 13 May 2012

Virtualised Debian Squeeze on Readynas

 After burning my Readynas through reckless upgrades I noticed some comments referencing chroot. This promised a way to have an up to date system and protection from rogue packaging problems.

root@velvet:/# cat /etc/debian_version
4.0
 
apt-get install debootstrap 
cd /usr/share/debootstrap/scripts
ln -s lenny squeeze
debootstrap squeeze /c/home/squeeze

chroot /c/home/squeeze



root@velvet:/# cat /etc/debian_version
6.0.4

All very simple and very nice.

Wednesday, 9 May 2012

Readynas recovery

Trying to install Alien on my Readynas Pro 2, which runs the ancient Debian etch release, corrupted the os and needed a rebuild of my environment. This entry documents the recovery steps.

restoring ssh access

Java 

 Java install seems to have got a lot more complicated since the initial install a year or so back.  I could not find an apt package. So manual download from:

http://www.java.com/en/download/linux_manual.jsp?locale=en
Linux (self-extracting file) filesize: 32.8 MB

to  /opt/java/jre1.7.0_04, then  try to  patch up a bit...

update-alternatives --install "/usr/bin/java" "java" "/opt/java/jre1.7.0_04/bin/java" 1 
update-alternatives --set java /opt/java/jre1.7.0_04/bin/java


(Some related stuff, did not use options  or more)

 BaseX


The BaseX package  required a forced install, due to the broken java..

velvet:/tmp# dpkg --force-depends -i bas*.deb
(Reading database ... 21903 files and directories currently installed.)
Preparing to replace basex 7.2.1-1 (using basex_7.2.1-1_all.deb) ...
Unpacking replacement basex ...
dpkg: basex: dependency problems, but configuring anyway as you request:
 basex depends on default-jre | java6-runtime; however:
  Package default-jre is not installed.
  Package java6-runtime is not installed.
Setting up basex (7.2.1-1) ...
velvet:/tmp# basex
[warning] /usr/bin/basex: No java runtime was found
[warning] /usr/bin/basex: Unable to locate /usr/share/java/tagsoup.jar in /usr/share/java
[warning] /usr/bin/basex: No JAVA_CMD set for run_java, falling back to JAVA_CMD = java
Saving properties in "/root/.basex"...

Installed tagsoup from the deb file
Updated my init.d script (based on the Redis one). And invoke at boot, see

Node.js

https://github.com/joyent/node/wiki/Installation easy :-)

npm install -g express
npm install -g ejs
npm install -g forever

Redis

Saturday, 5 May 2012

Upgrading basex on debian

velvet:/c/home/node# wget http://ftp.us.debian.org/debian/pool/main/b/basex/basex_7.2.1-1_all.deb
--2012-05-05 11:15:06--  http://ftp.us.debian.org/debian/pool/main/b/basex/basex_7.2.1-1_all.deb
Resolving ftp.us.debian.org... 35.9.37.225, 64.50.236.52, 128.30.2.36, ...
Connecting to ftp.us.debian.org|35.9.37.225|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2317708 (2.2M) [application/octet-stream]
Saving to: `basex_7.2.1-1_all.deb'

100%[======================================>] 2,317,708    635K/s   in 4.1s    

2012-05-05 11:15:11 (547 KB/s) - `basex_7.2.1-1_all.deb' saved [2317708/2317708]

velvet:/c/home/node# ls
bahblah-node_xslt-34c6509  BaseXRepo flight-stream  paste-bin
basex_7.2.1-1_all.deb    BXNfactbook local        retwis-js
BaseXData     faye  localbuild     soccore
basex-node     faye.old nohup.out      upload
velvet:/c/home/node# dpkg -i *.deb
(Reading database ... 28572 files and directories currently installed.)
Preparing to replace basex 7.1.1-1 (using basex_7.2.1-1_all.deb) ...
Unpacking replacement basex ...
Setting up basex (7.2.1-1) ...
velvet:/c/home/node# basex -v
[warning] /usr/bin/basex: Unable to locate /usr/share/java/tagsoup.jar in /usr/share/java
BaseX 7.2.1 [Standalone]
Try 'help' to get more information.

Currently the install does not provide httpserver support.

Sunday, 29 April 2012

listman

The BaseX 7.2.1 release brings together a couple of interesting features that I wanted to check out. A  RESTXQ implementation based on Adam Retter's XML Prague 2012 paper, and an approach to XQuery Update that allows a return value as well as persisting XML changes. It seemed like fun to try these with Alain Couthures XSLTForms

The result..


The code is in 3 main XQuery modules


The code is in bitbucket See readme.md for installation instructions. The BaseX RestXQ implementation is here

Todo: 
Get Xforms required css to work http://en.wikibooks.org/wiki/XSLTForms/CSS
track edit versions/conflicts








Wednesday, 11 April 2012

xsltforms

Does funny things to css..
<?xml-stylesheet href="/lib/xsltforms/xsltforms.xsl" type="text/xsl"?><?xsltforms-options debug="yes"?><!DOCTYPE html SYSTEM "about:legacy-compat">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ev="http://www.w3.org/2001/xml-events">
<head>
<title>Listman v0.0.1</title>
<base href="http://localhost:8984/restxq/listman/" />
<link href="/lib/bootstrap/css/bootstrap.css" rel="stylesheet" type="text/css" />
<link href="/lib/bootstrap/css/bootstrap-responsive.min.css" rel="stylesheet" type="text/css" />
<link href="/listman/app.css" rel="stylesheet" type="text/css" />
<!-- Le fav and touch icons -->
view raw gistfile1.xml hosted with ❤ by GitHub

<style type="text/css">XML Parsing Error: not well-formed Location: http://localhost:8984/lib/bootstrap/css/bootstrap-responsive.min.css Line Number 1, Column 10:.clearfix {*zoom:1;}
.clearfix:before, .clearfix:after {display:table;content:"";}
{}
</style>
view raw End up as.. hosted with ❤ by GitHub

The answer is <?css-conversion no?> See http://en.wikibooks.org/wiki/XRX/XSLTForms_and_eXist#Modifying_your_CSS_to_work_with_XSLTForms

Wednesday, 4 April 2012

ubuntu 11.10

Still a disaster. 
http://deviceguru.com/ubuntu-11-10-without-shell-shock/

Tuesday, 13 March 2012

upgrades

Upgraded thinkpad t42 to ubuntu 11.10. A disaster!
Managed to get rid of  Unity. Remove overlay scrollbars
 but onboard wifi still broken 
http://ubuntuforums.org/showthread.php?t=1496856&page=2

To checkout
android ide and http://raquelvelez.com/blog/2012/03/deploying-to-nodejitsu/

Friday, 9 March 2012

factbook progress

Switched back to passport.js from everyauth. I find Everyauth difficult to use and it seems to have jade dependency. A shame because it's register step and redirect control seems to be what I want.

Trying..
http://fortawesome.github.com/Font-Awesome/#

Still having performance problems. The borders display takes 35s for 613 borders.

Thursday, 16 February 2012

XQuery parser in Javascript performance

"even more scary is how lightning fast the exact same program is in javascript... "
@wcandillon

What are the numbers?

Gunther Rademacher's REX parser generator can target several languages including Javascript.

The Javascript output is designed for jrunscript. To see what it expects use the option:
Main program=simple
jrunscript did not work for me, nor did Rhino, probably  due to my lack of familiarity  with these tools.
I wanted to run using Node.js and this required a few changes to the javascript
  1. changing  the writeOutput function to capture the result 
  2. setting the module exports module.exports = xquery30
The node main module is:

var Xq=require("./xq3.js");
var out=[];
var parser=new Xq("2+3",out);
parser.parse_XQuery();
console.log(out.join("")); 

The numbers:

time node app.js
real 0m0.278s
user 0m0.212s
sys 0m0.020s
No doubt the c++ is faster. Perhaps the winner is EBNF.

Github: xqparserperf 

Payment service

Gumroad is a new payment service aimed at digital goods sales that works in over 190 countries. As someone who builds small web applications with subscription fees, I can honestly say that receiving payments is the least fun part of development. It’s fun getting paid, but dealing with payment provider APIs can be stressful to say the least. So I read with interest when Vadim Demedes sent us his Node Gumroad client (License: MIT, GitHub: vdemedes / node-gumroad, npm: gumroad).
dailyjs.com

 http://news.ycombinator.com/item?id=2406614

Monday, 13 February 2012

XQuery parser performance

This post was updated on 15th Feb with the BaseX 7.1.1 results

A comparison of XQuery engine performance running the XQuery parser from xquerydoc project. The test parses the XQuery program string "2+3":

import module namespace p="XQueryV30" at "XQueryV30.xq";
p:parse-XQuery("2+3")

Engines

  • Zorba XQuery Engine, Version: 2.1.0
  • BaseX 7.1.1 Beta  [Standalone]
  • Saxon-HE 9.4.0.2J
  • MXQuery 0.6.0

java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)


Running on Ubuntu 11.04 on a Thinkpad T42.

Zorba


time zorba -f -q test_xqparser.xq 
real 0m24.562s 
user 0m21.489s 
sys  0m0.240s

BaseX

Results for version 7.1.1 (BaseX711-20120215.234615)

time basex test_xqparser.xq
real 0m1.601s
user 0m1.260s
sys 0m0.088s
Results for version 7.1.0
time basex test_xqparser.xq
real 96m29.589s
user 54m35.961s
sys 0m19.533s

Saxon


time saxon-xq test_xqparser.xq 
real 0m2.673s
user 0m2.372s
sys  0m0.140s

MXQuery


java -Xms1024m -Xmx1024m  -jar mxquery.jar -f test_xqparser.xq
MXQuery 0.6.0
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

All scripts are run from the xqparserperf/src directory.

Github: xqparserperf 

(import-existing-source-code-to-github)

Sunday, 29 January 2012

No space on velvet

linux-no-space-left-on-device-error-with-50-free-space

velvet:/tmp# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md0 4184756 1722772 2252324 44% /
tmpfs 16 0 16 0% /USB
/dev/c/c 1930585268 906876 1929678392 1% /c
/dev/c/c_2011_07_08_22_18
1930585268 208984 1930376284 1% /c_2011_07_08_22_18
velvet:/tmp# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 65536 65536 0 100% /
tmpfs 126562 1 126561 1% /USB
/dev/c/c 30285824 40703 30245121 1% /c
/dev/c/c_2011_07_08_22_18
30285824 26 30285798 1% /c_2011_07_08_22_18
view raw gistfile1.sh hosted with ❤ by GitHub

Friday, 27 January 2012

Sunday, 22 January 2012

How to tag classical mp3 files

Basic Guidelines

  • The Artist should contain just the composer not the performer. For details, see below and ClassicalReleaseArtistStyle. 
  • The ReleaseTitle should include the title of the release followed by the major performer (i.e. the name of orchestra or quartet) inside parenthesis. For details, see below and ClassicalReleaseTitleStyle 
  • The TrackTitle should contain the overall work (name of the symphony etc.) followed by ':' and then the actual name of that movement. For details, see below and ClassicalTrackTitleStyle 
  • Convert the release to a VariousArtistsRelease if it contains works by more than one composer (see ConvertReleaseToMultipleArtistsEdit and ClassicalReleaseArtistStyle). 

http://musicbrainz.org/doc/Classical_Style_Guide

Thursday, 12 January 2012

Update basex on velvet

velvet:~# wget http://ftp.us.debian.org/debian/pool/main/b/basex/basex_7.0.2-1_all.deb
--2012-01-12 22:35:23-- http://ftp.us.debian.org/debian/pool/main/b/basex/basex_7.0.2-1_all.deb
Resolving ftp.us.debian.org... 35.9.37.225, 128.30.2.36
Connecting to ftp.us.debian.org|35.9.37.225|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2122208 (2.0M) [application/octet-stream]
Saving to: `basex_7.0.2-1_all.deb'
100%[======================================>] 2,122,208 542K/s in 4.3s
2012-01-12 22:35:27 (487 KB/s) - `basex_7.0.2-1_all.deb' saved [2122208/2122208]
velvet:~# ls *.deb
basex_7.0.2-1_all.deb
velvet:~# dpkg -i basex_7.0.2-1_all.deb
(Reading database ... 28580 files and directories currently installed.)
Preparing to replace basex 6.7-1 (using basex_7.0.2-1_all.deb) ...
Unpacking replacement basex ...
dpkg: warning - unable to delete old directory `/usr/share/maven-repo/org/basex/basex/6.7': Directory not empty
Setting up basex (7.0.2-1) ...
velvet:~# basex -v
/root/.basex: "JAXRXPATH" not found.
/root/.basex: "JAXRXPORT" not found.
BaseX 7.0.2 [Standalone]
view raw gistfile1.sh hosted with ❤ by GitHub

Sunday, 8 January 2012

tomcat on velvet


Install:
 apt-get install tomcat5.5 tomcat5.5-admin tomcat5.5-webapps
then
fix this issue tomcat-55-no-jdk-found


/etc/init.d/tomcat5.5 start
# Directory for per-instance configuration files and webapps
CATALINA_BASE=/var/lib/tomcat5.5

# Use the Java security manager? (yes/no)
TOMCAT5_SECURITY=yes

Change to
# Use the Java security manager? (yes/no)
TOMCAT5_SECURITY=no 

http://velvet:8180

Tuesday, 3 January 2012

npm changes

Changed to a global express install

 npm install -g express
 npm install -g ejs


Monday, 2 January 2012

node-inspector

Installed on thinkpad and velvet

velvet:~# npm install -g node-inspector
/usr/local/bin/node-inspector -> /usr/local/lib/node_modules/node-inspector/bin/inspector.js
node-inspector@0.1.10 /usr/local/lib/node_modules/node-inspector
├── paperboy@0.0.3
└── socket.io@0.8.7