LINUX JOURNAL PERSONAL DESKTOP • GreaseMonkey • Project Utopia • Ruby on Rails • 3-D Audio • she OCTOBER 2005 ISSUE 138
AN _
ssc Take the money and exec(): bil reader driver
PUBLICATION +
WHERE'S MY FILE?
Comparing file search tools
Fight spam
and win
with THE
BOOK OF
POSTFIX
l7
Managing your TODOs,
from Evolution to Emacs
An in-kernel FTP client
Decrypting shc-encrypted
shell scripts
An 8-speaker f
3-D sound system
Fixing other people's Web
sites with GreaseMonkey
USA $5.00 CAN $6.50
www.linuxjournal.com
1 0
o 71486 03102
JOURN
Since 1994: The Original Magazine of the Li
OCTOBER 2005
Community
PLUG IT IN AND_
1 illfi^D
yjyj v
Project Utopia makes
desktop hardware
play nice
ov't-o-p-ba.ncl a. clm'mis-f rcL-/ior> chi/cl's p/ccy
Systems
Aclminis-fraJ-or
AlterPath" Manager
AlterPath" BladeManager
AlterPath' OnSite
The Next-Generation IT Infrastructure
Cyclades AlterPath™ System is the industry's most comprehensive Out-of-Band Infrastructure (OOBI) system. The AlterPath
System allows remote data center administration, eliminating the need for most time-consuming, remedial site visits. When fully
deployed in your data center, Cyclades AlterPath System lowers the risks associated with outages, improves productivity and
operational efficiency, and cuts costs.
Each component of the AlterPath System is designed to seamlessly integrate into the enterprise, able to scale in any direction.
Whether you need serial console management of networking equipment, KVM for access to Windows® servers, branch
management, IPMI or HP iLO for service processor management or advanced power management, the AlterPath System delivers.
Cyclades brings it all together, making OOBI administration seem like child’s play.
Over 85% of Fortune 100
choose Cyclades.
www.cyclades.com/lja
1.888.cyclades • sales@cyclades.com
cyclades
= 2005 Cyrlodes Corporation All right! reserved All other trademarks and product images are property of their respective owners Produo information subject to change without notice
■ 11 nmnntyt»'i
tlMhlltllHlHI
otlHUHlHttlllt
;/v>; 3
co
V.wv.v
• t *T 1 *
f ' ( tj
Er?e^CT
8x AMD Opteron Processor 940 sockets
Supports 800 series Opteron CPUs with dual core tech.
Up to 128GB DDR Registered ECC memory
Support 4 Ranks memory module
1350W Redundant PSU 3+1
Support IPMI server management
Industry 19" rack-mountable 5U chassis
4 x Gigabit Ethernet ports, and 4 PCI-X slots
Up to 1 0 hot-swap HDDs with option HDD canister
Modularization design, I/O may vary
8-Way AMD Opteron Server Benchmark Rating
SPEC CPU2000: COMPILERS COMPARED
SPECint_rate_base2000
IWILL H8501 ►
Opteron sockets
IWILL H8501 Barebone System
= . V.V:.Vv;V.;>H
AMD
Opteron
--It ..
.. Mil ***
. .
.. ***** iMtumtimiiMMiti
‘V “HontiMUMimimimiKi,,,
•m rv **•*•*
•V rsV iVVAV
'«> WAV*
"I
Accommodate
128GB RAM
( IWILL Other Outstanding^ Motherboards & Small Form Factor: )
▼ DK8-HTX
EQE3
TECHNOLOGY
HTX allows access
into the AMD64 Direct
Connect Architecture
through a variety of
HTX connectors and
slots, integrating
outside devices
directly resident on
the HyperTransport
technology bus.
▼ DK8ES
▼ DK88
▼ ZMAX-DP / ZMAX-D2
Dual processors Small Form Factor
• 2 AMD Opteron Processor 940 sockets
• Supports 2xx Opteron CPUs with dual core tech.
• Up to 16GB DDR Registered ECC memory
• 1 x PCI-X 64bit 133/100/66MHz expansion slot
• 2 x PCI 64bit 100/66MHz expansion slots
• HTX-Pro support Pathscale InfiniPath HTX Adapter
• 2 x Gigabit Ethernet ports (Intel chipset)
• AGP 8MB on board & support IPMI
• 2 AMD Opteron Processor 940 sockets
• Supports 2xx Opteron CPUs with dual core tech.
• Up to 16GB DDR Registered ECC memory
• 3 x PCI-X 64bit 133/100/66MHz expansion slots
• 2 x PCI-Express x16 expansion slots
(one in PCI-Express x2 mode)
• 2 x Gigabit Ethernet ports (Broadcom chipset)
• AGP 8MB on board & support IPMI
• 2 AMD Opteron Processor 940 sockets
• Supports 2xx Opteron CPUs with dual core tech.
• Up to 64GB DDR Registered ECC memory
• 2 x PCI-Express x8 expansion slots
• 1 x PCI-X 64bit 133/100/66MHz expansion slot
• 1 x PCI-X 64bit 100/66MHz expansion slots
• 2 x Gigabit Ethernet ports (Broadcom chipset)
• AGP 8MB on board & support IPMI
• 2 AMD Opteron Processor 940 sockets
• Supports 2xx Opteron CPUs with dual core tech.
• Up to 4GB DDR Registered ECC memory
• lx AGP 8X, lx PCI and lx mini PCI slot (ZMAX-DP)
• 2x PCI-Express support SLI Tech.(ZMAX-D2)
• 3x 3.5" HDD bays, and lx 5.25" CD-ROM bay
• lx GbE, lx IEEE1394, 8x USB 2.0 ports
• 300W Power supply
IWILL USA Corp.
9004 Research Drive
Irvine, CA92618
Tel: +1 949 753-5488
Fax: +1 949 753-5499
Visit www.iwill.net for more information.
Or contact us: sales @ iwillusa.com, oem@iwHlusa.com
IWILL reserves the right to change specifications or other product information without notice. This publication could include technical
inaccuracies or photographical errors. IWILL provides this publication as is without warranty of any kind, either express or implied, including
the implied warranties of merchantability or fitness for a particular purpose. Some jurisdictions do not allow disclaimer of express or implied
warranties in certain transactions; therefore, this disclaimer may not apply to you.
Iwf 11
COVER STORY
66 PROJECT UTOPIA
Linux's long-standing tradition of isolating the user from the hardware might be
great for security, but it can be a real pain when you just want to snag some
photos from your camera or check for wireless access points you're allowed to
use. But there is a plan. This month, Robert Love covers Project Utopia.
LINUX
JOURNAL
OCTOBER 2005 ISSUE 138
FEATURES
52 FIXING WEB SITES WITH
GREASEMONKEY
This Web site is fine, but it could
really use....Redesign other people's
Web sites to your liking, on the fly.
NIGEL MCFARLAN E
60 THE LINUX FOR KIDS
EXPERIMENT
Can a Linux dad get his family moved
to a secure, easy-administration
box without giving up the fun
and education?
PAUL BARRY
66 PROJECT UTOPIA
Traditionally, Linux has protected
the hardware from the user for
security. When apps need to
understand the hardware, new
modes of communication are arising.
ROBERT LOVE
INDEPTH
72 BUILDING A CALL CENTER
WITH LTSP AND SOFT
PHONES
You don't need to put a phone and
a computer at every desk. Use a
soft phone on an almost-thin client.
MICHAEL GEORGE
78 DIRT CHEAP 3-D SPATIAL
AUDIO
Look out! Bogey at 10 o'clock high!
Your next simulator project can
have realistic sound above, below
and on all sides of the user.
ERIC KLEIN, GREG S. SCHMIDT,
ERIK B. TOMLIN AND
DENNIS G. BROWN
88 TAMING THE TODO
Is your computer helping you get
work done, or making more work
for you? Try these software options
to get your act together.
SACHA CHUA
COVER PHOTO:
BETHANY PASEMAN
EMBEDDED
42 DEVELOPMENT OF A
USER-SPACE APPLICATION
FOR AN HID DEVICE,
USING LIBHD
We won't show you the money, but
we'll show you the code for the
device that shows you the money.
EOIN VERLING
TOOLBOX
14 AT THE FORGE
Ruby on Rails
REUVEN M. LERNER
22 KERNEL KORNER
Network Programming in the Kernel
PRADEEP PADALA AND
RAVI PARIMI
34 COOKING WITH LINUX
Trekking through the Desktop Jungle
MARCEL GAGNE
38 PARANOID PENGUIN
Limitations of she, a Shell Encryption
Utility
NALNEESH GUAR
COLUMNS
47 LINUX FOR SUITS
The Only Silo
DOC SEARLS
96 EOF
The Universal Internet Time Source
ADRIAN VON BIDDER
REVIEW
71 THE BOOK OF POSTFIX
DON MARTI
DEPARTMENTS
4 FROM THE EDITOR
6 LETTERS
10 UPFRONT
70 NEW PRODUCTS
81 ADVERTISERS INDEX
95 MARKETPLACE
NetworkManager gets notifications of new
network hardware and available access
points, so all you have to do is find a hotspot
(page 66).
NEXT MONTH
HACK ANYTHING
Internet radio doesn't have to tie you
to the computer. Dan Rasmussen, Jon
Morgan and Paul D. Norton have
updated a classic radio design with
the ability to tune in Internet streams.
Stuart Brorson covers the electronic
design automation tools needed to
work with schematics and crank out
professional-looking board designs
that you can build yourself or order
from a PCB house.
If you think your favorite pinball
machine is complicated now, try
interfacing it to a Linux box. John Bork
covers digital I/O technigues to help
you connect to useful devices such as
solenoids and switches.
WWW.LINUXJOURNAL.COM OCTOBER 200513
rm -rf /opt/bs
A Linux desktop shouldn't be a kick in the teeth.
BY DON MARTI
T o understand the IT indus¬
try, start with On Bullshit
by Harry G. Frankfurt. Prof.
Frankfurt poses, but doesn’t
answer, the question of why there is
so much B.S. in our society. He
compares his subject to shoddy con¬
struction, and that’s an analogy we
can work with, because in software
we’re working at the thrilling edge
of language and craftsmanship. We
have the tools for dealing with B.S.
in computer languages. Try to B.S. a
compiler and that’s a bug. It’s time
to tackle the B.S. problem head-on
and start reporting bugs in human
communications too.
Consider this filler, I mean
essay, to be a bug report on the
big companies that are doing
Linux for the desktop. “Let’s
‘position’ Linux as a simplified
desktop for ‘transactional users’”,
they say. That’s right—employees,
if your company gives you Linux,
that means Management thinks
you’re a human servlet. Decision¬
makers and content creators get a
proprietary desktop OS.
Of course, offending the
employees’ pride might not show
up on a TCO spreadsheet. But no
executive would want to admit to
running a division full of transac¬
tional, replaceable, outsourceable
“human resources”.
But what about Clayton
Christensen, disruptive innovation
and The Innovator’s Dilemmal
Doesn’t the cheap, good-enough
contender always grow the fea¬
tures and stability it needs to win?
Yes, when it lets in the customers
left pressing their noses against
the Expensive Stuff Store window.
In the 1980s Macintosh let you do
layouts even if you couldn’t afford
phototypesetting. In the 1990s
Linux let you put up a Web server
without blowing the price of a
Coupe de Ville on a UNIX box.
But selling less-capable prod¬
ucts to customers who can get the
good stuff doesn’t fly. Seen an
F-20 at an air show lately? It was a
capable airplane, but it was posi¬
tioned as an “export fighter” for air
forces that weren’t allowed to
have, or couldn’t afford, the F-16.
Naturally, countries held out for
the “real” fighter. Information free¬
dom ideals can go only so far when
vendors patronize Linux cus¬
tomers. “Aww, the little transaction
worker filled out a Web form! Isn’t
that cute?”
Desktop Linux marketing is
doing more harm than good, but
work is under way to make Linux
out-perform the other OSes. Robert
Love’s Project Utopia is bringing
together the desktop interface and
the necessary tweaking of hardware
to make things work smoothly, not
just securely (page 66).
Michael George has an example
of how a thin-client environment
almost works to solve a problem,
but the project needed one key
local app, the soft phone. See a
hybrid approach to a VoIP station
that works as a phone and a PC on
page 72.
One of the projects where soft¬
ware excellence, not transaction-
workerism, has triumphed, is
Mozilla Firefox. Mozilla expert and
author Nigel McFarlane died last
month, leaving us with one last
article (page 52). Let Firefox serve
as an example for the standards the
desktop is coming to meet because
all B.S. aside, it has to@
Don Marti is editor in chief of Linux
Journal.
OCTOBER 2005
ISSUE 138
EDITOR IN CHIEF Don Marti, ljeditor@ssc.com
EXECUTIVE EDITOR Jill Franklin, jill@ssc.com
SENIOR EDITOR Doc Searls, doc@ssc.com
SENIOR EDITOR Heather Mead, heather@ssc.com
ART DIRECTOR Garrick Antikajian, garrick@ssc.com
TECHNICAL EDITOR Michael Baxter, mab@cruzio.com
SENIOR COLUMNIST Reuven Lerner, reuven@lerner.co.il
CHEF FRANCAIS Marcel Gagne, mggagne@salmar.com
SECURITY EDITOR Mick Bauer, mick@visi.com
CONTRIBUTING EDITORS
David A. Bandel • Greg Kroah-Hartman • Ibrahim Haddad •
Robert Love • Zack Brown • Dave Phillips • Marco Fioretti •
Ludovic Marcotte • Paul Barry • Paul McKenney
PROOFREADER Geri Gale
VP OF SALES AND MARKETING Carlie Fairchild, carlie@ssc.com
MARKETING MANAGER Rebecca Cassity, rebecca@ssc.com
INTERNATIONAL MARKET ANALYST James Gray, jgray@ssc.com
REGIONAL ADVERTISING SALES
NORTHERN USA: Joseph Krack, +1 866-423-7722 (toll-free)
EASTERN USA: Martin Seto, +1 905-947-8846
SOUTHERN USA: Laura Whiteman, + 1 206-782-7733 x119
INTERNATIONAL: Annie Tiemann, +1 866-965-6646 (toll-free)
ADVERTISING INQUIRIES ads@ssc.com
PUBLISHER Phil Hughes, phil@ssc.com
ACCOUNTANT Candy Beauchamp, acct@ssc.com
LINUX JOURNAL IS PUBLISHED BY, AND IS A REGISTERED
TRADE NAME OF, SSC PUBLISHING, LTD.
PO Box 55549, Seattle, WA 98155-0549 USA • linux@ssc.com
EDITORIAL ADVISORY BOARD
Daniel Frye, Director, IBM Linux Technology Center
Jon "maddog" Hall, President, Linux International
Lawrence Lessig, Professor of Law, Stanford University
Ransom Love, Director of Strategic Relationships, Family and Church
History Department, Church of Jesus Christ of Latter-day Saints
Sam Ockman, CEO, Penguin Computing
Bruce Perens
Bdale Garbee, Linux CTO, HP
Danese Cooper, Open Source Diva, Intel Corporation
SUBSCRIPTIONS
E-MAIL: subs@ssc.com • URL: www.linuxjournal.com
PHONE: +1 206-297-7514 • FAX: +1 206-297-7515
TOLL-FREE: 1-888-66-LINUX • MAIL: PO Box 55549, Seattle, WA
98155-0549 USA • Please allow 4-6 weeks for processing
address changes and orders • PRINTED IN USA
USPS LINUX JOURNAL (ISSN 1075-3583) is published monthly by
SSC Publishing, Ltd., 2825 NW Market Street #208, Seattle, WA
98107. Periodicals postage paid at Seattle, Washington and at
additional mailing offices. Cover price is $5 US. Subscription rate
is $25/year in the United States, $32 in Canada and Mexico, $62
elsewhere. POSTMASTER: Please send address changes to Linux
Journal, PO Box 55549, Seattle, WA 98155-0549. Subscriptions
start with the next issue. Back issues, if available, may be ordered
from the Linux Journal Store: store.linuxjournal.com.
LINUX is a registered trademark of Linus Torvalds.
LINUX
JOURNAL
41 OCTOBER 2005 WWW.LINUXJOURNAL.COM
The Power of Choice
Command the game with your next I/O move.
Modularity. Scalability. Reliability. Cost-effectiveness.
These represent the solid foundations that SBE delivers to
OEMs for building innovative end solutions. Partnering with
SBE for networking and communications I/O solutions allows
you to take advantage of proven technology and field-tested
products designed to optimize performance for your unique
application needs.
SBE offers a full spectrum of interface cards, ranging from It
and T3 to Gigabit Ethernet and IPsec/SSL acceleration. These
boards are available in multiple form factors, including PCI, PMC,
and PTMC. Customers have the choice of buying these boards
individually or bundling any of the PMC/PTMC modules with our
intelligent core processing platforms to create a flexible, cost-
efficient blade solution ideal for serving demanding telecom
applications. Full Linux support is available on every board.
► Channelized T3
► 24-port T1/E1
► LAN/Ethernet
► Storage
► IPsec/SSL Encryption
► Blade platforms
► I/O and beyond...
Linux 0 * tWw'
flexibility on demand I 925-355-2000 I info@sbei.com I www.sbei.com
Ultimate Linux Box Cooling?
I was impressed by the desire to make a
quiet PC—more people should complain
to their OEMs/System integrators about
this—it is the only way it will be fixed
[“Ultimate Linux Box”, August 2005]. But
I am not sure that removing 100% of the
airflow is a positive thing for overall sys¬
tem performance or stability.
I’m used to reading the annual Ultimate PC
article in Maximum PC magazine each year,
and they give a great deal of detail and
many more pictures than the Ultimate
Linux Box article [August 2005]. However,
the big difference between the two articles
is that your Ultimate Linux Box does much
more customization.
More on the ULB
I have seen motherboards designed where the
processor power supply components can
exceed the design rating from the suppliers
without airflow. Even if things don’t go bad
enough to cause system stability issues, it can
damage the processor by allowing the CPU
voltage to go out of specification. Intel is so
concerned about this—they are telling moth¬
erboard manufacturers to add circuits to the
motherboard to monitor the temperature of
the processor power supply and modulate the
clock of the processor if things get too hot.
See Section 9.4 in download.intel.com/
design/Pentium4/guides/30235604.pdf.
Thanks—and looking forward to more sub¬
minute kernel compiles.
Robin
Whenever you experiment with any alternate
cooling method, always measure and log
temperatures. — Ed.
Linux/BSD Confusion
My child is almost two in the picture. The
laptop is running KDE on LreeBSD. He
quite plainly CALLED it “Linux”. My wife
and I both looked at each other and at him,
and he said it again. He has also said
“Ethernet”. We are afraid, very afraid.
Thanks for a (decade of a) great publication!
kurtseel
The big question is, if I want to replicate the
Ultimate Linux Box, where would I get
details on the custom-made power supply
cooler modifications?
I went to the Resources page, but many of
the things I would need to access require that
I’m a Linux Journal subscriber. I have sub¬
scribed to the Linux Journal in the past, but
for the last year, I simply run out and pur¬
chase Linux Journal.
Dean
Articles from that issue will he openly avail¬
able soon. We’ll look for more info on the
power supply mods. — Ed.
Ten Years of Progress
The shock has worn off from seeing my
name in Linux Journal , and I’m able to
write again. [See “Ten Years Ago in Linux
Journal ”, July 2005, page 14.] Has it been
ten years since “Novice to Novice”
appeared in Linux Journal ? It must be. I
stopped writing the column after my first
child was born and—bless it!—if it’s not
his tenth birthday already.
And how much has Linux changed (and
stayed the same) in ten years. Though I
haven’t quite made the switch away from
Microsoft, I did recently install Ledora
Core. In ten years, installation has vastly
improved since when I used version 0.99 of
Slackware. The Ledora installation was rela¬
tively fast and idiot-proof. Everything
worked except the modem, and although
I’m finally getting DSL installed, I wanted
modem access as backup. Turns out I have a
PCTel modem, which seems unsupported by
the 2.6+ kernel and by the drivers currently
out there in Webland.
(Hmmm, could be another “Novice to
Novice” here.)
But what’s blown me away about Linux are
the live CDs. Knoppix and the variations are
fantastic not only for emergencies but also
just to learn *nix, which is why I got
involved with Linux originally.
Ah well! Thanks for remembering me after
all these years. Yes, I still have the 0.99
Slackware CD with the grinning Bob. It just
seems right to keep it.
Dean
Become Boring and Pigeonholed,
Please
Hi. I’ve been a subscriber to LJ for several
years, and I’ve never figured out your
niche. Servers, or desktops? Sysadmins,
application programmers, system develop¬
ers or home users? Your intention seems
to be: appeal to everyone. Unfortunately
for me, you cover so many different
topics that there is very little in each issue
for me.
If you can’t figure out your niche, I’ll let my
subscription expire in January.
jh
When different areas of information technol¬
ogy can stop learning from each other, we’ll
pick a “niche”. — Ed.
Ergonomic Comments on Ultimate
Linux Box
The case on the cover for your Ultimate
Linux Box is gorgeous [August 2005].
Beautiful. Amazing.
It also blows. It’s a terrible design. Ghastly.
Ideal for a computer show, but awful to use.
There’s no leg room on the box. Lor many
folks, that would mean sitting obliquely or
too far from the keyboard—both would
cause strain.
610CTOBER 2005 WWW.LINUXJOURNAL.COM
Reduce Your Deployment and Support Costs
MBX is the leader for your server and appliance manufacturing needs
Supermicro 5013G-MB
• Intel® Pentium 4 Processor® at 3.0EGHz
• 1U Rackmount Chassis
• 512MB PC3200 DDR
• Maxtor 80GB Serial ATA Hard Drive
• Dual Onboard Gigabit NIC’s
Includes CDROM, Floppy and Video
Lifetime toll free tech support
3 Year Warranty
$959
or lease for $33/mo.
Or Promote Your Brand
• Same Configuration as Above
• Custom Branded With Your Logo
• Worldwide Deployment and Support
• Custom Branded Packaging Available
• Configurations in 2U and 4U Available
• Custom OS and Software Install
• Custom Chassis Color Available
• No Minimum Quantity Required
$999
or lease for $38/mo.
MBX is the leader in custom appliances. Many premier application developers have chosen MBX as
their manufacturing partner because of our experience, flexibility and accessibility. Visit our website or
better yet, give us a call. Our phones are personally answered by experts ready to serve you.
MBX"
MOTHERBOARDEXPRESS
www.mbx.com
1.800.688.2347
Intel, Intel Inside, Pentium and Xeon are trademarks and registered trademark of Intel Corporation or its subsidiaries in the United States and other countries. Lease calculated for 36 months, to approved business customers. Prices and
specifications subject to change without notice. Setup fee may apply to certain branding options. Motherboard Express Company. 1101 Brown Street Wauconda, IL. 60084.
Tactile response acts as a brake reducing the
amount of impact on the end of the fingers
(while, agreed, increasing the finger travel),
so the keyboard used may actually increase
hand stress for some. Its placement isn’t
adjustable for height, and that can be catas¬
trophic for arm stress.
On a much less important note, I’d point
out that the case puts the cooling up very
high indeed, into airspace often several
degrees hotter than the rest of the room.
Paul Pomerleau
The coolant loops need to run well above the
motherboard in order to get adequate con¬
vective cooling. For daily use, you can build
a tall case without the monitor mounts or
keyboard shelf. — Ed.
Another Happy Reader
Here’s a photo of my son Merit (about 26-
months old) sitting on his trusty fire truck
checking out my July 2005 issue of LJ.
When he was done, he went back into the
office and picked up an Advanced C
Programming book!
tim
Pipe Tip
“Text Manipulation with sed” by Larry
Richardson had some useful hints [July
2005]. For instance, I wasn’t aware of the !
modification to the range field.
But writing to a file at the same time you are
reading it is decidedly dangerous. You are
depending on the pipe buffer between cat and
sed to hold the entire contents of the file.
You are also banking on the assumption that
cat will be started and allowed to fill its
buffer before the file is written and, there¬
fore, truncated at zero length. A far better
way to do this is in two steps:
sed -e ' s/$/ mycomputer/' < \
/etc/exports > tempfile \
&& mv tempfile /etc/exports
The mv command is executed only if sed
returns without problems. You don’t want to
be overwriting important files with the
wrong data!
Allen Brown
Her First Computer T-Shirt
My daughter (2.2-years old) made the transi¬
tion. She was introduced to Potato Guy and
Tux Paint on my Linux box. On this day, she
learned everything she needed to know
about the mouse operations. A very proud
moment for me, she is on the road to becom¬
ing a geek just like her daddy. Elizabeth Su
WOHID Certified (Wireless Optical Human
Interface Device), T-shirt awarded shortly
after. You can’t start them too young. This is
only the beginning for them. I am just happy
to be there.
Photo of the Month: a Linux Father's Day
I had a great Father’s Day and hope all the
other open-source dads did too. Of course,
the best gift was Paige, my going-to-be-
seven-in-September daughter—and you
will see she painted Tux on a rock for me
this Father’s Day. She always has been a
good drawer, and her favorite program is
Tux Paint with Tux Racer a close second.
James M. Susanka
Photo of the month gets you a one-year
subscription or extension. Send photos to
ljeditor@ ssc.com.
Jesse Apple
LETTERS CONTINUED ON PAGE 94
8IOCTOBER 2005 WWW.LINUXJOURNAL.COM
The Stmight MkPeople
SINCE 1991
ABERDEEN
YOU CAN BUY THESE NOW
OR WAIT TILL DELL FREEZES OVER
iBERDEEN STONEHAVEN A141
1U Dual Opteron™ 4 SATA/SCSI
High performance dual server for top-of-the-line
processing power with ultra-dense storage
capacity.
• Dual AMD Opteron™ Processors w/HyperTransport
and 1MB Cache
• AMD 8000 Series Chipset w/64-bit Support
• Up to 16GB DDR-400 Reg. ECC Memory
• Up to 4 x 400GB (1,6TB) Hot-Swap SATA or
4 x 300GB (1.2TB) Hot-Swap SCSI Drives
• 400W AC Power Supply w/PFC
• 5-Year Limited Warranty
iBERDEEN STONEHAVEN A261
2U Dual Opteron™ 6 SATA/SCSI
The highest performing 2U server available for the
money. “Staggering ... Powerhouse Performance ...
Highest Webbench numbers we've seen to date” -
PC Magazine, December 27, 2004.
• Dual AMD Opteron™ Processors w/HyperTransport
and 1MB Cache
• AMD 8000 Series Chipset w/64-bit Support
• Up to 16GB DDR-400 Reg. ECC Memory
• Up to 6 x 400GB (2.4TB) Hot-Swap SATA or
6 x 300GB (1.8TB) Hot-Swap SCSI Drives
• 460W Hot-Swap Redundant Power Supply
• 5-Year Limited Warranty
3U Dual Opteron™ 8 SATA/SCSI
Gargantuan storage beast with a capacity of up to
3.2TB, room for a dual-height tape drive, at an
incomparable cost/TB ratio.
• Dual AMD Opteron™ Processors w/HyperTransport
and 1MB Cache
• AMD 8000 Series Chipset w/64-bit Support
• Up to 16GB DDR-400 Reg. ECC Memory
• Up to 8 x 400GB (3.2TB) Hot-Swap SATA or
8 x 300GB (2.4TB) Hot-Swap SCSI Drives
• 760W Hot-Swap Redundant Power Supply
• 5-Year Limited Warranty
Starting at
$
1,895
Starting at
$
2,875
Starting at
$
2,975
ABERDEEN STONEHAVEN A124
1U Quad Opteron™ HPC
64-bit HPC environment workhorse server/cluster
node. Superior cooling with plenty of power to
handle any project.
• Cuad AMD Opteron™ 800 Series Processors
• AMD 8000 Series Chipset w/64-bit Support
• Up to 32GB DDR-400 Reg. ECC Memory
• Up to 2 x 300GB (600GB) SCSI Hard Drives
• 500W Power Supply
• Ultra Cool with Superb Air Flow
• 5-Year Limited Warranty
2U Quad Opteron™ 3 SATA/SCSI
Robust 64-bit server ideal for the HPC environment
as a high performance server. Able to provide all the
power and I/O for large databases and memory
intensive projects.
• Cuad AMD Opteron™ Processors w/HyperTransport
and 1MB Cache
• AMD 8000 Series Chipset w/64-bit Support
• Up to 32GB DDR-400 Reg. ECC Memory
• Up to 3 x 400GB (1,2TB) Hot-Swap SATA or
3 x 300GB (900GB) Hot-Swap SCSI Drives
• 700W Power Supply
• Ultra Cool with Superb Air Flow
• 5-Year Limited Warranty
4U Quad Opteron™ 8 SATA/SCSI
Best of both worlds, all-inclusive server with
enterprise-class 64-bit HPC Quad power along with
maximum storage capacity.
• Quad AMD Opteron™ Processors w/HyperTransport
and 1MB Cache
• AMD 8000 Series Chipset w/64-bit Support
• Up to 32GB DDR-400 Reg. ECC Memory
• Up to 8 x 400GB (3.2TB) Hot-Swap SATA or
8 x 300GB (2.4TB) Hot-Swap SCSI Drives
• 950W3+1 Hot Swap Redundant Power Supply
• Ultra Cool with Superb Air Flow
• 5-Year Limited Warranty
Quads Starting at
$
8,265
Quads Starting at
$
8,125
Quads Starting at
$
9,625
Trademarks are of their respective owners. IjOlO
888-297-7409
www flhprdppnino pom/linux
W W vvtCLL/V/X 1111LLA
To go along with this month's
theme of Personal Desktop, here
are some articles from the Linux
Journal Web site that will help
you find your way through
OpenOffice.org, try out some
Linux audio software and rescue
data from a hosed USB device:
» Do you want to move to
OpenOffice.org but aren't sure
what to expect? Are you trying
to convince friends and/or family
members to give OOo a try, but
they want to know about the
learning curve? If so, Bruce
Byfield's article "OOo Off the
Wall: What New Users Need to
Know About OpenOffice.org"
(www.linuxjournal.com/article/
8443) is suggested reading.
Bruce sheds some light on OOo's
"interface shortcomings" and
"the limits of its on-line help",
as well as the "logic of its inter¬
face design and the importance
of styles and templates in an
efficient work flow".
» Audio for Linux has come a long
way in the past couple of years,
and Dave Phillips continues his
tour of what's new for musicians
and engineers, whether full-time
or part-time. In recent months, he's
introduced us to Freewheeling, "a
powerful loop-based performance
tool" (www.linuxjournal.com/
article/8445), as well as
QSynth and QJackCtl, GUI front
ends that "make Linux audio
tasks easier and faster, letting
you get straight to the music"
(www.linuxjournal.com/
article/8354).
» Finally, Collin Park shares his
story of "Flow a Corrupted USB
Drive Was Saved by GNU/Linux"
(www.linuxjournal.com/article/
8366), offering hope to those of
us who have lost important data
and will lose it again.
What's New in Kernel Development
After a long and difficult life, DevFS is final¬
ly being removed from the Linux kernel.
Created by Richard Gooch, DevFS has been
around for years, and it represented a serious
attempt to cure the runaway /dev directory.
Developing DevFS was an uphill battle
against many detractors, but Richard did suc¬
ceed in creating a very useful tool. In the end,
however, critics of DevFS won out, citing
“unfixable races” and other problems, and
Richard vanished from kernel development
completely. Greg Kroah-Hartman and others
then developed udev as a replacement for
DevFS. Some lingering sense of the 2.6 kernel
as a stable tree has made this decision slightly
controversial even now, but almost certainly
it’s not enough to influence the outcome.
Farewell DevFS—it was a valiant effort.
Recently, various folks have reported com¬
pilation problems when trying to compile the
2.4 kernel with GCC version 4, and some
developers have posted patches to address these
issues; however, Marcelo Tosatti has stated that
it is simply too late in the day for these sorts of
patches to make it into the 2.4 tree. Unlike 2.6
development, the maintainers of 2.4, 2.2 and 2.0
have not decided to follow suit and abandon the
idea that their trees must aim for stability.
Marcelo has been trying to rein in 2.4 develop¬
ment ever since the first 2.6 kernel came out,
but he has still allowed large IDE changes, new
hardware support and other patches whose inva¬
siveness would typically fly in the face of a
push for stability. And with 2.6 development
showing no sign of slowing down, Marcelo has
been under constant pressure to incorporate new
features into 2.4 to be available to folks who
needed 2.4’s stability. With the advent of the
w.x.y.z tree, however, some of this pressure has
undoubtedly flagged, and Marcelo has been
able to tighten up the restrictions on what can
and cannot get into 2.4 at this late date.
The git versioning system continues to
grow and strengthen. Andrew Morton’s
-mm tree will be available as a git repository,
although Andrew himself has no plans to use
any versioning tool for actual development.
The ALSA Project has migrated development
to git, as has iibata. Marcelo Tosatti’s 2.4 tree
also will use git for ongoing development.
Linus Torvaids is still very strongly involved
with the project, and although mailing-list
traffic has tapered off somewhat from its fran¬
tic early weeks, much of this is explained by
the fact that folks now understand the basics
of the tool, and the fundamental concepts no
longer need to be explained to newcomers.
In the midst of all the version-control
upheaval, it’s hard to know for certain if the
new w.x.y.z stable kernels are working out. But
several kernel folks, including Jeff Garzik and
Alan Cox, feel that this tree successfully pro¬
vides a stable kernel to supplement the 2.6
tree’s ongoing large-scale development. Greg
Kroah-Hartman and Chris Wright, the primary
maintainers of the w.x.y.z tree, do seem to be
doing a rigorous job, not only collecting and
applying patches, but adhering to Linus
Torvaids’ strict guidelines on what patches may
be applied, and how and when they may be
accepted. A number of aspects make this project
less appealing than doing real development
work, but Chris and Greg seem to be bearing up
nicely, and the rest of us are the beneficiaries.
Martin J. Bligh has put together a set of
automatic testing scripts that compile and boot
all official kernel releases (including the w.x.y.z
kernels) and several prominent branches like the
-mm tree, within 15 minutes of their release. If
a kernel boots successfully, Martin’s scripts hit
it with a variety of benchmarks. Compilation
and boot results are recorded, benchmark results
are graphed and everything is made available as
a set of ongoing kernel.org Web pages. This is
the sort of project that will not solve all bugs,
but it will identify many trivial bugs, track per¬
formance problems across multiple kernel
releases and may identify hard-to-fmd bugs that
regular users would not normally see.
The relatively recent introduction of
Signed-Off-By tags in kernel patch submis¬
sions has made a huge difference in providing a
trail of authorship, so that if anything like the
SCO lawsuit occurs again, it will be easy to
prove who wrote any disputed source code.
This was, in fact, Linus Torvaids’ stated pur¬
pose in introducing the Signed-Off-By header.
When first introduced, the idea was quite amor¬
phous, with few details settled. Since then, var¬
ious wrinkles have been introduced to improve
its usefulness. One of the most recent of these
is the addition of a From header as the first line
of the body of patch e-mails. This header iden¬
tifies the tme author of a given patch. Before
this wrinkle, the tme author was assumed to be
the person with the bottom-most Signed-Off-By
header. This, however, became confusing and
was not always adhered to. The From header is
intended to leave no doubt as to the original
authorship of a given patch.
— ZACK BROWN
1 OB OCTOBER 2005 WWW.LINUXJOURNAL.COM
Cyc/ade s AlterPath™OnSite makes
branch c-f-fice administ ration child's p/&-y
Systems
Administrator
l'iyPa.4h **»*S‘ f
( in^ernei
r""'\
CDMA
Office
AlterPath OnSite
>
The Next-Generation IT Infrastructure
Cyclades AlterPath™ OnSite is the most comprehensive remote site and
branch office administration appliance available. This small, inexpensive
solution for controlling network equipment, servers and other IT infrastructure
devices can
• Access, diagnose and restore remote IT devices quickly
• Download software to multiple devices automatically and simultaneously
• Configure user information, system settings and operating parameters
• Send alerts of intrusions, equipment failures and alarms
The AlterPath OnSite combines the functionality of both serial console and KVM
over IP, allowing IT administrators to manage multiple servers and network
devices through a single appliance. Cyclades brings it all together making
remote site and branch office administration seem like child's play.
Over 85% of Fortune 100
choose Cyclades.
www.cyclades.com/ljb
1.888.cyclades ■ sales@cyclades.com
cyclades
C MOS Cptlndfli Corporamm AM nyhn raswwd AO crihnr trodaraarts and produfl images or* property o! far rwpedw* nmnn fodwt udnrmataa uAjkj id
Nigel McFarlane
With the sudden death of Nigel
McFarlane, the Web Development and
Open Source Software communities,
both in Australia and around the
world, have lost one of their most
well-known authors, consultants and
pundits.
Although in many ways a very pri¬
vate person, Nigel had a professional
and personal network that spanned
the globe and included such on-line
luminaries as the lead engineer for the
open-source browser Firefox Ben
Goodger, and countless others in the
Open Source, Web Development and
Linux communities. Since his passing,
many community sites, in a number of
languages, have expressed their sor¬
row, a testament to Nigel's influence.
A real Melbourne boy, describing
the city proudly as "the World's most
liveable", Nigel had science degrees
from both the University of
Melbourne and LaTrobe University.
Even when speaking in Sydney, he was
always keen to get home as soon as
possible, where he would bushwalk
and ramble, swim and surf.
Nigel forged a global reputation
from his beloved Melbourne, in a way
impossible until the 1990s. Many oth¬
ers have and will follow his lead, but as
with much of what he did, here, too,
Nigel was a pioneer. Since 1997, Nigel
had become well known and respect¬
ed in the Web Development and more
recently Open Source Technology com¬
munities through the publication of
several successful books on JavaScript,
Mozilla and most recently, the increas¬
ingly popular free open-source brows¬
er Firefox.
Two earlier books on JavaScript,
Instant JavaScript in 1997 and the co¬
authored Professional JavaScript in 2001,
are still considered by many to be
among the best books on the subject.
More recently, the benchmark Rapid
Application Development with Mozilla, and
Firefox Hacks carved out a place in the
increasingly important Open Source
community.
Nigel's writing extended to the
columns "Searching for Substance" for
InformIT, and articles for such publica¬
tions as Linux Journal, DevX, Builder.com,
CNet, The Age and the Sydney Morning
Herald. Nigel was an entertaining speak¬
er as well as a writer. I particularly recall
chairing a conference session that Nigel
presented late last year. Often confer¬
ence-goers are anxious to get early
places in the meal queue, but although
we had gone overtime for lunch, Nigel
captivated the room. When offered the
opportunity to break, the entire room
turned it down, glued as they were to
Nigel's presentation.
Generous with his time, energies
and knowledge, Nigel contributed to
mailing lists, newsgroups and forums,
as well as speaking to audiences large
and small at conferences and for user
groups. His reach went far beyond
HLA Adventure - Version 3.ID : hp: 8407 g: 242 exp: 281
http://nembers.tr ipod.con/' v panks/hlaadv.htnl
talk to dragon
The dragon bellows,'I an the last of ny kind. And here you are, a
dirty human, cone to slay me?! I breathe fire upon trees, rustling
the bad kani out of the poison forests and valleys of this forgotten
world. I keep ny peace inside these caves, only to be disturbed by
the wretched hunans which pollute this landscape! I was once happy
in ny silence here, yet you cone here to nake ne niserable! I won't
allow it. Knave! Prepare to feel ny wrath!!!'
The dragon noves about angrily, stomping the floor!
examine dragon
A large dragon with tough scales and powerful claws.
Vou are carrying:
lantern
rope
key
sword
longsword
broadsword
HLA Adventure
members.tripod.com/~panks/hlaadv.html
When Zork appeared on the scene in
the late 1970s, computer enthusiasts
from around the world were instantly
hooked on the interactive fiction
genre known fondly as the Text
Adventure game.
HLA Adventure is the latest in a
long line of public domain and free
software text adventures being
released by people all over the world.
It combines elements from MUDs,
Advanced Dungeons & Dragons and
J.R.R. Tolkien's famous
The Lord of the Rings.
Using verbs and nouns to communi¬
cate with the game world, the player
moves about HLA Adventure with but a
simple goal in mind: slay the menacing
dragon at the end of a large expanse
of caves. While solving this main quest,
the player is also presented with nine
other unique quests, which allow the
player to find items and equip
weapons, armor and a brightly lit
lantern. Even a magical flute plays a
role—useful in putting magical beasts
to sleep.
Players will encounter hellhounds,
werewolves, vampires, hobbits, ghosts,
barbarians and demigorgons. Talk to
creatures in the game with the TALK
TO command. Once you have acquired
the necessary armament and passed
the requisite number of quests, you
can then enter into the cave and slay
the dragon for good.
Despite some bugs in the game,
HLA Adventure is a solid, robust open-
121 OCTOBER 2005 WWW.LINUXJOURNAL.COM
Plug the Levanta Intrepid™ into your network and perform the most important
Linux management tasks in a fraction of the time you spend now. And gain
power and flexibility that you've never had before:
Levanta Intrepid™
™ Fast & Portable: Provision servers or workstations practically
anywhere, anytime - in minutes. Swap them around, mix it up.
™ Flexible: Supports commodity hardware, blades, virtual machines,
and even mainframes.
™ Out of the Box: Includes pre-defined templates for servers,
workstations, & software stacks. Or create your own.
™ Total Control: Track any file changes, by any means, at any time.
And undo them at will.
™ Disaster Recovery: Bring dead machines quickly back to life,
even if they're unbootable.
30-Day
Money-Back Guarantee
Order online by 10/30/05
Get $500 Off
Enter PROMO CODE: LJ1005
Based upon technology that's already been proven in Fortune 500
enterprise data centers. Now available in a box, priced for smaller
environments. Just plug it in and go.
© 2005 Levanta, Inc. All rights reserved. Levanta and the Levanta logo are registered marks of Levanta, Inc.
LEVANTA*
www.levanta.com
1 . 877. LEVANTA
Ruby on
Rails
Explore a Web development framework that comes
with its own Web server, magically keeps track of
details for you and integrates new code without
restarting, by reuven m. lerner
R uby, an interpreted programming language that looks
and feels like a cross between Smalltalk and Perl,
has been around for about ten years. Ruby has been
gaining in popularity over the last few years, partly
because of the release of English-language books and docu¬
mentation. In addition, programmers have become more inter¬
ested in finding an alternative to Perl and Python for their gen¬
eral-purpose programming needs.
Ruby’s popularity might have continued to grow slowly
were it not for Ruby on Rails, a Web development framework
that has become the focus of enormous attention. Everyone in
the Web development world seems to be talking about Rails;
magazine articles, blog postings, conference tracks and even
some new books all are dedicated to Rails. Rails is supposed to
be elegant, easy to use and easy to modify. Even developers
with no previous Ruby experience are switching to Rails.
Does Rails live up to the hype surrounding it? To a large
degree, I believe the answer is “yes”—it has a relatively shal¬
low learning curve, it connects easily and quickly to relational
databases and it makes the creation of many small- and medi¬
um-sized sites faster and easier than I would have expected.
But, of course, no framework is perfect, particularly one that
was released publicly only one year ago. It remains to be
seen whether Rails can hold up against more-established
technologies on several different fronts.
This month, we begin to look at several aspects of Ruby on
Rails, so you can decide for yourself if my assessment is accurate.
We begin by installing and configuring a basic Rails application.
Over the next few installments of At the Forge, we will extend our
application in several different ways, considering the ways in
which Rails allows us to create and modify our applications.
Installing Rails
The first step in creating a Rails application is to install Ruby
and then Rails itself. Most modern Linux distributions come
with Ruby, although only the latest released version as of this
writing (1.8.2) works with the most recent version of Rails
(0.12.1). New versions of Rails have been coming out fre¬
quently, which means that one or both of these versions might
have changed by the time you read this.
Assuming you have installed Ruby, you next need to install
Gems. It provides access to the Ruby Gems library, which is
something of a cross between SourceForge and Perl’s CPAN
(see the on-line Resources). Download and unpack the most
recent .tar.gz file:
tar -zxvf rubygems-0.8.10.tar.gz
Enter the directory as the root user and type:
ruby setup.rb all
This installs the entire Gems package. Among other
things, this installs the gem program in /usr/bin. You then
can install Rails, which is distributed via Gems, with the
following command:
gem install --remote rails
As with such systems as CPAN and Debian’s apt, the gem
program is smart enough to identify and download any depen¬
dencies it might encounter. By default, you need to answer “y”
explicitly when asked if you are interested in installing any
dependencies. Because Rails depends on a number of other
packages, you should be sure to answer “y” when prompted.
When you are returned to the shell prompt, you can assume
that Rails has been installed. However, this is not quite enough.
If you are interested in working with a relational database,
you also need to install a database interface library. Because
I work with PostgreSQL, I installed the pure Ruby client,
called postgres-pr:
gem install --remote postgres-pr
Somewhat confusingly, there also is a set of PostgreSQL
client libraries (called postgresql) that can be used with Ruby.
However, it seems as though most Rails developers are work¬
ing with the postgres-pr library, at least for now.
Creating an Application
Once Rails is installed, we can create a simple “Hello, world”
program. To do this, we use the rails command, which is
installed in /usr/bin/ by default. Because our example applica¬
tion is a Weblog, we call the application blog. For reference,
the name of the application doesn’t have to be linked to the
name of the URL under which it will appear. Type:
rails blog
Running this produces a fair amount of output, listing the
files that have been created on our filesystem. When we give
only a single name, blog, the application is created inside of a
directory with that name. We can keep all of our applications
inside of a single container directory, such as -/Rails, with:
mkdir -/Rails
rails -/Rails/blog
If we look inside the newly created application directory,
we see a number of directories and files. The script directory
contains administrative programs, written in Ruby, of course.
The public directory contains static HTML files, as well as
images, stylesheets, JavaScript code and templates that you
1 41 OCTOBER 2005 WWW.LINUXJOURNAL.COM
EmperorLinux
...where Linux & laptops converge
You choose your laptop ••• from a wide selection of top tier laptops manufactured by
IBM/Lenovo, Dell, Sharp, and Sony. They come in all sizes from two pound ultra-portables to eight
pound desktop replacements; get exactly as much Linux laptop as you need. Need help deciding?
Our experts will help you select a Linux laptop to meet your needs.
The Meteor: 3lb Linux
The SilverComet: 4 lb Linux
• Sharp Actius MM20/MP30
• 10.4" XGA screen
• 1.6 GHz Transmeta Efficeon
• 20-40 GB hard drive
• 512-1024 MB RAM
• CDRW/DVD (MP30)
• 802.1 lb/g wireless
• ACPI hibernate
• 1" thin
• Ask about the 3D Molecule
• Sony VAIO S380
• 13.3" WXGA+ screen
•X@l 280x800
• 1.6-2.13 GHz Pentium-M
• 40-100 GB hard drive
• 256-1024 MB RAM
•CDRW/DVD orDVD-RW
• 802.1 lb/g wireless
• ACPI hibernate
• Ask about the 17" Gazelle
You choose your distribution ••• from among the most popular Linux distributions
available. We'll install the distribution you select, then we'll install our custom, laptop-specific kernel
and configure your distribution for full hardware support, including: X at the native resolution,
wireless ethernet, power management, 3-D graphics, optical drives, and more.
The Toucan: 5 lb Linux
The Rhino: 7 lb Linux
• IBM/Lenovo ThinkPad T series
• U.l“ SXGA+/15.0" UXGA
• X@1400xl050/X@l 600x1200
• ATI FireGL graphics
• 1.6-2.13 GHz Pentium-M 7xx
• 40-80 GB hard drive
• 512-2048 MB RAM
• CDRW/DVD or DVD-RW
• APM suspend/hibernate
• Ask about the 3 lb Raven X41
• Dell Latitude D810/M70
• 15.4" WUXGA screen
• X@1920xl200
• NVidia Quadra or ATI Radeon
• 1.73-2.26 GHz Pentium-M 7xx
• 30-100 GB hard drive (7200 rpm)
• 256-2048 MB RAM
•CDRW/DVD or DVD±RW
• 802.1 la/b/g wireless, GigE
• Ask about the tiny Koala XI
To: ttuppurtCeinjKrorlinLOc.coi
Frau: ciiXtoBenJhaBebase.net
Subject: Configuration of
Just bought a new uirelj
to learn how to configj
my PC and ay laptop
anywhere in ay houM
And I still nent it to
Let EmperorLinux do the rest Since 1999, EmperorLinux has provided pre-installed
Linux laptop solutions to universities, corporations, and individual Linux enthusiasts. We specialize
in the installation and configuration of the Linux operating system on a wide range of the finest laptop
and notebook computers made by IBM/Lenovo, Dell, Sharp, and Sony. We offer a range of the latest
Linux distributions, as well as Windows dual boot options. All systems come with one year of Linux
technical support by both phone and email, and full manufacturers' warranties apply.
www.EmperorLinux.com 1-888-651-6686
Model prices, specifications, and availability may vary. All trademarks are the property of their respective owners.
YOUR
HIGH PERFORMANCE
COMPUTING SOLUTION
HAS ARRIVED.
VXRACK™ with the Intel® Xeon™ processor
helps you simplify computing operations,
accelerate performance and
accomplish more in less time.
Choose i
v I ) convene
one of the 3
convenient rack sizes
VXR-128
Rack accomodating up to
128 VXBJades/256 Processors
40TB of aggregated Storage
1.5TB of Global Memory
Power Distribution Included
Patented Architecture
Advanced Cooling System
Integrated InfiniBand Cable Mgnt.
$ 2,190.00*
VXR-96
Rack accomodating up to
96 VXBiades/192 Processors
36TB of aggregated Storage
1.1 STB of Global Memory
Power Distribution Included
Patented Architecture
Advanced Cooling System
Integrated InfiniBand Cable Mgnt.
$ 1,750.00*
VXR-72
Rack accomodating up to
72 VXBIades/144 Processors
27TB of aggregated Storage
864GB of Global Memory
Power Distribution Included
Patented Architecture
Advanced Cooling System
Integrated InfiniBand Cable Mgnt
$ 1,590.00*
TECHNOLOGIES
For more Information call
or visit us at
inlei, Mai logo, Intel inside, Intel insoe logo, Intel Centura),, Intel Genuine logo, Celeron, Intel ?tecn, ntal SpeedStep, Itanum, Pentium, enc
VXB-7221B
Intel SE7221B Motherboard
800MHz Front Side Bus
Intel® Pentium® 4 3.2GHz
1GB DDR2 400 Memory
Single 40GB 7200RPM ATA Drive
One PCI/Express Slot Available
DuaM 0/100/1000 Intel Lan Port
35QW Power Supply
$ 985.00
VXB-7501W
Intel SE7501W Motherboard
533MHz Front Side Bus
2 x Intel® Xeon™ 3.06GHz
2GB DDR 333 ECO Reg.Mem
Single 40GB 7200RPM ATA Drive
One PCI/X Slot Available
Dual 10/100/1000 Intel Lan Port
3SOW Power Supply
VXB-7520J
Intel SE7520J Motherboard
800MHz Front Side Bus
2 x Intel® EM64T Xeon™ 3.2GHz
2GB DDR2 400 ECO Reg.Mem
Single 40GB 7200RPM ATA Drive
One PCI/Express Slot Available
Dual 10/100/1000 Intel Lan Port
500W Power Supply
$ 2 , 950.00
Choose one or more
type of VXBIade
$ 2 , 355.00
Add, Mutiply,That’s it.
Easy as 1, 2, 3...
For example you choose the following: One VXR-96 with
48 Dual Intel® EM64T Xeon™ and 40 Single Intel® Pentium®4.
You take 1 (VXR-96) + 48 (VXB-7520J) + 40 (VXB-7221 B)...That’s it
THE FUTURE OF CLUSTER TECHNOLOGY
CIARA TECHNOLOGIES...A GLOBAL SOLUTION PROVIDER.
Ciara Technologies is a world-class computer systems manufacturer. Ciara designs, develops,
manufactures, markets, services, and supports a variety of computer systems Including graphic
workstations, rackmount and tower servers, networked storage and the newly acclaimed VXRACK™
Cluster Technology. The company's state of the art supercomputer cluster is based on the Intel IA32
and IA64 architectures and utilizes Linux operating systems. We are proud to be recognized by Intel as
an “Intel Premier Provider". Choosing Ciara is choosing a single point of contact for all your rr
requirements. All our products are built under the ISO 9001 standards and regulations. The growth of
Ciara enabled the company to move its 300+ employees,in February 2003, to an ultra-modern plant of
576,000 sqft.. Ciara now has the capability of producing more than 500,000 units per year.
866-7VX-RACK (866-789-7225)
WWW.VXRACK.COM
I Pontfum III Xeon are trademarks or registered trademarks c< Intel Corporation or its stieldlanes in the Untied States and other countries
Ja-fbr typo^Hf^iy grrarE, pfiotographes srrors, prfcirg srrara. AH pricteg fl LK cbtla. 3-^prig aid BpplicablB taxes are not relucted.
may use in your application.
The directory you are mostly likely to work with is app,
which contains the application itself. The app directory
contains subdirectories named models, views and con¬
trollers. This design reflects the fact that Rails uses the
MVC (model/view/controller) style widely used in many
modern desktop and Web applications.
In an MVC architecture, we divide our work into three
parts—the controller, which acts like a switchboard, invok¬
ing the appropriate model and view; the model, which con¬
tains the data and some of the logic; and the view, which
displays information to the user. If you have ever built a
database-backed site with PHP and Smarty templates or
with Zope and its Page Templates or even with Java and
JavaServer Pages (JSPs), you already are familiar with at
least some of these ideas. Rails simply makes them more
explicit with its prenamed directory structure.
Although it can’t do much, we now can start our empty
Rails application with:
cd ~/Rails/blog
ruby script/server
This starts the WEBrick HTTP server on port 3000. To
access this fairly empty Rails site, we point our browsers to
an appropriate IP address or hostname. In my particular
case, I started Rails on my test server, whose IP address
is 192.168.2.3. I thus point my Web browser to
http://l92.168.2.3:3000/. And sure enough, there I see a
“Welcome on board” message, indicating I have set up
Rails correctly.
Customizing the Behavior
Now that we know how to see the default message, let’s move
toward a “Hello, world” program. In Rails, there are two basic
ways to do this. We can create a controller that returns HTML
to the user’s browser, or we can create a view that does the
same. Let’s try it both ways, so that we can better understand
the relationship between controllers and views.
If all we want to do is include a simple, static HTML docu¬
ment, we can do so in the public directory. In other words, the
file blog/public/foo.html is available under WEBrick—started
by executing blog/script/server—at the URL/foo.html.
Of course, we’re interested in doing something a bit more
interesting than serving static HTML documents. We can do
that by creating a controller class and then defining a method
within that class to produce a basic “Hello, world” message.
Admittedly, this is a violation of the MVC separation that Rails
tries to enforce, but as a simple indication of how things work,
it seems like a good next step.
To generate a new controller class named MyBlog, we
enter the blog directory and type:
ruby script/generate controller MyBlog
Each time we want to create a new component in our Rails
application, we call upon script/generate to create a skele¬
ton. We then can modify that skeleton to suit our specific
needs. As always, Rails tells us what it is doing as it creates the
files and directories:
exists app/controllers/
exists app/helpers/
create app/views/my_blog
exists test/functional/
create app/controllers/my_blog_controller.rb
create test/functional/my_blog_controller_test.rb
create app/helpers/my_blog_helper.rb
Also notice how our controller class name, MyBlog, has
been turned into various Ruby filenames, such as
app/views/my_blog and app/helpers/my_blog_helper.rb. Create
several more controller classes, and you should see that all of
the names, like LooBar, are implemented in files with names
like foo_bar. This is part of the Rails convention of keeping
names consistent. This consistency makes it possible for Rails
to take care of many items almost magically, especially—as we
will see next month—when it comes to databases.
The controller that interests us is my_blog_controller.rb. If
you open it up in an editor, you should see that it consists of
two lines:
class MyBlogController < ApplicationController
end
In other words, this file defines MyBlogController, a class
that inherits from the ApplicationController class. As it stands,
the definition is empty, which means that we have neither
overridden any methods from the parent class nor written any
new methods of our own. Let’s change that, using the built-in
render_text method to produce some output:
class MyBlogController < ApplicationController
def hello_world
render_text "Hello, world"
end
end
After adding this method definition, we can see its results
by going to http://192.168.2.3:3000/MyBlog/hello_world.
Notice how the URL has changed: static items in the
public directory, such as our file foo.html, sit just beneath the
root URL, /. By contrast, our method hello_world is accessed
by name, under the controller class that we generated. Also
notice that we did not need to restart Rails in order to create
and test this definition. As soon as a method is created or
changed, it immediately is noticed and integrated into the
current Rails system.
If we define an index method for our controller class, we
can indicate what should be displayed by default:
class MyBlogController < ApplicationController
def hello_world
render_text "Hello, world"
end
def index
render text "I am the index!"
1 81 OCTOBER 2005 WWW.LINUXJOURNAL.COM
Let’s be ^ treme
Introducing Appro Xtreme Servers & Workstations
with 8 DIMM Sockets per CPU
2-way or 4-way, Single or Dual-Core AMD Opteron™ processors
Largest memory capacity - 8 DIMM Sockets per CPU, up to 128GB
PCI-Express technology to increase I/O bandwidth and reduce system latency
Outstanding Remote Management - IPMI 2.0 compliant
Cable-free design, ready to run, simple to install, service and maintain
Support for Windows® or Linux OS
Ideal for memory-intensive and l/O-intensive applications
1U/2UI3U Servers
and Workstation
AMD Opteron™ Processors - AMD64 dual-core technology reduces memory latency and increases data throughput
- Dual-core processors with Direct Connect Architecture deliver the best performance
per watt in the industry with little or no increase in power consumption or heat dissipation.
HPC Cluster Solutions
Appro delivers high-performance computing solutions to help you maximize productivity
for a solid ROI. On-site maintenance and installation services are also available.
For more information, please visit www.appro.com
or call Appro Sales at 800.927.5464, 408.941.8100.
end
end
Of course, it’s not that exciting to be able to produce static
text. Therefore, let’s modify our index method such that it uses
Ruby’s built-in Time object to show the current date and time:
def index
render_text "The time is now " + Time.now.to_s + "\n"
end
And voila! As soon as we save this modification to disk,
the default URL (http://192.168.2.3:3000/MyBlog/, on my
computer) displays the time and date at which the request was
made, as opposed to a never-changing “Hello, world” message.
Let’s conclude this introduction to Rails by separating the
controller from its view once again. In other words, we want to
have the controller handle the logic and the view handle the
HTML output. Once again, Rails allows us to do this easily by
taking advantage of its naming conventions. For example, let
us modify our index method again, this time removing its
entire body:
def index
end
This might seem strange at first glance. It tells Rails that
the MyBlog controller class has an index method. But it does¬
n’t generate any output. If you attempt to retrieve the same
URL as before, Rails produces an error message indicating that
it could not find an appropriate template.
Because the template is a view, we can define it inside of
the blog/app/views directory of our application. And because
we are defining the index view for the MyBlog class, we modi¬
fy the index.rhtml file in the my_blog subdirectory of views.
Notice how Rails turns ThisName into this_name when it
comes to directories. Doing so saves users from having to
think about capitalization in URLs, while staying consistent
with traditional Ruby class naming conventions.
.rhtml files are a Ruby version of the same kind of template
that you might have seen before. It acts similarly to ASP and
JSP syntax, with <% %> blocks containing code and <%= %>
blocks containing expressions that should be interpolated into
the template. However, nothing stops us from creating an
.rhtml template that actually is static:
Hello, again!
Hello, again!
Consider what happens now if you attempt to load
MyBlog in your browser. The controller class MyBlog is
handed the request. Because no method was named explicitly,
the index method is invoked. And because index doesn’t
produce any output, the my_blog/index.rhtml template is
returned to the user.
Finally, let’s take advantage of our template’s dynamic
properties to set a value in the controller and pass that along to
the template. We modify our index method to read:
def index
@now = Time.now.to_s
end
Notice how we have used an @ character at the beginning
of the variable @now. I found this to be a little confusing at
first, as @ normally is used as a prefix for instance variables
in Ruby. But it becomes fairly natural and logical after a
little time.
Finally, we modify our template such that it incorporates
the string value contained in @now:
Hello, world!
Hello, world!
11 is now <%= @now %>.
Once again, you can retrieve the page even without restart¬
ing Ruby. You should see the date and time as kept on the serv¬
er, updated each time you refresh the page.
Conclusion
Ruby on Rails is, without a doubt, one of the most talked-about
Web technologies to emerge in the past few years. This month,
we saw how straightforward it is to create a new Rails applica¬
tion, to create a controller and a view and to integrate them
using a combination of naming conventions and relatively stan¬
dard template syntax. However, we did not discuss views, par¬
ticularly those associated with a relational database. Next
month, we will do exactly that, connecting Rails to the
PostgreSQL database. I believe doing so will begin to show
why people are so excited about Rails and why it might be a
good tool for many Web developers to learn.
Resources for this article: www.linuxjournal.com/article/
8457.0
Reuven M. Lerner, a longtime Web/database con¬
sultant and developer, now is a graduate student in
the Learning Sciences program at Northwestern
University. His Weblog is at altneuland.lerner.co.il,
and you can reach him at reuven@lerner.co.il.
201 OCTOBER 2005 WWW.LINUXJOURNAL.COM
MORE SPACE. LESS MONEY.
Unlimited Affordable Network Storage
Everybody needs more space. And they need to
spend less money. What if you can both have more space
and spend less money?
What if you could put IV 2 terabytes in only 3 rack
units? What if that IV 2 terabytes cost less than $10,000? In¬
cluding the SATA disk drives. Imagine if you could glue it
all together with a RAID appliance into one system. What if
you could add as much storage as you wanted, one shelf at a
time, and never have to 'fork-lift’ anything?
Coraid’s new SATA EtherDrive Storage allows you
to do just that. Using industry standard SATA disk drives,
EtherDrive Storage connects disks directly to your Ethernet
network. Each disk appears as a local drive to any Linux,
FreeBSD or Solaris system using our open ATA-over-Eth-
ernet (AoE) protocol. Since the disks just appear as local
drives you already know how to use them.
The EtherDrive® SATA Storage Shelf is a 3U rack-
mount network appliance that contains 15 SATA drive slots.
Its triple redundant power supply protects you from your
most likely failure. Its dual Gb Ethernet interfaces allow
your data to go fast; 200MB per second. And at a very af¬
fordable price. List price for the EtherDrive Storage Shelf,
without disks, is only $3,995.
Our companion product, the RAIDBlade RAID
controller, allows a virtually unlimited number of Storage
Shelves to be combined into a set of logical AoE storage de¬
vices.
Now you can have unlimited storage at a very af¬
fordable price. For complete information, visit our website
at www.coraid.com, or call, toll-free, 1-877-548-7200. And
we’ll show how we’ve made network storage so affordable,
you can have all the space you want.
www.coraid.com
info@coraid.com
1.706.548.7200
CO RAID
Network
Programming in
the Kernel
Take a tour of the kernel's networking functionality by
writing a network client that runs in kernel space.
BY PRADEEP PAOALA AND RAVI PARIMI
11 Linux distributions provide a
wide range of network appli¬
cations—from daemons that
provide a variety of services
such as WWW, mail and SSH to client
programs that access one or more of these
services. These programs are written in
user mode and use the system calls pro¬
vided by the kernel to perform various
operations like network read and write.
Although this is the traditional method of
writing programs, there is another inter¬
esting way to develop these applications
by implementing them in the kernel. The
TUX Web server is a good example of an
application that runs inside the kernel and
serves static content. In this article, we
explain the basics of writing network
applications within the kernel and their
advantages and disadvantages. As an
example, we explain the implementation
of an in-kemel FTP client.
Advantages and Disadvantages of
In-Kernel Implementations
Why would one want to implement
applications within the kernel? Here are
a few advantages:
■ When a user-space program makes a
system call, there is some overhead
associated in the user-space/kernel-
space transition. By programming all
functionality in the kernel, we can
make gains in performance.
■ The data corresponding to any appli¬
cation that sends or receives packets
is copied from user mode to kernel
mode and vice versa. By implement¬
ing network applications within the
kernel, it is possible to reduce such
overhead and increase efficiency by
not copying data to user mode.
■ In specific research and high-perfor¬
mance computing environments, there
is a need for achieving data transfers
at great speeds. Kernel applications
are useful in such situations.
On the other hand, in-kernel imple¬
mentations have certain disadvantages:
■ Security is a primary concern within
the kernel, and a large class of user¬
mode applications are not suitable to
be run directly in the kernel.
Consequently, special care needs to
be taken while designing in-kernel
applications. For example, reading
and writing to files within the kernel
is usually a bad idea, but most appli¬
cations require some kind of file I/O.
■ Large applications cannot be imple¬
mented in the kernel due to memory
constraints.
Network Programming Basics
Network programming is usually done
with sockets. A socket serves as a com¬
munication end point between two pro¬
cesses. In this article, we describe net¬
work programming with TCP/IP sockets.
Server programs create sockets, bind
to well-known ports, listen and accept
connections from clients. Servers are usu¬
ally designed to accept multiple connec¬
tions from clients—they either fork a new
process to serve each client request (con¬
current servers) or completely serve one
request before accepting more connec¬
tions (iterative servers). Client programs,
on the other hand, create sockets to con¬
nect to servers and exchange information.
FTP Client-Server Interaction
Let’s take a quick look at how an FTP
client and server are implemented in user
mode. We discuss only active FTP in this
article. The differences between active
and passive FTP are not relevant to our
discussion of network programming here.
Socket Programming Basics
Here is a brief explanation of the design
of an FTP client and server. The server
program creates a socket using the
socket () system call. It then binds on a
well-known port using bi nd () and
waits for connections from clients using
the 1 i sten () system call. The server
then accepts incoming requests from
clients using accept () and forks a new
process (or thread) to serve each incom¬
ing client request.
The client program creates a control
socket using socket () and next calls
connect () to establish a connection with
the server. It then creates a separate
socket for data transfer using socket ()
and binds to an unprivileged port
client server
Figure 1. The FTP protocol uses two sockets: one
for control messages and one for data. Flere's how
the first connection, used for commands, gets set up.
221 OCTOBER 2005 WWW.LINUXJOURNAL.COM
The bolA7 craiL
Easy, high-performance clustering. For years, many searched, but none could
find it. Some said it didn't exist. But not the Penguin.
Penguin Computing® made easy, high-performance clustering a quest. Now
you can find Linux hardware and software solutions, configured to order,
driven by Scyld's commercially supported, industry-leading Linux clustering
software.
For the turnkey clusters you need to run even your most important applica¬
tions, come to Penguin Computing. Penguin Computing's dedicated experts,
who are 100% focused on Linux, are waiting to serve.
O
d)
Powerful, easy clustering. It's the once and future thing. Love what you do (^)
CL
E
o
.o
www.penguin
□□□□
mmnn
SCYLD
registered trademark of Penguin Computing, Inc. Scyld, Scyld Software, Scyld Beowulf, and the Scyld Block Logo are trademarks of Scyld Software, Inc. Linux is a registered trademark of Linus Torvalds. Other names are for informational purposes only and may be trademarks of their respective owners.
(>1024) using bi nd (). The client now 1 i sten () s on this port
for data transfer from the server. The server now has enough
knowledge to honor a data transfer request from the client.
Finally, the client uses accept () to accept connections from the
server to send and receive data. For sending and receiving data,
the client and server use the wri te () and read() or sendmsgQ
and recvmsgO system calls. The client issues close () on all
open sockets to tear down its connection to the server. Figure 1
sums it up.
FTP Commands
Here is a list of a few FTP commands we used. Because our
program provides only a basic implementation of the protocol,
we discuss only the relevant commands:
a TYPE I\r\n command to the server to request this.
Figure 2 is a diagram that shows a few FTP commands and
their responses from the server.
Socket Programming in the Kernel
Writing programs in the kernel is different from doing the
same in user space.
We explain a few issues concerned with writing a network
application in the kernel. Refer to Greg Kroah-Hartman’s arti¬
cle “Things You Never Should Do in the Kernel” (see the on¬
line Resources). First, let’s examine how a system call in user
space completes its task. For example, look at the socket ()
system call:
■ The client sends a USER \r\n command to the
server to begin the authentication process.
To send the password, the client uses PASS password\r\n'.
■ In some cases, the client sends a PORT command to inform
the server of its preferred port for data transfer. In such
cases, the client sends PORT \r\n.
The RFC for FTP requires that the al-a4 constitute the 32-
bit IP address of the client, and pl-p2 constitute the 16-bit
port number. For example, if the client’s IP address is
10.10.1.2 and it chooses port 12001 for data transfer, the
client sends PORT 10,10,1,2,46,225.
■ Some FTP clients request, by default, that data be trans¬
ferred in binary format, while others explicitly ask the serv¬
er to enable data transfer in binary mode. Such clients send
client 1 f server
USER parimi\r\n
331 Password required for parimi\r\n
PASS foobar\r\n
230 User parimi logged in\r\n
PORT a1,a2,a3.a4.p1.p2\r\n
200 PORT command successful
TYPE l\r\n
200 TYPE set to I
Figure 2. The client issues FTP commands over the control connection to set up
the file transfer.
sockfd = socket(AF_INET,S0CK_STREAM,0);
When a program executes a system call, it traps into the
kernel via an interrupt and hands over control to the kernel.
Among other things, the kernel performs various tasks, such as
saving contents of registers, making changes to address space
boundaries and checking for errors with system call parame¬
ters. Eventually, the sys_socket() function in the kernel is
responsible for creating the socket of a specified address and
family type, finding an unused file descriptor and returning this
number back to user space. Browsing through the kernel’s code,
we can trace the path followed by this function (Figure 3).
User Space
socketQ
Kernel Space
1. Save registers
2. Change address
space boundaries
3. Check system call
parameters for errors
4. Miscellaneous other
checks
1. sock_create()
sys_socket()- +~ 2 . sock_map_fd()
3. Get socket descriptor
Copy socket descriptor to user-space
Figure 3. Behind the scenes of a system call: when user space executes socket(),
the kernel does necessary housekeeping and then returns a new file descriptor.
Design of an FTP Client
We now explain the design and implementation of a kernel
FTP client. Please follow through the code available at the
Linux Journal FTP site (see Resources) as you read through
this article. The main functionality of this client is written in
the form of a kernel module that adds a system call dynamically
that user-space programs can invoke to start the FTP client pro¬
cess. The module allows only the root user to read a file using
FTP. The user-space program that calls the system call in this
module should be used with extreme caution. For example, it is
easy to imagine the catastrophic results when root runs:
./a.out 10.0.0.1 10.0.0.2 foo_file /dev/hdal/*
and overwrites /dev/hdal with a downloaded file from 10.0.0.1.
Exporting sys_call_table
We first need to configure the Linux kernel to allow us to add
new system calls via a kernel module dynamically. Starting
241 OCTOBER 2005 WWW.LINUXJOURNAL.COM
DON'T BE SQUARE!
GET CUBED!
L Series Laptop - LS125Q-L
Light &Thin Performance
Starting at $1,562.54
G Series Laptop - GW1550-L
Essential Technology on a Budget
Starting at 31,302.54
X Series Laptop - XW1550-L
Extreme Technology & Performance
Starting at $1,608.04
R Cubed Technologies has provided
pre-installed Linux laptops without OS
tax since 2003 We customize the
Fedora Linux distribution for each
laptops configuration providing support
for: PCMCIA, USB, FireWire, X,
CD/DVD/CDRW/DVDRW, Sound, Power
Management, Ethernet, Modem,
Wireless, and more. Our laptops are
equipped with Intel Centrino Mobile
Technology. We also otter Windows dual
boot options All of our laptops come
with a one year parts and labor warranty.
Visit us online at www.shoprcubed.com
or call 309.34.CUBED for details.
Technologies'
HarrJ to Br/Ov TeeAJMhltylfoS 10 Life.
309.34.CUBED
www.shoprcubed.com
All models, prices, and availability may vary. All trademarks are the prop*
©Copyright 2005
with version 2.6, the symbol sys_call_table is no longer
exported by the kernel. For our module to be able to add a sys¬
tem call dynamically, we need to add the following lines to
arch/i386/kernel/i386_ksyms.c in the kernel source (assuming
you are using a Pentium-class machine):
extern void *sys_caH_table;
EXPORT_SYMBOL(sys_call_table);
After recompiling the kernel and booting the machine into it,
we are all set to run the FTP client. Refer to the Kernel Rebuild
HOWTO (see Resources) for details on compiling a kernel.
Module Basics
Let’s examine the code for the module first. In the code snip¬
pets in this article, we omit error checking and other irrelevant
details for clarity. The complete code is available from the LJ
FTP site (see Resources):
#include
#include
#include
/* For socket etc */
#include
#include
#include
#include
#include
#include
#include
#include
#include
int ftp_init(void)
{
printk(KERN_INFO FTP_STRING
"Starting ftp client module\n");
sys_call_table[SYSCALL_NUM] = my_sys_call;
return 0;
}
void ftp_exit(void)
{
printk(KERN_INFO FTP_STRING
"Cleaning up ftp client module, bye !\n");
sys_call_table[SYSCALL_NUM] = sys_ni_syscall;
}
The program begins with the customary include directives.
Notable among the header files are linux/kernel.h for
KERN_ALERT and linux/slab.h, which contains definitions for
kmalloc() and linux/smp_lock.h that define kernel-locking rou¬
tines. System calls are handled in the kernel by functions with
the same names in user space but are prefixed with sys_. For
example, the sys_socket function in the kernel handles the
task of the socket () system call. In this module, we are using
system call number 223 for our new system call. This method
is not foolproof and will not work on SMP machines. Upon
unloading the module, we unregister our system call.
The System Call
The workhorse of the module is the new system call that per¬
forms an FTP read. The system call takes a structure as a
parameter. The structure is self-explanatory and is given below:
struct params {
/* Destination IP address */
unsigned char destip[4];
/* Source IP address */
unsigned char srcip[4];
/* Source file - file to be downloaded from
the server */
char src[64];
/* Destination file - local file where the
downloaded file is copied */
char dst[64];
char user[16]; /* Username */
char pass[64]; /* Password */
};
The system call is given below. We explain the relevant
details in next few paragraphs:
asmlinkage int my_sys_call
(struct params __user *pm)
{
struct sockaddr_in saddr, daddr;
struct socket *control= NULL;
struct socket *data = NULL;
struct socket *new_sock = NULL;
int r = -1;
char ^response = kmalloc(SNDBUF, GFP_KERNEL);
char *reply = kmalloc(RCVBUF, GFP_KERNEL);
struct params pmk;
if(unlikely(!access_ok(VERIFY_READ,
pm, sizeof(pm))))
return -EFAULT;
if(copy_from_user(&pmk, pm,
sizeof(struct params)))
return -EFAULT;
if (current->uid != 0)
return r;
r = sock_create(PF_INET, SOCK_STREAM,
IPPR0T0_TCP, &control) ;
memset(&servaddr,0, sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_port = htons(PORT);
servaddr.sin_addr.s_addr =
htonl(create_address(128, 196, 40, 225));
261 OCTOBER 2005 WWW.LINUXJOURNAL.COM
XEON.
Flexibility to power the enterprise.
From mail servers to databases, ZT Servers powered by the 64-bit
Intel® Xeon™ Processor can run the full range of 32-bit applications
and offer extended flexibility for your 64-bit needs. So you can create
powerful, all-purpose IT infrastructure that enhances business agility
- and the bottom line.
Intel®Xeon™ Processors3 GHz
- Upgradable to Dual Intel® Xeon™ Processors
3.60 GHz, 1MB L2 Cache, 800MHz FSB
■ Intel® E7320 Chipset ServerBoard
■ 512MB ECC Registered DDR 333 SDRAM (Upto4GB)
■ 2xSeagate® 300GB 10,000rpm SCSI Hard Drive (Raid l)
■ 4x1" SCSI Hot-Swappable Drive Bays
■ Slim CD-ROM & Floppy Drive
■ 2 x Intel® 10/100/1000 Gigabit Network Controller
■ 1U RackmountChassisw/420WCold-Swappable PowerSupply
■ 3-Year Limited Warranty and First Year On-Site Service
$ 3,999
Dual Intel® Xeon™ Processors 3GHz
- Upgradable to Dual Intel® Xeon™ Processors
3.60 GHz, 1MB L2 Cache, 800MHz FSB
■ Intel® E7520 Chipset ServerBoard
■ 1GB ECC Registered DDR2 400 SDRAM (Upto 16GB)
■ 8 x Seagate® 300GB SATA Hard Drive (Total 2.4TB storage)
■8x1" Hot-Swap SATA Drive Bays
■ DVD±RW Burner & Floppy Drive
■ 8 Channel SATA Controller (RAID o, 1,5,10,50 Support)
■ 2xlntel® 10/100/1000 Gigabit Network Controller
■ 3U RackmountChassis w/550W PowerSupply
■ 3-Year Limited Warranty and First Year On-Site Service
Dual Intel® Xeon™ Processors 3GHz
- Upgradable to Dual Intel® Xeon™ Processors
3.60 GHz, 1MB L2 Cache, 800MHz FSB
■ Intel® E7520ChipsetServer Board
■ 1.0GB ECC Registered DDR 333 SDRAM (Upto 16GB)
■ 4 x Seagate® 250GB SATA Hard Drive (Total itb storage)
■ 6x1" Hot-Swap SATA Drive Bays
■ Slim CD-ROM & Floppy Drive
■ 8 Channel SATA Controller (RAID o,i,JBOD Support)
■ 2xIntel® lO/lOO/lOOOGigabitNetworkController
■ 2U RackmountChassisw/550W PowerSupply
■ 3-Year Limited Warranty and First Year On-Site Service
$ 2,499
Intel® Xeon™ Processors 3GHz
- Upgradable to Dual Intel® Xeon™ Processors
3.60 GHz, 1MB L2 Cache, 800MHz FSB
■ Intel® E7320 Chipset ServerBoard
■ 1.0GB ECC Registered DDR 333 SDRAM (Upto 16GB)
■ 4 x Seagate® 250GB SATA Hard Drive (Total itb storage)
■ 7x1" Hot-Swap SATA Drive Bays
■ DVD±RW Burner & Floppy Drive
■ 4 Channel Raid Controller (RAID o, 1,1/0,5, JBOD Support)
■ 2xlntel® 10/100/1000 Gigabit Network Controller
■ 4U Rackmount Chassis w/550W Redundant Cooling Power Supply
■ 3-Year Limited Warranty and First Year On-Site Service
■ 3 .
New Powerful
64-bit Server Line
Friendly Server Specialists
Complete Solution Provider
Flexibility and Variety:
Customize platforms with
leading-edge products
We Build the Future !
$ 4,299 $ 2,199
# 1. OEM Computer Manufacturer New Accounts Receive Free Gift
■ 3 year warranty with lifetime tech support ■ Personal attention (Dedicated Technical Sales Team)
■ Reseller and volume pricing available. ■ Call now to customize using the latest technology
Find out how ZT Insider Program can help maximize your Business Solution
Goto
ztgroup.com/go/linuxjoumal
Call
866- ZTGROUP (866-984-7687)
Prmotecode: Ij 1005
Purchaser is responsible for all freight costs on all returns of merchandise. Full credit will not be given for incomplete or damaged returns. Absolutely no refunds for merchandise returned after 30 days. All prices and configurations are subject to change without
notice and obligation. Opened software is non-refundable. All returns have to be accompanied with an RMA number and must be in re-sellable condition including all original packaging. System’s picture may include some equipments and/or accessories, which
are not standard features. Not responsible for errors in typography and/or photography. All rights reserved. All brands and product names, trademarks or registered trademarks are property of their respective companies. Intel, Intel logo, Intel Inside, Intel Inside logo,
Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, Pentium, and Pentium III Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
How do you make
high-performance
computing
even better?
Customize it.
Using lntel®-based servers
to maximize power and
interoperability, Intel® Premier
Providers build high-performance
technology solutions customized
to your enterprise’s needs.
As members of the most elite
Intel® channel program, Premier
Providers deliver the latest IT
solutions, and have priority access
to parts and technical support.
Insist on the Best.
premier
PROVIDER
Find the Intel® Premier Provider
that’s right for you.
www.insistthebest.com
r = control->ops->connect(control,
(struct sockaddr *) &servaddr,
sizeof (servaddr), 0_RDWR);
read_response(control, response);
sprintf(temp, "USER %s\r\n", pmk.user);
send_reply(control, temp);
read_response(control, response);
sprintf(temp, "PASS %s\r\n", pmk.pass);
send_reply(control, temp);
read_response(control, response);
We start out by declaring pointers to a few socket structures, kmalloc () is the ker¬
nel equivalent of malloc () and is used to allocate memory for our character array. The
array’s response and reply will contain the responses to and replies from the server.
The first step is to read the parameters from user mode to kernel mode. This is cus¬
tomarily done with access_ok and veri fy_read/veri fy_wri te calls. access_ok checks
whether the user-space pointer is valid to be referenced, veri fy_ re ad is used to read
data from user mode. For reading simple variables like char and i nt, use __get_user.
Now that we have the user-specified parameters, the next step is to create a control
socket and establish a connection with the FTP server. sock_create () does this for
us—its arguments are similar to those we pass to the user-level socket () system call.
The struct sockaddr_in variable servaddr is now filled in with all the necessary
information—address family, destination port and IP address of the server. Each
socket structure has a member that is a pointer to a structure of type struct prot o_o p s.
This structure contains a list of function pointers to all the operations that can be per¬
formed on a socket. We use the connect () function of this structure to establish a
connection to the server. Our functions read_response() and send_reply() transfer
data between the client and server (these functions are explained later):
r = sock_create(PF_INET, SOCK_STREAM,
IPPROTO_TCP, &data);
memset(&claddr,0, sizeof(claddr));
claddr.sin_family = AF_INET;
claddr.sin_port = htons(EPH_PORT);
clddr.sin_addr.s_addr= htonl(
create_address(srci p));
r = data->ops->bind(data,
(struct sockaddr *)&claddr,
sizeof (claddr));
r = data->ops->listen(data, 1);
Now, a data socket is created to transfer data between the client and server. We fill
in another struct sockaddr_in variable claddr with information about the client—
protocol family, local unprivileged port that our client would bind to and, of course,
the IP address. Next, the socket is bound to the ephemeral port EPH_PORT. The func¬
tion 1 i sten () lets the kernel know that this socket can accept incoming connections:
a = (char *)&claddr.sin_addr;
p = (char *)&claddr.sin_port;
send_reply(control, reply);
read_response(control, response);
strcpy(reply, "RETR ") ;
strcat(reply, src);
strcat(reply, "\r\n");
send_reply(control, reply);
read_response(control, response);
28 OCTOBER 2005 WWW.LINUXJOURNAL.COM
The Intel® Xeon™ processor now provides high
availability for your 64-bit applications and clusters.
The Intel® Xeon™ processor now works harder
for your business than ever. With innovative
features that enable power-saving options,
flexible memory, I/O and storage configuration.
And, of course, continued support for all
your existing 32-bit applications.
How can clusters featuring the Intel Xeon processor serve you?
intel.com/go/xeon
Atipa Technologies
4921 Legends Drive
Lawrence, KS 66049
888-222-7822
www.atipa.com
Bell Computers
187 Pacific Street
Pomona, CA 91768
909-598-1006
www.bellcomputer.com
Western Scientific
9445 Farnham Street
San Diego, CA 92123
858-565-6699
www.westernscientific.com/intel
©2005 Intel Corporation. Intel, the Intel logo, Intel Inside and Intel Xeon are trademarks or registered trademarks
of Intel Corporation or its subsidiaries in the United States and other countries. All rights reserved.
Compilers
are building
the 64-bit
applications
infrastructure.
C, C++, F77, F95 and HPF • 32-bit and 64-bit Linux
Optimized for AMD64 and IA32/EM64T • Full 64-bit support
Workstation, Server and Cluster configurations • Fast compile times
Native OpenMP • Native SMP auto-parallelization • Cache tiling
Function inlining • SSE/SSE2 Vectorization • Loop unrolling
Interprocedural optimization • Profile-feedback optimization
Large file support on 32-bit Linux • 64-bit integers and pointers
F77 pointers • Byte-swapping I/O • VAX and IBM extensions
OpenMP/MPI/threads debugging • OpenMP/MPI/threads profiling
Interoperable with g77/gcc/gdb • PDF and printed documentation
Electronic purchase, download and upgrades • Tech support
Network-floating licenses • Academic and volume discounts
Visit www.pgroup.com to download a free PGI evaluation package
and see the latest tips and techniques for porting to 64-bit systems.
The Portland Group
www.pgroup.com ++01 (503) 682-2806
STMicroelectronics
The registered trademarks and marks are the property of their respective owners.
As explained previously, a PORT command is issued to
the FTP server to let it know the port for data transfer. This
command is sent over the control socket and not over the
data socket:
64-bit
GAUSSIAN
new_sock = sock_alloc();
new_sock->type = data->type;
new_sock->ops = data->ops;
r = data->ops->accept(data, new_sock, 0);
new_sock->ops->getname(new_sock,
(struct sockaddr *)address, &len, 2);
Now, the client is ready to accept data from the server.
We create a new socket and assign it the same type and ops
as our data socket. The accept () function pulls the first
pending connection in the listen queue and creates a new
socket with the same connection properties as data. The new
socket thus created handles all data transfer between the
client and server. The getnameO function gets the address at
the other end of the socket. The last three lines in the above
segment of code are useful only for printing information
about the server:
if((total_written = write_to_file(pmk.dst,
new_sock, response)) < 0)
goto err3;
The function wri te_to_f i le deals with opening a file in
the kernel and writing data from the socket back into the file.
Writing to sockets works like this:
void send_reply(struct socket *sock, char *str)
{
}
send_sync_buf(sock, str, strlen(str),
MSG_DONTWAIT);
int send_sync_buf
(struct socket *sock, const char *buf,
const size_t length, unsigned long flags)
{
struct msghdr msg;
struct iovec iov;
int len, written = 0, left = length;
mm_segment_t oldmm;
msg.msg_name = 0;
msg.msg_namelen = 0;
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
msg.msg_control = NULL;
msg.msg_controllen = 0;
msg.msg_flags = flags;
oldmm = get_fs(); set_fs(KERNEL_DS) ;
Gaussian 03 is the premier electronic structure
program. Chemists and other scientists use it to
study important molecules and reactions related
to drug design, materials science, catalysis, and
other areas of leading edge and commercial
research interest.
See www.gaussian.com to learn about
the latest Gaussian 03 innovations that make
it applicable to very large molecules previously
out of reach of accurate models.
Gaussian, Inc builds Gaussian 03 for 64-bit
AMD64 and EM64T processor-based systems
using PGI Compilers and Tools.
repeat_send:
msg.msg_iov->iov_len = left;
msg.msg_iov->iov_base = (char *) but +
written;
len = sock_sendmsg(sock, &msg, left);
return written ? written : len;
}
The send_reply() function calls send_sync_buf (), which
does the real job of sending the message by calling
sock_sendmsg() . The function sock_sendmsg() takes a pointer
to struct socket, the message to be sent and the message
length. The message is represented by the struture msghdr. One
of the important members of this structure is i ov (io vector).
The iovector has two members, iov_base and iov_len:
struct iovec
{
/* Should point to message buffer */
void *iov_base;
/* Message length */
_kernel_size_t iov_len;
};
These members are filled with appropriate values, and
sock_sendmsg() is called to send the message.
The macro set_f s is used to set the FS register to point
to the kernel data segment. This allows sock_sendmsg() to
find the data in the kernel data segment instead of the user-
space data segment. The macro get_f s saves the old value
of FS. After a call to sock_sendmsg(), the saved value of FS
is restored.
Reading from the socket works similarly:
int read_response(struct socket *sock, char *str)
{
len = sock_recvmsg(sock, &msg,
max_size, 0);
return len;
}
The read_response() function is similar to send_reply () .
After filling the msghdr structure appropriately, it uses
sock_recvmsg() to read data from a socket and returns the
number of bytes read.
A User-Space Program
Now, let’s take a look at a user-space program that invokes our
system call to transfer a file. We explain the relevant details for
calling a new system call:
#define_NR_my_sys_call 223
_syscalll(long long int, my_sys_call,
struct params *, p);
int main(int argc, char **argv)
{
struct params pm;
/* fill pm with appropriate values */
r = my_sys_call(&pm);
}
#d e f i n e N R_my_sy s_c all 223 assigns a number to our
system call. _syscalll() is a macro that creates a stub for the
system call. It shows the type and number of arguments that
our system call expects. With this in place, my_sys_call can be
invoked just like any other system call. Upon running the pro¬
gram, with correct values for the source and destination files, a
file from a remote FTP server is downloaded onto the client
machine. Here is a transcript of a sample run:
# make
make -C /lib/modules/2.6.9/build SUBDIRS=/home/ppadala/ftp modules
make[1]: Entering directory '/home/ppadala/linux-2.6.9’
CC [M] /home/ppadala/ftp/ftp.o
Building modules, stage 2.
MODPOST
CC /home/ppadala/ftp/ftp.mod.o
LD [M] /home/ppadala/ftp/ftp.ko
make[1]: Leaving directory '/home/ppadala/linux-2.6.9 1
# gcc do_ftp.c
# ./a.out 152.2.210.80 /README /tmp/README anonymous anon@cs.edu
Connection from 152.2.210.80
return = 215 (length of file copied)
Conclusions
We have seen a basic implementation of an FTP client within
the kernel. This article explains various issues of socket pro¬
gramming in the kernel. Interested readers can follow these
ideas to write various network applications, such as an HTTP
client or even a Web server in the kernel. Kernel applica¬
tions, such as the TUX Web server are used for high-perfor¬
mance content serving and are well suited for environments
that demand data transfer at high rates. Careful attention has
to be paid to the design, implementation and security issues
of such applications.
Resources for this article: www.linuxjournal.com/article/
8453.0
Pradeep Padala is a PhD student at the University of
Michigan. His general interests are in distributed
systems with specific emphasis on scheduling and
fault tolerance. He is the author of the NCurses
Programming HOWTO and contributes to various
open-source projects. More about him can be found on his Web
site at www.eecs.umich.edu/~ppadala.
Ravi Parimi has a Master's degree in Computer
Engineering and currently works in Silicon Valley,
California. His main interests are in operating
systems, networking and Internet security. He
has been using Linux since 1998 and aspires to
be a kernel hacker. In his free time, he pursues Vedic studies
and Chess.
321 OCTOBER 2005 WWW.LINUXJOURNAL.COM
Where Open
Minds Meet...
October 5-6 • 2005 • Olympia 2 • London
The UK's only event for business
and the technical community
Come along to LinuxWorld Expo and tackle IT business issues, gain
real-time Linux and open source solutions, meet key suppliers, ask
technical points and get answers, discover 'how-to', see new
technologies, source products and network with the entire
community; experts, colleagues and suppliers;
Register NOW at www.linuxworldexpo.co.uk
for FREE entry into the exhibition, featuring;
• The Great Linux Debate
• OSC/Open Source Academy
• FREE Product Briefings and Demonstrations
• FREE Showcases of Technology
• FREE Case Study Presentations
# FREE Open Forum Europe Advice Centre
# .org village
# Internet Cafe with Wireless Connection
• Pre-registered visitors can take the LPI exam
for ONLY £25 - Saving £100
CONFERENCE PROGRAMMES
Technical and Enterprise:
Two streams, two days of informative and in-depth
sessions covering key issues and topics for technical
and enterprise.
Linux in Business:
Practical applications, benefits and analysis
of Linux and open source in business:
SPEAKERS INCLUDE:
Klaus Knopper 'Engineer, Consultant and Software
Developer, KNOPPER.NET& Maker, Knoppix
Mark Shuttleworth, President, Ubuntu Foundation
Rasmus Lerdor [Engineer, Yahoo! Inc.
Paul Everitt, Founder, Zope Europe Association
Linux for the Technical team:
What's here now; and what's coming next.
SPEAKERS INCLUDE:
Andrew Eddie, Senior Systems Integrator,
Toowoomba City Council, Australia
Kevin Carmony, President and CEO, Unspire Inc
Michael Tiemann, Vice President Open Source Affairs,
Red Hat
Brian Green, Director of Solutions Management, Novell
Delegate places are limited, so book today and also
benefit from:
• FREE LPI Examination - Saving £125
• GUARANTEED seating at The Great Linux Debate
• FREE Entry to the exhibition
View the full conference programme
and book your place TODAY! -
www.linuxworldexpo.co.uk
Register NOW at www.linuxworldexpo.co.uk
GOLD SPONSOR
D0U
PLATINUM SPONSORS
NOVell s|sf ORACLE
OWNED BY
*IDG
WORLD EXPO
ORGANISED BY
turret
GROU P |
All trademarks acknowledged. E&OE. Programme may be subject to change. Correct at time of press.
Trekking
through the
Desktop
Jungle
Is it easier to find a document on a faraway Web
server than one on your own hard drive? Try some
search programs to dig up the files you need.
BY MARCEL GAGNE
T hat certainly does make it difficult, Francois. When I
asked you to locate the wine order from last month and
you told me it was somewhere on your disk, I didn’t
expect that it was sitting “somewhere on your disk” in
quite this way. This is possibly the most disorganized home direc¬
tory I’ve ever seen. Every document is in the same folder, and all
the files are cryptically named. What were you thinking, mon amil
Quoi? Well, of course there is a way to find it. If the docu¬
ment still exists somewhere on your disk, we’ll find it. We just
need to use the right tools. Later, though—our guests will be
here any moment and...too late, Francis, they are already
here! Welcome, everyone, to Chez Marcel , home of fine Linux
fare and exquisite wines. Please sit and make yourselves com¬
fortable. Francis will fetch your wine immediatement.
Francis, head to the east wing of the wine cellar and bring
back that 2001 Nuits Saint George Pinot Noir we’ve been tast¬
ing, er, I mean, subjecting to quality control. Vite!
That wine, mes amis , just happens to represent part of an
order lost in one of Francis’ documents on his computer.
Trouble is, he doesn’t remember which document. What we
need to do, is set him up with a desktop search engine. Luckily,
this just happens to be the basis of tonight’s menu, so we all
will profit from my faithful waiter’s lack of organization.
The original desktop search engine, mes amis , is something
that’s been around in Linux from the beginning, and that’s the
find command. This is an amazingly powerful tool and one that
is easily overlooked in this age of cutting-edge graphical desk¬
tops. In its most basic form, find is used like this:
find starting_dir [options]
One of those options is -print, which makes sense only if
you want to see any kind of output from this command. You
easily could get a listing of every file on the system by starting
at the top and recursively listing the disk:
find / -print
Of course, it makes more sense to search for something, for
instance, all the MP3-type music files sitting on your disk.
Because you know that the files end in a .mp3 extension, you
can use that to search:
find / -name "*.mp3" -print
This is also great for locating big files you haven’t looked
at in forever. Maybe it’s time to do a little archiving of those
old files, but how do you find only them? Say you want to look
for anything that has not been modified (this is the -mtime
parameter) or accessed (the -atime parameter) in the past 12
months. The -o option is the “or” in this equation:
find /home/marcel -size +1024 \( -mtime +365 -o -atime +365 \) -Is
In case you are curious, the back-slashes in front of the
parentheses are escape characters; they are there to make sure
the shell does not interpret them in ways you do not want it
to—in this case, the open and close parentheses on the second
line. The preceding command also searches for files that are
greater than 500KB in size. That is what the - size +1024
means, because 1024 refers to 512-byte blocks. The - Is at the
end of the command tells the system to do a long listing of any
files it finds that fit the search criteria. So far so good?
The find command is fairly simple to use on the surface,
but it also has many command-line options and (as you can
see) interesting ways of passing the results of a search to other
commands, so that the results can be narrowed down or fine-
tuned. Getting to know find is a great idea, but there are alter¬
natives that are a little friendlier.
Many people out there have grown up in the graphical
world of KDE or GNOME, so desktop tools have been created
in each of these environments. Even so, my experience indi¬
cates that these excellent tools are, for many users, as equally
overlooked as find. Let’s have a look at those now.
Let’s begin our search for search tools under KDE. Click
the application launcher and look for a submenu labeled Find.
The Find menu has two options, one for files and one for Web
search (which, by default, launches Konqueror on the Google
Web site). You also can fire up the files search tool by using
the Alt-F2 quick launch (program name: kf i nd). When the
application starts, the Find Files/Folders dialog appears. It
contains three different tabs, and each is designed to help you
locate the information you need. They are labeled
Name/Location, Contents and Properties.
Under the Name/Location tab, specify the starting folder,
either by entering it manually or by clicking the Browser but¬
ton and navigating over to it using the KDE file navigator.
There’s also a field labeled Named where you enter part of a
filename using Linux metacharacters. For instance, if I wanted
to find all the files with Cooking anywhere in the title, I would
enter *cooking*. By default, this is a case-insensitive search,
so upper- and lowercase don’t matter in terms of the search
results. You can, however, override this behavior by clicking
the Case-sensitive search check box.
Under the Contents tab, the real action takes place.
Generally speaking, I don’t have a problem locating a file by
name. It’s the content that is the real issue. Which of your sev¬
eral hundred documents contains a reference to a particular
341 OCTOBER 2005 WWW.LINUXJOURNAL.COM
Name/Location ~~J Contents | Properties j
File type: All Files & Folders
Containing text [wine
Q Case sensitive Q Regular expression
□ Indude binary files
Search metainfo sections: I* [ for:
1 Etnd j
Save As...
Name ▼
In Subfolder
I s ”, J
Modified
Per-
lir<
' Cooking June 2004 Democrazy.txt
15,119
18/03/04 10:02 am
1
[7 Cooking June2005 lntranets.txt
2005/
13,749
11/03/05 07:39 pm
1
7] Cooking Mar2004Xnesttxt
12,133
09/12/03 12:22 pm
1
[J! Cooking May2004 Superkaramba.txt
11,581
17/02/04 04:54 pm
1
—I
0 Cooking May2005 Crossing Platforms.txt
2005/
10,896
11/02/05 04:20 pm
[7 Cooking Nov 2004 High Peformance.txt
2004/
12,338
13/08/04 05:53 pm
71 Cooking Oct 2004 Security.txt
2004/
12,753
08/07/04 04:50 pm
7 Cooking Sept 2004Wireless.txt
2004/
12,157
11/06/0411:48 am
I
[7 Cooking Sept2005 Wireless.txt
2005/
13,059
09/06/05 10:43 am
1
<
1
VLM
Ready. 20 files found
Figure 1. KFind makes it easy for Marcel to locate all those columns that
mention "wine".
Figure 2. The GNOME search tool allows you to search by name as well as text
within a file.
word or phrase is a more difficult search than which has a par¬
ticular word in the name. The Contents tab lets you enter your
search text (again, case-insensitive by default), regular expres¬
sion searches and so on. You even can specify that Kfind
search through binary files and not only documents (Figure 1).
There’s also a meta-info search feature for things like MP3
files that contain embedded information, such as title and artist.
Finally, the Properties tab provides a means of searching
for files or folders based on creation or modification date, own¬
ership and more.
Similarly, GNOME users have access to the GNOME
search tool (program name: gnome-search-tool), a similar
program that lets you search based on filename, file content
(text search) and date. Choose Search for files in the GNOME
Places menu (I’m running 2.10 in this example), and this
brings up the file find dialog (Figure 2).
When the dialog first appears, there isn’t much to see. The
defaults are to search for a file by name, which you enter in the
Name contains field. Below that is your starting folder for the
search, the default being your home directory. To get the full
power of the GNOME search tool, click on the arrow next to
the label that says Show more options. A new field appears
through which you can specify some text in the file itself.
Finally, directly below the text search field, is one other
option that can be quite complex. A drop-down box labeled
Available options includes size, date and ownership search cri¬
teria that can be applied to narrow down your search results
even further.
If you’ve been following search technology in any way,
you’ll know that there’s a lot of excitement concerning desktop
search engines these days—think Google for your desktop. In
fact, Google does provide such a tool, but alas, only for non-
Linux operating systems. However, this is not to say that desk¬
top search tools don’t exist for Linux.
One such tool is Roberto Cappuccio’s Kat, a desktop search
engine and indexing tool that makes it easy and fast to do full-
text searches in a variety of document formats (for example,
PDF, OpenOffice.org, KWord and so on). You also can search
for images using thumbnails and more.
The Kat Web site (see the on-line Resources) provides bina¬
ry packages for a number of distributions, so you may not need
to build from source. Should you need to, however, the process
is nothing more than the classic extract-and-build five-step. In
terms of prerequisites, you need the SQLite database and its
development libraries.
To use Kat, simply start the program (name: kat) and a
plain three-pane window appears where you will do your work
We’ve got
problems with your
name on them.
At Google, we process the world’s information and make it
accessible to the world's population. As you might imagine,
this task poses considerable challenges. Maybe you can help.
We’re looking for experienced software engineers with superb
design and implementation skills and expertise in the
following areas:
• high-performance distributed systems
• operating systems
• data mining
• information retrieval
• machine learning
• and/or related areas
If you have a proven track record based on cutting-edge
research and/or large-scale systems development in these
areas, we have brain-bursting projects with your name on
them in Mountain View, Santa Monica, New York, Bangalore,
Hyderabad, Zurich and Tokyo.
Ready for the challenge of a lifetime? Visit us at
http://www.google.com/lj for information. EOE
WWW.LINUXJOURNAL.COM OCTOBER 2005135
Figure 3. Using kfile hooks, Kat can index almost anything.
Figure 4. As Kat creates the new catalog, the program reports statistics on the process.
and your searching. The first step is to create a catalog. To do
this, click File on the menu bar and select New.
When creating a new catalog, a four-tabbed window
appears. The first tab, labeled Catalog, is where you enter the
starting directory, the name of the catalog and other identifying
information. On the second tab, labeled Metadata, you’ll see a
list of the various metadata engines that are available to Kat for
indexing. You can remove different formats, but most likely,
this will stay as is (Figure 3). Similarly, the Fulltext tab. Under
Thumbnails, you can select the size of the thunmbnails created
during the index process.
A status window keeps you abreast of the number of files
and folders scanned, as well as the size of the collection
(Figure 4).
This brings us to the one big drawback of a tool like this. If
Figure 5. Although the initial indexing can take some time, Kat searches are
blazingly fast.
the folder for which you are creating a catalog is large, this can
take an amazing amount of time. Be prepared or keep your cat¬
alogs confined to a reasonable collection of files. I tried to
index my own home directory in its entirety at nearly 6.6GB of
data—suffice it to say, that was a mistake.
Once a catalog has been created, finding information is
blazingly fast. Simply click on the search icon on the far right
(the magnifying glass), enter your search term and Kat returns
the results of the search almost instantly (Figure 5).
According to the clock on the wall, it would appear, mes
amis , that closing time has arrived. Before we leave this topic
of desktop search engines, I’d like to mention another package
with the friendly, puppy-dog name of Beagle. Beagle is built
on Mono (the open-source .Net implementation) and requires
an inotify-enabled kernel. Neither is uncommon in the more
modern distributions. Beagle also shows promise in that it is
very fast and works silently in the background, keeping an eye
on what you tell it while automatically updating its catalog of
information. Unfortunately, Beagle is very much alpha code
and not quite ready for prime time, as they say (although it is
included with the new SUSE Linux Professional 9.3).
Nevertheless, Beagle is a tool to watch, and I’ve included the
link in the on-line Resources.
Please raise your glasses, mes amis , and let us all drink to
one another’s health. A votre sante! Bon appetit!
Resources for this article: www.linuxjournal.com/article/
8456.0
Marcel Gagne is an award-winning writer living
in Mississauga, Ontario. He is the author of
Moving to the Linux Business Desktop (ISBN
0-131-42192-1), his third book from Addison-
Wesley. He also makes regular television appear¬
ances as Call for Help's Linux guy. Marcel also is a pilot and a
past Top-40 disc jockey. He writes science fiction and fantasy
and folds a mean Origami T-Rex. He can be reached via
e-mail at mggagne@salmar.com. You can discover a lot of
other things (including great Wine links) from his Web site
at www.marcelgagne.com.
3 6 H OCTOBER 2005 WWW.LINUXJOURNAL.COM
Best Price
Best Performance. Best Support
Enterprise Router™
The Enterprise features four 8 Gbps
buses, dual CPUs, and redundant AC or
DC power supplies. It can route multiple
0C3 or 0C12 circuits at wire speed.
The Transport " is ImageStream's best¬
selling router. With its small footprint,
business-class features, and competitive
price, the TransPort is an ideal rout
er for T1 and El applications that
demand low latency wire-speed
performance.
The TransPort includes 128 MB
RAM, three 10/100 ethernet ports,
and an expansion slot for add-on cards.
It also features the ImageStream Linux™
router distribution, which supports most
WAN protocols and advanced features
including NAT firewall, peer-to-peer traffic
control, bridging, bandwidth limiting,
QoS, dynamic routing, VPN, and more.
Like all ImageStream routers, the
TransPort includes 12 months of free
24/7 technical support, a full 12-month
warranty on parts and labor, free software
upgrades for life, and the industry's only
money back performance guarantee.
Gateway Router™
ImageStream's Gateway is the industry's
lowest cost 0C3 router. The upgraded
dual bus Gateway 64™ can route multiple
DS3/E3 or 0C3 circuits at wire speed.
UJelL Connected
The TransPort provides three
10/100 ethernet ports for
flexible LAN connectivity.
Faoless CPU
The Transport's highly
efficient CPU minimizes
power consumption and
cooling requirements.
Best Software The ImageStream Linux™
router distribution has everything you need to
deploy advanced network applications including
secure shell, menu-based configuration, real-time
monitoring, IP firewall, bridging, interface bonding,
dynamic routing, QoS, IPsec VPN, free software
upgrades for life, and more.
Best Support ImageStream routers include
a full year of free 24/7 support. When you consider
total cost of ownership, ImageStream routers cost
less up front and over time.
Rebel Router™
The Rebel is the industry's lowest cost
DS3/E3 router. The 1U Rebel can route
one DS3/E3 or up to 16 T1/E1 circuits
at wire speed.
Rf Router
The R1 is designed for extended tempera¬
ture applications including outdoor installa¬
tions. The R1 has a small footprint, and can
be installed in set-top, wall-mount, and
rackmount applications.
Easy Indicators
The TransPort provides front
panel LEDs to show LAN
connection status.
-8.7 in. pi 0.0 in_
Sized Riyht The Transport's small footprint
allows it to be installed just about anywhere.
• Guaranteed Performaoce our 31-day
performance guarantee ensures your router will
function as promised. If your ImageStream router
does not work as specified in writing, and our
support team cannot correct the problem, you can
return your router for a full refund of the purchase
price. See our web site for details.
ZM ImageStream.
Routers for the Real World"
800.813.5123
www. imagestream. com
1 . 574 . 935.8484
ImageStream, Enterprise Router, Gateway Router, Gateway 64, Rebel Router, Rl, TransPort, ImageStream Linux and "Routers for the Real World"
are trademarks of ImageStream Internet Solutions, Inc. Linux is a registered trademark of Linus Torvalds. Specifications are subject to change
without prior notice. *Please refer to ImageStream's Web site for more information on wire-speed specifications and the performance guarantee.
Limitations
of she,
a Shell
Encryption
Utility
The shell script compiler, she, obfuscates shell
scripts with encryption—but the password is in the
encrypted file. Could an intruder recover the original
script using objdump? by nalneesh guar
he is a popular tool for protecting shell scripts that con¬
tain sensitive information such as passwords. Its popu¬
larity was driven partly by auditors’ concern over pass¬
words in scripts, she encrypts shell scripts using RC4,
makes an executable binary out of the shell script and runs it as
a normal shell script. Although the resulting binary contains
the encryption password and the encrypted shell script, it is
hidden from casual view.
At first, I was intrigued by the she utility
(www.datsi.fi.upm.es/~frosal/sources/shc.html) and consid¬
ered it as a valuable tool in maintaining security of sensitive
shell scripts. However, upon further inspection, I was able to
extract the original shell script from the shc-generated exe¬
cutable for version 3.7. Because the encryption key is stored in
the binary executable, it is possible for anyone with read access
to the executable to recover the original shell script. This arti¬
cle details the process of extracting the original shell exe¬
cutable from the binary generated by she.
she Overview
she is a generic shell script compiler. Fundamentally, she takes
as its input a shell script, converts it to a C program and runs
the compiler to compile the C code. The C program contains
the original script encrypted by an arbitrary key using RC4
encryption. RC4 is a stream cipher designed in RS A laborato¬
ries by Ron Rivest in 1987. This cipher is used widely in com¬
mercial applications, including Oracle SQL and SSL. Listing 1
demonstrates running she.
The two new files, named with the .x and .x.c extensions to
the name of the source shell script, are the executable and an
intermediate C version. Upon executing pub.sh.x, the original
shell source is executed, she also specifies a relax option, -r.
The relax option is used to make the executable portable.
Basically, she uses the contents of the shell interpreter itself,
Listing 1. Running she
[userl@shiraz test]# cat pub.sh
#!/bin/sh
echo "Hello World"
userl@shiraz test]# ./pub.sh
Hello World
[userl@shiraz test]# she -v -r -f pub.sh
she shll=sh
she [-i]=-c
she [-x]=exec '%s'
she [-1]=
she opts=
she: cc pub.sh.x.c -o pub.sh.x
she: strip pub.sh.x
[userl@shiraz test]# Is
pub.sh pub.sh.x pub.sh.x.c
[userl@shiraz test]# ./pub.sh.x
Hello World
such as /bin/sh, as a key. If the shell binary were to change, for
example, due to system patching or by moving the binary to
another system, the shc-generated binary does not decrypt
or execute.
I inspected the shell executable using strings and found no
evidence of the original shell script. I also inspected the inter¬
mediate C source code and noted that it stores the shell script
in encrypted octal characters, as depicted in Listing 2.
Listing 2. The original shell script becomes an RC4-encrypted string in the C
version.
static char text[] =
"\223\004\215\264\102\216\322\060\300\070\101\217\277\161\033\130"
"\217\145\370\170\106\257\176\301\057\132\172\044\217\247\276\222"
"\203\076\334\201\323\107\064\334\120\132\001\241\267\052\203\216"
"\116\232\156\337\121\145\235\003\156\244\142\246\117\200\206\014"
"\004\153\372\152\030\262\171\275\137\342\247\367\231\315\353\151"
"\264\241\230\105\344\053\034\247\342\142\156\305\327\255\036\111"
"\234\061\013\355\300\336\324\257\175\124\222\044\132\040\276\067"
"\007\002\371\063\021\320\060";
The C source code also includes as arrays the password as
well as other encrypted strings. Therefore, anyone with access
to the source code easily can decrypt and view the contents of
the original shell script. But what about the original shell binary
executable generated by she? Is it possible to extract the
original shell script from nothing but the binary executable?
The answer to this question is explored in the next section.
Extraction Approach
I generated and reviewed the C source code for several shell
scripts to better understand how the shell source is encrypt¬
ed and decrypted. Fundamentally, she uses an implementa¬
tion of RC4 that was posted to a Usenet newsgroup on
September 13, 1994. I set off by first identifying the encryp-
38 OCTOBER 2005 WWW.LINUXJOURNAL.COM
tion key and the encryption text. The
objdump utility came in handy for
this, bjdump, part of GNU binutils,
displays information about object
files. First, we use objdump to retrieve
all static variables, for this is where
the encryption key and the encrypted
shell text are stored. Listing 3 pro¬
vides a brief overview of objdump.
The first column of the output in
listing 3 specifies the starting address¬
es in hexadecimal, followed by the
stored data in the next four columns.
The last column represents the stored
data in printable characters. So some¬
where in the first four columns of the
output is the array of characters that
form the encryption key (password)
and the encrypted shell script.
Comparing the original C source
code and Listing 3, you can see that
the password most likely begins at
address 0x804a540. After comparing
other executables, I determined that the
first address after the zeros leading the
“Please contact your provider” text
usually is the starting address. To
retrieve these arrays, such as the one
depicted in Listing 2, we also need to
look at the disassembled code. We use
objdump again here, except this time
with the -d option, for disassemble, as
shown in Listing 4.
The last two columns represent
assembly instructions. The movl
instruction is used to move data—
movl Source, Dest. The Source and
Dest are prefixed with $ when referenc¬
ing a C constant. The push takes a single
operand, the data source, and stores it at
the top of stack.
Now that we have the basics of
objdump, we can proceed to extract the
encryption password and eventually
the shell code.
In the intermediate C code pro¬
duced by she, about nine arrays are
referenced by the variables pswd, shll,
inlo, xecc, lsto, chkl, opts, txt and
chk2. The pswd variable stores the
encryption key, and the txt variable
stores the encrypted shell text, she
hides the useful information as smaller
arrays within these variables. Thus,
obtaining the actual array involves two
steps. First, identify the length of the
array. Second, identify the starting
address of the array.
The objdump output needs to be
looked at in detail to obtain the actual
I Need a sharper
^ development
jBfek tool for your
fljf application’s
or database?
AMD
Opteron
V
SQL is
only one
of our
options...
C-TREE PLUS® DATABASE TECHNOLOGY I FEATURED I CUSTOMER
OPENS UP YOUR OPTIONS
HIGHLIGHTS
TESTIMONIAL
SQL offers a convenient and
easy-to-use database interface.
ISAM provides powerful performance
with precision indexing control in a
small footprint. With c-tree Plus you
can simultaneously enjoy BOTH!
Superior ISAM indexing technology
PLUS an industry-standard SQL
interface provide blazing fast data
management for every
environment. Break the limitations
of a single solution and open up
your database options. Experience
the benefits c-tree Plus can deliver
to your application!
• Fast, reliable, and
portable
• Low deployment
cost
• No DBA required
• Professional
technical support
• Source code
• 64-bit support
• 16-exabyte file
support
• Memory files
• Embeddable
database
• Full OLTP support
“We have reviewed
Oracle and some of
the other big
relational databases
and chose FairCom
for our database
development needs.
With c-tree Plus, we
see transactional
volume that is 8 to 10
times faster than what
we can get with other
databases. I have
been using c-tree
based solutions since
the 80 's and highly
recommend it... ’’
Visit our Web site for
more testimonials
about c-tree!
Database your way.
See for yourself —
download c-tree Plus® Today!
Go to www.faircom.com/go/open for a FREE evaluation of c-tree Plus!
Other company and product names are registered trademarks or trademarks of their respective owners. © 2005 FairCom Corporation
WWW.LINUXJOURNAL.COM OCTOBER 20051 39
Listing 3. objdump browses the object file for interesting-looking strings.
/usr/bin/objdump --section=.data -s pub.sh.x
pub.sh.x: file format elf32-i386
Contents of section .data:
804a4e0
804a4f0
804a500
804a510
804a520
804a530
804a540
804a550
804a560
00000000
00000000
00000000
63742079
00000000
00000000
e554f49f
7a9beb67
eba28b7e
00000000
00000000
506c6561
6f757220
01000000
00000000
93dcd6dc
60277cb2
7e615a3a
3ca80408
00000000
73652063
70726f76
00000000
00000000
bb0bdc9b
dd9e0886
6d37d51a
00000000
00000000
6f6e7461
69646572
00000000
00000000
ad60edd0
0797aeec
97c2eall
.... Please conta
ct your provider
.T.
...—aZ:m7
804a68a, respectively. This way, we are able to obtain the start¬
ing addresses and lengths of all nine variables. Next, we need
to be able to decrypt the original shell script using only the
binary as input.
In she, before the shell script itself is encrypted, many
other pieces of information are encrypted. Furthermore, the
RC4 implementation maintains state between encrypting
and decrypting each individual piece of information. This
means that the order in which she encrypts and decrypts
information must be maintained. Failure to do so results in
illegible text. To extract the original shell script, we need to
perform several decryptions. For this step, I wrote a small
program called deshc, using the existing code from one of
the intermediate C files. The program reads two files as its
input, the binary executable and an input file that specifies
the array lengths and addresses, deshc executes the following
four steps:
Listing 4. The output of obj dump -d pub. sh . x shows information needed
to find the encrypted script. Lines in parentheses were added.
8048e52:
: 68
28
01
00
00
push
$0x128
(Length of encryption key)
8048e57:
: 68
40
a5
04
08
push
$0x804a540
(Key address)
8048e5c
e8
17
fb
ff
ff
call
0x8048978
8048e61
83
c4
10
add
$0x10,%esp
8048e64
83
ec
08
sub
$0x8,%esp
8048e67
6a
08
push
$0x8
(Length of shll)
8048e69:
: 68
72
a6
04
08
push
$0x804a672
(shll address)
8048e6e
e8
a0
fb
ff
ff
call
0x8048al3
8048e73
83
c4
10
add
$0x10,%esp
8048e76
83
ec
08
sub
$0x8,%esp
8048e79
6a
03
push
$0x3
(length of inlo)
8048e7b:
: 68
8a
a6
04
08
push
$0x
8048e80:
: e8
8e
fb
ff
ff
call
0x8048al3
array length and the starting address. My first hint here is to
look for all addresses that are within the data section
(Listing 2) of the disassembled object code. Next, seek out
all the push and mov commands in Listing 4. Addresses will
be different for different scripts, but when you encrypt a
few scripts and read the resulting C code, the patterns
become familiar.
The 804a540 address seems to correspond to the pswd vari¬
able, the encryption key. The length of the useful portion of the
encryption key is represented by 0x128, or 296 in decimal
form. Similarly, the next variables, shll and inlo, have useful
lengths of 0x8 and 0x3 and starting addresses of 804a672 and
Reads binary executable.
■ Extracts data section from the disassembled output.
■ Retrieves individual arrays based on input file.
■ Decrypts individual arrays in order, so that the RC4 state is
maintained.
Based on the objdump output, I have arrived at the follow¬
ing array lengths and addresses for the pub.sh.x executable:
pswd
0x128
0x804a540
shll
0x8
0x804a672
inlo
0x3
0x804a68a
xecc
0xf
0x804a68e
Isto
0x1
0x804a6a4
chkl
0xf
0x804a6a6
opts
0x1
0x804a6be
txt
0x76
0x804a6e0
All of these parameters are used in an input file to deshc,
which then decrypts and prints the original shell script.
Conclusion
An approach to extract the shell source code successfully from
she version 3.7 generated binary executable was demonstrated.
The pub.sh script was used for illustrative purposes only. I
have indeed tested the deshc program on executables that I did
not create and without access to the source code or the original
shell script.
Francisco Garcia, the author of she, recently released ver¬
sion 3.8. It uses somewhat different data structures and
improves upon the security of the previous version.
Nevertheless, I believe that embedding the encryption pass¬
word within the binary executable is dangerous and prone to
extraction as discussed in this article.@
Nalneesh Gaur, CISSP, ISAAR works at Diamond Cluster
International as a BS7799 Lead Auditor.
401 OCTOBER 2005 WWW.LINUXJOURNAL.COM
servers ► direct-
More products, better service, guaranteed.
GO STRAIGHT TO THE SOURCE!
Introducing the Servers Direct Blade System with
the power of Intel® Xeon™ Processor
Increased computing power in a
smaller footprint and simplified
maintenance help you expand your
enterprise solution to meet the most
intense application demands.
SDB-1100H Servers Direct Blade System
Featuring a Server Direct Server Compute Blade
powered by dual ® Xeon™ 800FSB Processors.
Benefit of using Servers Direct Blade System:
•More power, bandwidth, and processing performance to meet the demanding requirements of
departmental workloads
• Deliver world-class performance for peak server workloads
•Future 64-bit-enabled applications
•High performance small form factor SCSI hard drives (RAID 1 with 2 HDDs)
•Ethernet I/O for demanding, data-intensive applications
•Ability to easily add hotswappable SCSI hard drives, and additional Ethernet, or Fibre Channel
I/O for increased application performance (RAID 1E requires use of the HDDs on the SBX82 as
well as the HDDs on the SBESCSI)
1U Xeon Entry Level Server
SDR-1300T
Highest performing with Dual Xeon
800MHz. Excellent with general
purpose applications and provide the
most power.
• Intel Xeon Processor 2.8Ghz with 800FSB1 MB Cache
(Dual Processor Option)
•Intel Extended Memory 64Technology
• 1U Chassis with 420W power supply
•Supermicro server board with Intel® E7320 (Lindenhurst
VS) Chipset
•Kingston 512MB DDR333 ECC Reg. RAM (2x256MB)
• 1 pc x Seagate 80GB SATA 7200RPM hard drive
• 2 x 1" Hot-swap SATA drive bays
•Integrated ATI Rage XL SVGA PCI video controller
•2x Intel® 82541 Gl Gigabit Ethernet Controllers
•2x SATA Ports via 6300ESB SATA Controller RAID 0,1
Supported
$999
2U Xeon Processing Server
SDR-2103T
High-density 2U platform optimized for
performance and flexibility; ideal for
Web hosting, data center, terminal
services and High Performance
Computing (HPC)
• Intel Xeon Processor 3.0Ghz with 800FSB1 MB Cache
(Dual Processor Option)
•Intel Extended Memory 64Technology
• 1U Intel Chassis with 700W PFC power supply
•Intel® Server Board SE7520JR2
•Kingston 512MB DDR333 ECC Reg. RAM (2x256MB)
•6pcs xSeagate 160GB SATA/150 W/ncq 7200rpm 8MB
Cache
•Intel SRCS16 6Channel SATA RAID Controller Card
•Integrated ATI Rage XL SVGA PCI video controller
•2x Intel® PRO/1000 MT Server Network Connections
(Intel® 82546GB controller)
$3,499
5U File Server
SDR-5301S
Outstanding performance, excellent
data protection, and advanced
management for departmental servers.
• Intel Xeon Processor 3.0Ghz with 800FSB1 MB Cache
(Dual Processor Option)
•Intel Extended Memory 64Technology
•Intel SC5300LX Chassis with Redundant 730W Power
Supply
•Intel server board w/lntel® E7520 (Lindenhurst) Chipset
•Kingston 1024MB DDR400 ECC Reg. RAM (2x512MB)
•Adaptec 2200S SCSI RAID Controller Card
•Include 6-Drive SCSI Hot-Swap Cage Kit
• 6 x Seagate 36GB SCS110K RPM U320 SCA hard drive
•ATI Rage XL SVGA PCI video controller with 8MB of video
memory
•Dual Intel® PR0/1000 Server Network Connections
$4,999
3U Clusterable SATA SAN Nodes
SDR-3303T
Provides a flexible, price and
performance advantages to the storage
needs of Small to Medium Business
(SMB) market segment.
•Intel 3U SAN Storage Enclosure w/700W Redundant Power
Supply
•Integrated Intel SE7501HG2 Server Board
•2x Integrated SAN Intel Xeon 3.06Ghz/533FSB Processor
•Integrated SAN 2x256MB Compact Flash Memory Cards
w/SAN Mgt Software
•3 X Integrated Intel SRCS16 6-Channel SATA RAID Cards
•Kingston 1024MB DDR266 ECC Reg. RAM (2x512MB)
• 16pcs x Western Digital WD2500SD RAID SATA
•Integrated SAN Management Software & Storage System
$13,599
Big business power, small business price tag
Your business requires solid server solutions. With Servers Direct server systems based
on the Intel® Xeon™ Processor, you can count on high availability, maximum efficiency
and proven performance to help you meet your business reliability requirements.
1 . 877 . 727.7127 sales@serversdirect.com
Development
of a User-Space
Application for
an HID Device,
Using libhid
When it's time to get a new device working on
Linux, every piece of information helps, whether it's
reading the hardware documentation, snooping
data, reading sample code or even running utilities
on a non-Linux OS. by eoin verling
T he Matrix is a USB bill validator, sometimes known as
a note reader or bill acceptor, made by Validation
Technologies International. The bundled software was
developed for Microsoft Windows, but fortunately the
device comes with low-level technical documentation that
defines device-specific aspects, such as flow control, status
bytes and local status LEDs.
The device is a Human Interface Device (HID), as iden¬
tified by an enumeration process upon connection. The
Windows device manager reports the device as such, as
does usbfs on Linux. This article is specific to this particu¬
lar HID device, so including all of its code probably is
unnecessary, but it should provide help for developing
other HID-class devices.
After some initial research, I decided to develop user-space
code using an in-development library called libhid, which pro¬
vides a cross-platform way to access and interact with USB
HID devices, libhid is implemented on top of libusb, so it
does not depend directly on the kernel’s USB support.
Another option for driving the Matrix is to use libusb
directly, but doing so would be re-inventing the libhid
wheel. A third option is to implement the Matrix driver as a
kernel module, but it would incur the large overhead of
learning kernel particulars. This option also would render
the code platform-specific.
Device
descriptor
1
Siring
descriptor
f
Report
descriptor
Physical
descriptor
Figure 1. A USB device's descriptors, stored in its ROM, hold information about it.
sticks and force-feedback game controllers. Also included in
the HID class are devices that may not require human interac¬
tion but do provide data in a similar format to HID-class
devices, such as bar-code readers and, in my case, the Matrix
note reader.
Information about a USB device is stored in segments of its
ROM called descriptors. A diagram of the descriptor structure
is provided in Figure 1, where an overall view of the hierarchy
can be seen. When a USB device is attached to a USB bus, an
enumeration process takes place that equates to the descriptors
on the device being read into memory. Information about an
HID-class device is contained in its HID report descriptors.
I plugged the device in to the Linux box in order to read the
descriptors and monitor the device, the machine and the com¬
munications. I did this to try to get as much information as
possible so I could have a better understanding of how to write
code for the device.
A key component of these report descriptors is the usage
information, which is defined in the USB HID Usage Tables
(see the on-line Resources). Usage values describe three basic
types of information about the device:
■ Controls—information about the state of the device, such as
on/off or enable/disable.
■ Data—all other information that passes between the device
and the host.
■ Collections—groups of related controls and data.
Investigation
USB devices are categorized into device classes. A modem is
in the communications class, and a speaker falls into the audio
class. The HID class mainly consists of devices that people use
to control computers. Examples of HID devices are mice, joy-
Taken together, the usage page and usage number define a
unique constant that describes a particular type of device or
part of that device. For example, on the Generic Desktop usage
page (page number 0x01), usage number 0x05 is a game pad,
and usage number 0x39 is a hat switch.
421 OCTOBER 2005 WWW.LINUXJOURNAL.COM
Because my device is unique—it isn’t a mouse, joystick or
something commonly found in the examples of HID-class
devices—the usage page is set to 65,440, which is a vendor-
defined value. In comparing outputs of lsusb for other HID-
class devices, they all had a defined usage page, such as
Generic Desktop Controls or Game Controls. Because libhid
still is in development, few previous examples of code are
available to browse for reference. My work was much like an
exploratory investigation.
On Linux, with a standard Debian 2.6.9 kernel and usbutils,
I was able to see that Linux recognises the device as a USB
HID device, blnterfaceClass = HID, and loads the hiddev ker¬
nel module. This module, or piece of kernel code, is a generic
driver for HID devices. It is not specific to our needs—it main¬
ly is used for mice, joysticks and the like—so it needs to be
detached from the device or disabled (see the Communicating
with the Device section).
The device, like all USB devices, is enumerated upon
connection to the USB bus. So looking at the output of lsusb
-vvv, run as root, for more information is helpful in determining
what the device capabilities are. lsusb parses the usbfs filesys¬
tem into a more readable format:
[sample lsusb -vvv]
Bus 001 Device 004: ID 0ce5:0003
Device Descriptor:
idVendor 0x0ce5
idProduct 0x0003
Configuration Descriptor:
Interface Descriptor:
bNumEndpoints
blnterfaceClass
blnterfaceSubClass
blnterfaceProtocol
1
3 Human Interface Devices
0 No Subclass
0 None
HID Device Descriptor:
Report Descriptor: (length is 32)
Item(Global):Usage Page,data=[0xa0 0xff]65440
(null)
Item(Local ):Usage, data= [ 0x01 ] 1
(null)
Item(Main ):Collection, data= [ 0x01 ] 1
Application
Item(Local ):Usage, data= [ 0x03 ] 3
(null)
Item(Global):Logical Minimum,data=[ 0x00 ] 0
Item(Global):Logical Maxi mum,data=[ 0xff ]2 5 5
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x05 ] 5
Item(Main ): Input, data= [ 0x02 ] 2
Data Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position
Non Volatile Bitfield
Item(Local ): Usage, data= [ 0x05 ] 5
(null)
Item(Global):Logical Minimum,data=[ 0x00 ]0
Item(Global):Logical Maxi mum,data=[ 0xff ]255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x05 ] 5
Item(Main ): Output, data= [ 0x02 ] 2
Data Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position
Nonvolatile Bitfield
Item(Main ): End Collection, data=none
The above output—some of the information has been omit¬
ted—follows the hierarchy depicted in Figure 1. Some values
of note are:
■ idVendor and idProduct—unique identifiers for all USB
devices, used for identifying and accessing the device
in code.
■ bNumEndpoints—lists the number of endpoints available in
a device. This value actually means the number of endpoints
in addition to the default endpoint, endpoint 0, available in
every USB device.
1 QU
IER 42
nerator
ride network
O Free tech support
LHj
O Redundant UPS and ge
O Nations
2U
256kbps -80GB
$60/mo.
4U or Mid-tower
256kbps -80GB
$80/mo.
1/4 Rack
512kbps (14U) -165GB
$200/mo.
1/2 Rack
1 mbps (28U) -330GB
$350/mo.
www.layer42.net
All prices include 100Mbps port, Firewall,
24x7 Monitoring and DNS hosting
408-450-5740 2336-F Walsh Ave„ Santa Clara, CA 95051
WWW.LINUXJOURNAL.COM OCTOBER 20051 43
■ blnterfaceClass—the value that determines that a device is
an HID-class device.
■ blnterfaceSubClass—the subclass of a device, in this case,
HID. For example, the boot interface subclass of the device
must be bootable or available to the BIOS, such as a mouse
or keyboard.
■ blnterfaceProtocol—the protocol used. Possible values are 0
for none, 1 for keyboard or 2 for mouse; additional informa¬
tion is available in the HID spec.
Communicating with the Device
A block diagram depicting the flow of control of data is shown
in Figure 2. It may help in picturing where your code fits in
with respect to the libraries and the device. From my investiga¬
tion, I know that control messages periodically are written by
way of the control pipe, and interrupt reads are made through
endpoint 0.
The control pipe is used for three tasks: receiving and
responding to requests for USB control and class data; trans¬
mitting data when polled by the HID class driver, using the
Get_Report request; and receiving data from the host. The
Interrupt pipe is used for two tasks: receiving asynchronous, or
unrequested, data from the device and transmitting low-latency
data to the device.
What is the 5-letter word for
“an Inexpensive Compact Panel PC”?
The PPC-E5 is the Solution that Fits !
Setting up a Linux Panel PC can be a Puzzling experience. However, the
PPC-E5 comes ready to run with the Operating System installed on flash
disk. Apply power and watch the Linux X Window Graphic User Interface
appear on the vivid color LCD. Interact with the PPC-E5 using the
responsive integrated touchscreen. Everything works out of the box,
allowing you to concentrate on your application rather than building and
configuring device drivers. Just Write-lt and Run-lt. Starting at $995.
clilAC, inc.
Phone 618 - 529 - 4525 Fax 618 -457 -0110
2390 EMAC Way, Carbondale, Illinois 62901
World Wide Web: http://www.emacinc.com
myDri ver
libhid
I ibusb
Ctrl
pipe
int
pipe
device
Figure 2. The new driver uses libhid, which depends on libusb.
The kernel has a DEBUG feature that can be activated
in order to log extra information about what is happening
when communicating with the device. To do this, a file in
the kernel source needs to be modified. In the /usr/src/linux/
drivers/usr/input/hid-core.c file, these two lines need to be
changed from:
#undef DEBUG
#undef DEBUG_DATA
to:
#define DEBUG
#define DEBUG_DATA
The module needs to be recompiled and installed. Once this
is done, the modules should prove helpful in determining
whether your code is working and doing what you expect.
Sample code containing some helpful comments comes
with libhid. The file test_libhid.c in the libhid/test directory is a
good place to start writing code for the device. Below is a
snippet of that code, along with some more explanation of the
functions; details are omitted for brevity:
HIDInterface* hid;
hid_return ret;
HIDInterfaceMatcher matcher =
{ 0x0ce5, 0x0003, NULL, NULL, 0 };
ret = hid_force_open(hid, 0, &matcher, 3);
int const PATH_LEN = 2;
int const PATH_IN[2] = { 0xffa00001, 0xffa00003 };
int const WRITE_PACKET_LEN = 8;
char write_packet[8] =
{ 0x04,0x7f,0x7f,0x00,0x02,0x00,0x00,0x00 };
int const READ_PACKET_LEN = 5;
char read_packet[5] ;
ret = hid_set_output_report(hid,
PATH_IN,
441 OCTOBER 2005 WWW.LINUXJOURNAL.COM
PATH_LEN,
write_packet,
WRITE_PACKET_LEN);
ret = hid_interrupt_read(hid,
USB_ENDP0INT_IN+1,
read_packet,
READ_PACKET_LEN,
0 );
The first thing to do is identify the particular device we
want to talk to. This is done with the HIDInterfaceMatcher call
simply by entering the vendor ID and the product ID. These
two identifiers are all that is required to identify any USB
device. If you have more than one identical device, it is possi¬
ble to separate them by serial number, that is, two Matrix note
readers would have the same vendor ID and product ID but
different serial numbers. The HIDInterfaceMatcher call can do
this; see the comments in the test_libhid.c file.
After some variable setup, the next step is to detach the
kernel driver from the HID device. Upon insertion of the HID
device, the kernel usually loads the usbhid module, which we
don’t want. We do have a few options, however, for unloading
it or for not loading it in the first place. One such way is to
enter this command:
Figure 3. Understanding a device: one way to browse the available nodes of the
HID tree is to use the SystemSoft HID Browser.
2. A Windows application available from Amaud, one of the
libhid authors, also parses the usage tree and produces a
nice GUI output, as shown in Figure 3.
3. By parsing the output oflsusb -vvv, run as root, it is possi¬
ble to parse the tree manually to determine the path. This
process is explained in the comments of test_libhid.c code.
root@localhost #> modprobe -r usbhid
When the hid_force_open function
runs, it attempts, n times, to detach the
device before it fails. The device is
now free from any control, so our code
now “opens” the device. As with any
USB device, it is necessary to send
control information to the device to
activate it. This information must be
sent periodically in order for the device
to remain active. If the control poll
stops, the device deactivates after a cer¬
tain timeout.
Writing to a device requires the
HID usage path and its length, plus a
packet and its length. To find this out,
we need to parse the usage tree—the
output of 1 s u s b - v v v—and obtain the
path to the interface we want. As with
everything else, there are various meth¬
ods for determining the path. At this
stage, a lot of time was spent determin¬
ing what path to write to, and a number
of tools are helpful here, such as:
1. The test_libhid.c code: when the
correct vendor and product ID are
entered in the code, the function
hid_dump_tree, which uses the
MGE hidparser (see Resources),
which parses the HID usage tree and
places the available usages at its
leaves, outputs the available paths.
$119
qty 100
200 MHz ARM9
10/100 Ethernet
PC/104 bus
TS-7200 ARM9 Single Board Computer
$ 149 qty 1
* 32 MB SDRAM
(64 MB optional)
a 8 MB Flash
(16 MB optional)
* Compact Flash
» 10/100 Ethernet
* 2 USB ports
* 20 Digital I/O
» 2 Serial Ports
Options:
Shown with optional Compact Flash
9i Boots Debian stable from Compact Flash
x Boots TS-Linux from on-board Flash
" RS-485
" 8ch 12-bit A/D
* RTC (battery-backed)
We use our stuff.
Visit our TS-7200 powered website at
9i Many x86 and ARM based
SBCs and peripherals available
9i Call for custom designs
(480) 837-5200
www.embeddedARM.com
WWW.LINUXJOURNAL.COM OCTOBER 20051 45
From the above methods, we now have a path value we can
use for the hid_set_output_report. Once we know where to
write to, it’s a matter of what to send. This information should
be in the technical documentation that comes with the device,
and it can be verified with the USB-sniffing tools. As with the
particular device I was using, verifying the format of a packet
with the sniffing tools turned out to be important, as the infor¬
mation in the documentation didn’t match what the sniff log
reported (see the Snooping section).
Once the control message or output report is sent, we can start
to read from the read pipe, endpointO. The function needed is an
interrupt read function. It already exists in libusb, but a corre¬
sponding libhid function doesn’t. The developers of libhid simply
hadn’t come across a device that required it yet, so I studied the
format of the other functions and implemented it appropriately. I
also added a new error code to the existing list. These additions
are being considered for inclusion in the latest version of libhid.
At this stage, once the interrupt read value is stored, I then
parse this value, as per the Matrix documentation, to display
the results to the user. For this device, that equates to informa¬
tion such as, “A ten-euro note has been inserted” or “The cash
box is disconnected” and other such device-specific informa¬
tion. The details are unnecessary for the purposes of this arti¬
cle, but if anyone requires this detail, feel free to contact me.
This process is repeated for as long as the driver is running.
We must keep polling the device to keep it active. There is a
status LED on the device that turns green when the device is
active and remains orange when inactive. The goal for quite
some time was to make the little light go green.
Snooping
Snooping can be done with a number of utilities. This is where
I learned about the discrepancies between what the Matrix doc¬
umentation says and what actually happens:
[5037 ms] <<< URB 647 coming back <<<
-- URB_FUNCTION_CONTROL_TRANSFER:
PipeHandle = 8180C814
TransferFlags = 00000002 (DIRECTI0N_0UT)
TransferBufferLength = 00000005
TransferBuffer = 92al37ed
T ransferBufferMDL = fe9876e8
UrbLink = 00000000
SetupPacket
00000000: 21 09 00 02 00 00 05 00
[5038 ms] <<< URB 645 coming back <<<
-- URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER:
PipeHandle = fe9876a0 [endpoint 0x81]
TransferFlags = 00000003 (DIRECTION_IN)
TransferBufferLength = 00000005
TransferBuffer = fefeef08
TransferBufferMDL = 81al8f48
00000000: 00 20 00 00 la
UrbLink = 00000000
[5038 ms] >>> URB 648 going down >>>
-- URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER:
PipeHandle = fe9876a0 [endpoint 0x81]
TransferFlags = 00000003 (DIRECTION_IN)
TransferBufferLength
TransferBuffer
TransferBufferMDL
UrbLink
00000005
fefeef08
00000000
00000000
From the snoop log, we see the control message sent to the
device at the start, followed by a series of interrupt reads.
According to the documentation, “The Host sends [a] poll to
request information from Matrix at a periodic rate. Matrix
answers to the poll and reports all the happening events.” So, my
interpretation of this was to send periodic control write messages
to the device and read the responses from the interrupt endpoint.
Also according to the documentation, the format of the write
message is five bytes in length, so with this information, I used
the test_libhid.c file included with libhid to see what happens. I
found that functions within libhid give error codes if they fail and
that the /var/log/messages file, with the extra DEBUG informa¬
tion from the modified kernel file, reports useful errors.
Upon closer inspection of the snoop log, I saw that the
control write was, in fact, eight bytes in length. See
SetupPacket in snoop log output. The five bytes described
in the documentation seemed to represent the first five bytes
of the packet, and the last three bytes seemed to be padding.
That is, changing these last three bytes doesn’t seem to
affect the operation. Subsequent error-free testing, with the
packet set to eight bytes, confirmed that the documentation
had been misleading.
Conclusion
In terms of where to start with this project, I found the mailing
list for libhid to be helpful. The libusb mailing list also provid¬
ed guidelines. The Linux usbutils are quite useful in determin¬
ing what interfaces are available on the device and the meaning
of the descriptors.
The libhid source code, still in constant development, also
is a source of help. Because the code constantly is being devel¬
oped, it is a good idea to keep an eye on the Subversion reposi¬
tory for changes, including documentation changes such as
helpful comments in the code.
Acknowledgements
Special thanks to Charles Lepple and Arnaud Quette, the origi¬
nal authors of libhid, and also to Martin F. Krafft, who later
joined and led the rewrite. They all provided me with a lot of
help, and without them I certainly wouldn’t have gotten my lit¬
tle light to go green.
Also, thanks to my supervisor, Dr Paul O’Leary, at WIT, for
his encouragement and analytical skills. It always is good to have
an experienced pair of eyes to guide me in the right direction.
libhid uses the HIDParser framework made available
by MGE.
Resources for this article: www.linuxjournal.com/article/
8275.«#
of ceol agus
Eoin Veiling (everling@theverlings.conn) gualified
in 1998 and has been a sysadmin since. He cur¬
rently is undertaking a research Master's in parallel
computing at Waterford Institute of Technology,
Ireland. There's nothing he likes better than a bit
craic!
461 OCTOBER 2005 WWW.LINUXJOURNAL.COM
The Only Silo
"Edison's light bulb was important not because he was the first with the idea; as
many as ten others envisioned similar schemes. Rather it was significant because he
conceived not just a bulb but a whole electrified world."—Teresa Riordan, US News
BY DOC SEARLS
W e’ve been fighting closed and proprietary soft¬
ware for a long time now. And we’ve had lots
of success—enough, I think, that we need to
move to the next stage: to the marketplace.
We can see the problem when we look at how many closed
systems have open foundations: Google and Amazon on
Linux, Apple’s Mac OS X and Yahoo’s search infrastructure
on BSD. Also, countless closed Web habitats served up
by Apache.
Am I being unfair here? Perhaps a little. You can’t be open
in every possible respect, right? Some stuff needs to be locked
down or closed off. Customer data, future product plans, trade
secrets and “secret sauces” of one kind or another. But those
aren’t the issue.
The real issue is silos: closed habitats that serve as private
marketplaces that lock customers in and competitors out.
Dick Hardt of Sxip Networks gives a killer talk about
“Identity 2.0”. As Dick puts it, Identity 1.0 is a province of
walled gardens. Amazon, eBay, Flickr and Skype are all walled
gardens. They may be lovely places to hang out, but they are
also enclosed and private market spaces, as false in their own
way as the faux Venetian canals and Parisian streets in Las
Vegas hotels.
What makes them most different from closed systems of
the traditional sort is not a lack of interoperability—often they
have that—but the lock-in of personal data. You can’t take
and then it hits you://
LINUX ISN’T A POLITICAL STATEMENT.
IT’S A PROFIT STATEMENT.
Novell.
find out more at novell.com
©2005 Novell, Inc. All rights reserved. Novell is a registered trademark of Novell, Inc. in the United States and other countries.
WWW.LINUXJOURNAL.COM OCTOBER 2005147
your eBay reputation, or the business you’ve built inside
eBay’s walled garden, over to Amazon. Even trivial data, such
as your Skype contact list, isn’t portable. It’s locked inside a
space that is not your own.
To hear Dick describe it, Identity 1.0 is barely past
medieval. It’s a country of duchies and city states. But since
we’re so used to it, we can barely think outside its walls. Yet
that’s where we belong, he says. The world we want—the
Identity 2.0 world—is one of independent actors: free-range
customers, conducting business and building relationships in
ways that each individual controls and that work with many
different vendors.
The problem with the walled garden metaphor is that all the
familiar examples are native to the Web. Silos, on the other
hand, are everywhere, both on the Web and off. Nearly every
familiar business category you can name—banking, hospitality,
retailing, commercial aviation, car rental...even office equip¬
ment, such as copiers and printers—is a forest of silos. Take
airlines. I am a registered frequent flyer with United,
American, Delta, America West and Southwest. Yet the only
common way I can relate to any of the five is money. None of
my data in the United silo is available for my dealings with
American or Delta.
We’ve lived in a world packed with silos for so long
that we now confuse them with a free marketplace. We
left DG and formed a new systems integration company in
Raleigh called BAS, for Business Application Systems.
BAS’s goal was to produce what they called “machine-
independent software”. To be independent, Earl explained,
you needed software that was independent of every hard¬
ware vendor’s silo.
One day, early in my company’s relationship with BAS,
Earl explained the ideals of machine independence, all of
which are familiar to anybody acquainted with open source
today. (Although, naturally, BAS’s code was proprietary.)
When he was done with his rap, my business partner asked
the impolite question, “So how do you make your money?”
“We’re whores”, Earl replied. “We walk the streets with the
rest of them.” His point: they had no choice—except
among silos. (BAS, it turned out, bet on the wrong silo:
Texas Instruments’ DS990s.)
Silo was just one container metaphor kicking around in
those days. Others were smokestacks and stovepipes. Today
those metaphors have fallen behind silo in popularity. I sus¬
pect that’s because silos are completely contained. Unlike
smokestacks and stovepipes, they don’t have an opening at
the top.
The prototypical office building is a silo of sorts. With its
security systems, its employee and visitor badges, it comprises
what David Weinberger calls “Fort Business”:
The world we want—the Identity 2.0 world—is one of independent
actors: free-range customers, conducting business and building
relationships in ways that each individual controls and that work
with many different vendors.
actually believe that a choice of silos comprises all the con¬
ditions required for a free market. We can see how limited
this is when we look at the market category we call com¬
puters. A quarter century ago, we thought the category was
free and open because we had a choice of silos from IBM,
Digital, Data General, Wang and HR We thought the same
way about networks when our choices were OmniNet,
WangNet, IBM Token Ring, Sytek, Corvus, 3Com and
Ungermann-Bass.
I remember a long conversation I had with Ralph
Ungermann about how his company’s goods were “open”
because they interoperated with other networks. In a rela¬
tive sense, they may have been. But the market was essen¬
tially a field of silos. What he offered was inter-silo-oper-
ability. Good for its day, but nothing like the Net that was
to come—and which didn’t come from any one vendor
at all.
I remember Earl Gillmor talking about silos, way back
in 1980. Earl enjoyed a small measure of notoriety as a
member of a splinter group at Data General that lost “the
shoot-out at the Holiday Inn” in Durham, North Carolina—
an event immortalized in the early pages of Tracy Kidder’s
book, The Soul of a New Machine. After the shoot-out, Earl
This fort is, at its heart, a place apart. We report there every
morning and spend the next eight, ten or twelve hours inacces¬
sible to the “real” world. The portcullis drops not only to keep
out our enemies, but to separate us from distractions such as our
families. As the drawbridge goes up behind us, we become
businesspeople, different enough from our normal selves that
when we first bring our children to the office, they’ve been
known to hide under our desks, crying.
Within this world, the Web looks like a medium that exists to
allow Fort Business to publish on-line marketing materials and
make credit-card sales easier than ever....The Web isn’t primari¬
ly a medium for information, marketing or sales. It’s a world in
which people meet, talk, build, fight, love and play. In fact, the
Web world is bigger than the business world and is swallowing
the business world whole. The vague rumblings you’re hearing
are the sounds of digestion.
The change is so profound that it’s not merely a negation of
the current situation. You can’t just put a big “not” in front
of Fort Business and say, “Ah, the walls are coming down.”
No, the true opposite of a fort isn’t an unwalled city. It’s
a conversation.
4 8 ■ OCTOBER 2005 WWW.LINUXJOURNAL.COM
David wrote that more than six years ago, in Chapter 6
of The Cluetrain Manifesto. We still aren’t having the con¬
versation required to bring the walls down. True, there are
some significant conversations growing out of employee
blogs. For example, nothing has done more to bring down
Microsoft’s walls than interaction with outsiders by Robert
Scoble, Kim Cameron (a subject of last month’s column)
and about 2,000 other blogging Microsoft employees.
But the problem isn’t communication. It’s the structure
of markets themselves. I’m not talking about structure in an
architectural sense, but in a deeper way that’s more like
geology. Because the Internet is geological, not just archi¬
tectural. It has a nature that goes deeper than whatever
structures private efforts can provide. But that nature is
hard to see when your frames of reference are closed
and proprietary.
Like many in the Linux community (including my good
friend Eric Raymond), I have strong Libertarian sensibili¬
ties. I understand the liberating advantages of private prop¬
erty to societies and their economies. Ownership matters,
and ownership works. But we in the Free Software and
Open Source communities also know there are some things
that are beyond the scope of ownership and the control
ownership naturally implies. Earth below the crust is as
beyond the practical scope of ownership as the weather and
the stars. Yet they provide us with services so fundamental
we couldn’t live without them. One of those services is a
deep and easily ignored context for property: gravity. Real
estate would be meaningless without the gravitational pull
provided by a mass we’ll never see. The Net’s geology is
like that.
I’ve written many times about the NEA nature of the Net,
and of all free and open-source software: Nobody owns it,
Everybody can use it and Anybody can improve it. The same
applies to markets, and it’s time we started improving the ones
we’ve got, by putting silos in a context that makes clear their
limited advantages.
The Supreme Court missed a chance to do that with the
Brand X case. In a 6-3 decision that was handed down on
June 27, 2005—the same day as Grokster, which is a big
reason why not much of a fuss was made about it—the
Supremes upheld a 2002 FCC ruling that classified cable
broadband as a deregulated “information service” rather
than a “telecommunications service”. Unpacked, that means
the cable and telephone companies can (and will) be exclu¬
sive Internet service providers. Independent ISPs like
Brand X and Earthlink, which don’t own physical connec¬
tions to homes and businesses, are out of luck if the cable
and phone companies want to keep captive customers
to themselves.
More important, the FCC’s understanding of the Internet
achieved the stature of law with the Brand X decision. That
understanding is basically feature-rich broadcast. It’s a concept
of service anchored on the supply side of the highly asymmet-
and then it hits you://
YOU’RE WORKING WITH
LINUX 24/7 IN
EIGHT TIME ZONES. AND SO ARE WE.
Novell.
find out more at novell.com
©2005 Novell, Inc. All rights reserved. Novell is a registered trademark of Novell, Inc. in the United States and other countries.
WWW.LINUXJOURNAL.COM OCTOBER 2005149
What makes Linux so different, and so
successful, is that it's not designed as a silo.
rical distribution system the FCC has governed for most of the
last century.
Former FCC Chairman Michael Powell, in a speech at
the VON (Voice On the Net) conference one year ago, said,
“To realize the innovation dream that IP communications
promises, however, we must ensure that a willing provider
can reach a willing consumer over the broadband connec¬
tion.” He generously acknowledged “the importance of
consumer empowerment” and rights such as “Freedom to
Access Content”, which he explained with “Consumers
should have access to their choice of legal content.”
Generous as that may have been, it was no less top-down
than anything owned by Rupert Murdoch. Nowhere did he
acknowledge the Net’s most profound commercial grace:
supporting the ability of people to go into business, and to
do business, with anybody they please, anywhere.
Thomas Madsen-Mygdal, a young serial entrepreneur in
Denmark who hosts the delightful annual reboot conference in
Copenhagen, recently told me he likes and appreciates Flickr—
the on-line photo gallery phenomenon that has taken the world
by storm (and which was built on LAMP)—but that it has
“lock-in” issues:
I don’t mean “a total lock-in” in the traditional BigCo IT
sense. More like that if open data standards existed, the col¬
lective value would be in the commons—not on one photo
sharing site. The London bombings wouldn’t be about the
“flickr tag”, but about the “photo tag” or just the tag —which
in my book is much more aligned with our values and the
society we want to create.
Thomas’ higher-level concern is that “we’re selling out on
values of open standard and decentralization”. What Tim
O’Reilly calls the “architecture of participation”, Thomas says,
is turning into something that is “based on silos” in practice.
So, he adds, “I’m gonna try and push some open standards in
the photo sharing space to level the competition.”
He’ll do that, he says, by “dividing what is the commons
from what is the product. That way, thousands of photo sharing
products can create a collective value that’s a lot greater.
Competition will be on the product side, rather than on who
aggregates most of the commons.” Thomas’ site,
23people.com, is open for beta.
In September 2005, O’Reilly put on its second Web 2.0
conference. (Shouldn’t they call it Web 2.1?) In September
2004, Tim O’Reilly described Web 2.0 as “the Internet as a
platform”. Then he added:
We heard about that idea back in the late 1990s, at the
height of the browser wars, but that turned out to be a
false alarm. But I believe we’re now starting the third age
of the Internet—the first being the telnet-era command-line
Internet, the second the Web—and the third, well, that tale
grows in the telling. It’s about the way that open source
and the open standards of the Web are commoditizing
many categories of infrastructure software, driving value
instead to the data and business processes layered on top
of (or within) that software; it’s about the way that Web
sites like eBay, Amazon and Google are becoming plat¬
forms with rich add-on developer communities; it’s about
the way that network effects and data, rather than software
APIs, are the new tools of customer lock-in; it’s about the
way that to be successful, software today needs to work
above the level of a single device; it’s about the way that
the Microsofts and Intels of tomorrow are once again
going to blindside established players because all the rules
of business are changing.
That was a lead-in to the Web 2.0 conference. After
the conference, in an interview with Richard MacManus,
Tim said:
I actually ran a couple of panels on this at our Open Source
convention, a year and a half or two years ago—called “Open
Data—Do We Need a Bill of Rights for Web Services?” We had
people from Amazon, eBay and others trying to answer that
question: what does it mean when we’re investing our on-line
data in these sites? Will we end up with something like the
Open Source movement because the companies have ended up
locking in their users?
....But the actual data ownership is maybe less important, in
some areas, than people think. When we talk about user-
contributed data, we’re not just talking about my data prop¬
er (as in having your mail stored on Gmail or Yahoo! Mail
or whatever). We’re also talking about a kind of content
that users are contributing to a collective work. So for
example, Amazon Reviews—people don’t really care about
that in the same way. They’re not saying, “Oh I created that
review and I want to be able to export it to Barnes & Noble
as well.” They’re creating it in a particular context of
that community.
....Despite what I’ve said...data lock-in absolutely should be
a concern. I believe that data lock-in of various kinds is
going to be one of the key tools of business advantage in
the Internet era. I think that as companies realize this, they
will figure out how to be evil—so to speak (to use Google’s
terminology)—and I predict that we will in fact have some
major battles in that area.
As I said last month, one answer is to create ways to do
what Drummond Reed calls “Company Relationship
Management” (or CoRM), which should look far more
interesting and useful to companies than their own
Customer Relation Management (CRM) systems, which by
501 OCTOBER 2005 WWW.LINUXJOURNAL.COM
nature have no view outside the company’s own silo. In
fact, CRMs are one of the main ways companies maintain
their silos.
Another is to pay more attention to where the Net’s deep,
almost geological market-making infrastructure comes from.
It’s not from the physical cables that run to homes or from the
“services” available exclusively from cable and phone compa¬
nies, but from the open protocols that define the Net’s environ¬
ment. It’s also not from fancy private services inside corporate
walled gardens but from the raw building materials that make
deploying those services so free and easy.
Which brings us back to the L in the LAMP suite that
makes possible the last phrase above.
What makes Linux so different, and so successful, is that
it’s not designed as a silo. Linux didn’t come from a silo,
and it had no ambitions to be a silo. At one point, Linus
talked about “world domination”, but his tongue was in his
cheek—even if he was indulging in prophesy that would
prove out in the long run.
Linux was never a business. It was, and remains, a great
way to build anything, to support anything, for anybody.
That’s the fundamental virtue we need to fight for when we
go to battle.
Our battle, however, is not with the companies that use
open code to build walled gardens and silos. Our battle is
with the closed, top-down silo-oriented value system that
has been with us since the dawn of the Industrial Age. It’s
that lame old value system that prevents us from imagining
how we can improve markets that nobody owns and any¬
body can improve.
The best way to shed the old mentality is to embrace our
customers and not only their money. Today the preponderance
of inventiveness and productivity is out in the free world, in
the hands of free-range individuals. Linus Torvalds is the
prototypical example of one of those individuals. There are
countless more like him, producing all kinds of goods, expressing
all kinds of demand—much of which they are able to supply
for themselves, as Linus did, and with the help of others, as
the Linux community has done.
In fact, the only silo that matters is the most fundamen¬
tal and indivisible unit in the marketplace, the individual.
What we need is to create and support independence,
not dependence.
Work to free individuals, and to take advantage of what
they do with that freedom, and you’ll have a winning strategy
in the new marketplace we’re all making together.0
Doc Searls is senior editor of Linux Journal.
and then it hits you://
LINUX WORKS WITH ANY PLAN.
ESPECIALLY THE FY ’06 BUDGET.
Novell
find out more at novell.com
©2005 Novell, Inc. All rights reserved. Novell is a registered trademark of Novell, Inc. in the United States and other countries.
WWW.LINUXJOURNAL.COM OCTOBER 20051 51
Fixing Web Sites with
GreaseMonkey
Who says "View Source" on a Web page has to be
a read-only proposition? Re-mix your favorite Web
sites by changing styles, adding and removing
elements, and more, by nigel mcfarlane
H ere’s a strange thing: hacking open source isn’t done
only at midnight, in the spare room, hunched over the
protocol analyser, the breadboard, source code control
and some helpless device. No, sometimes it’s done
inside a different crucible entirely: a public world of shameless
posturing and self-promotion. A lurid and neon habitation of
signs, shops, styles and stuff populated by the babble of conver¬
sations both informed and banal. It’s a place of great joy and
great angst; a place of towering conservatism and the last bastion
of the radical voice. Within it, a good hairdo or a radically cut
legline can get you as far as a symbolic debugger, possibly even
further. Devices they may be, but of a different cut entirely from
those of hardware. Its denizens slip hyperactively in and out of
view like character actors with coffee addictions and inspired
agents. Of course, I refer to the World Wide Web.
In this article, you learn how to code in a new way, a way that’s
about changing media, not about changing programs. To enter this
nightclub and experience the beat, you need the right gear, and the
right gear is Mozilla Firefox and GreaseMonkey. Alfred Bester and
William Gibson are waiting, so ready your Mojo and prepare for
cyberspace insertion. But first, a bit of background.
Web Pages as Open Source
We tend to forget that the Web is open source, in a way. Some of
the Web’s infrastructure, browsers and servers, is traditional
open-source software, but the idea also applies to Web page con¬
tent. Appropriation of code is an everyday occurrence. Every day,
Web developers and designers use the View Source browser fea¬
ture to appropriate (industry term: steal) code and design from
other people’s pages. It was ever thus, and it remains so. Ideas
and code are shared freely and often; it’s an art design sensibility.
Most technical people have dabbled with Web develop¬
ment, and dabbling is an easy way to have a bad experience.
The big three technologies—HTML, CSS and JavaScript—
were riddled with bugs for many years after their inception.
That’s the experience that probably looms large for early
adopters who first tried it out in the 1990s and walked away in
disgust. Cross-browser code? No, thank you.
Fortunately, matters have improved tremendously as of late,
and the Web is reviving as a technology platform. Better stan¬
dards support, more standards support and the decline of hoary
old misgivings, such as Netscape Communicator 4.x and
Internet Explorer 5.0, have left Web developers with a nearly
clear shot at real portability, a shot frustrated only by the once
shiny but now fairly rusty Internet Explorer. In 2005, the buzz
is about Modern DHTML, Layout without Tables, Semantic
Markup and Asynchronous JavaScript and XML (AJAX).
Client-side Web development is coming back, and these are the
things of which it’s made. This time, the Web is backed by pro¬
fessionals with formal Web training and veterans with ten years
of experience. These people have their acts together, and it’s
possible to say things about Web technology that are no longer
drowned out by the static of incompatibility issues.
Supporting and colonizing this trend is the Mozilla Firefox
Web browser, and Mozilla technology in general. Of course,
Mozilla is fully open source, as open as a religious movement
can be, and so there’s plenty of room for experimentation. The
critical bit of Mozilla and Firefox is its interpreted nature. On top
of a big, bad, networked C++ rendering engine is a thin skin of
JavaScript scripts and XUL, an XML dialect. This makes Mozilla
a distant cousin to Emacs or Tcl/Tk, as it provides the whole
Firefox user interface by way of interpreted code. By writing an
extension, you can enhance this user interface and drop it in to
thousands of willing people’s daily experience. Go to
update.mozilla.org to see the endless possibilities made real by
this system. Every variant hardware device requires Linux kernel
driver support; every variant human expectation about user inter¬
faces requires a Firefox extension. That’s a lot of extensions.
Grabbing GreaseMonkey
GreaseMonkey is a Firefox extension (see the on-line
Resources). You have to click on the link twice, once to trust
www.mozdev.org and once afterward to install the extension.
GreaseMonkey differs from the other extensions because it
provides no specific user-interface enhancements of its own
other than a configuration dialog box. Instead, it creates a
macro-like scripting environment into which you put
JavaScript scripts. Those scripts operate on Web pages that you
specify. When such a page loads, your script goes to work on
the page content, no matter who provided it. You’re intercept¬
ing a content provider’s content and modifying it before it hits
you. No wonder GreaseMonkey’s been called “TiVo for the
Web”. I wrote about page modification tactics in Rapid
Application Development with Mozilla (Prentice Hall, 2004),
but GreaseMonkey has moved that idea into the mainstream by
supporting traditional Web-scripting techniques and by packag-
521 OCTOBER 2005 WWW.LINUXJOURNAL.COM
i i Take charge. Win the battle and take control, right from your comfy chair. It’s easy to conquer
La*/
'■"J. 4'' the challenges of managing serial devices in the data center with the CCM serial console
manager.* When used with DSView® 3, AVWorks®, or industry-standard SSH/Telnet client software,
you can remotely control servers, network gear, telco and power devices from a single interface.
With proactive alerts and offline buffering you can remotely diagnose failed devices and reduce
downtime without setting foot in the data center. Visit us at
Avocent
www.avocent.com/serialcontrol. And start looking for a new chair. The Power of Being There®
* Recliner recommended, but not included.
Avocent, the Avocent logo, DSView, AVWorks and The Power of Being There are registered trademarks of Avocent Corporation. © 2005 Avocent Corporation.
ing it all up into a digestible product.
For all Firefox extensions, you must restart Firefox com¬
pletely to finish the install. Use File—>Exit to do that safely.
Bucket-loads of pre-existing GreaseMonkey scripts are
available (see Resources). Before you get too excited though,
note that such scripts are tied to one Firefox installation and
have no effect on any server. On a Linux or UNIX box, such
scripts might affect a large user population, but they’re primar¬
ily a personal thing. For those readers switched on to people
problems, the broader implications should be obvious.
Spotlight on LinuxJournal.com
To see all this at work, in this article I hack the Linux Journal
Web site with GreaseMonkey. My esteemed editor, Don Marti,
even asked me to do this. A brave man indeed. [Maybe next
time they'll invite me to the Web site meeting. — Ed.]
Give me a hill and I’ll climb it. First up is a bit of scrutiny of
the site due for surgery. Recall it’s www.linuxjournal.com, if
you’re reading this in print. This is also the fun part; personal
tastes differ, and for user-side drivers—which effectively is what
GreaseMonkey scripts are—it’s entirely valid and professional to
be picky and subjective. In Mozilla-land, dogfood means testing
your fixed bugs for technical correctness, and catfood means test¬
ing your inventions against unreliable and subjective people who
might spring in any direction. It’s all catfood here, and there’s no
right or wrong. After reading this article, ZJ’s long-suffering site
maintainer will likely glare at me venomously or perhaps change
the site before this sees print. Design sensibilities, you see. Sorry
mate, they made me do it. Hard-core engineers should look away;
you might find this analysis distressing. On to the site.
Here’s a handful of observations.
■ The site icon, which appears in the location bar and on the
current tab if you use tabbed browsing is dinky and unin¬
spired. Oh well.
■ There’s advertising everywhere.
■ Linux Journal's supposed to be the granddaddy of technical
journals in open source, excluding academia and profession¬
al bodies. Where’s that indicated?
■ The headings are red. What’s with red? I’m not in a hurry.
On the plus side—my survival as a critic is at stake—the
site has a robust three-column layout and is clean overall.
Someone knows his or her stuff. Viewing the source, the layout
is all done with CSS, so that’s relatively modern; many indus¬
try sites still pump out the worst HTML you can imagine. The
excessive use of tags shows that LJ is halfway through
modernisation; there’s still some Semantic Markup work to go,
where meaningful tags are used as content descriptors instead
of the meaningless
. That update might improve the
site’s search engine performance or so it’s claimed.
The Right Tool for the Right Job
Now, of the above personal observations, some are simple to
rectify and do not require GreaseMonkey. If you dislike adver¬
tising, then the AdBlock extension is for you; there’s nothing,
or at least little, to code. Similarly, for a long time, all browsers
have supported user-specified stylesheets. If you install the
ChromEdit extension, you can get at that stylesheet without
having to grovel through the filesystem looking for it. Bring it
up via Tools^Edit User Liles, click the userContent.css tab
and start typing. To make headings blue, you might add:
hi.title a { color : blue Mmportant; }
@-moz-document domain(linuxjournal.com) {
hi.title a { color : blue Mmportant; }
}
The first rule applies to all Web sites; the second is a
Mozilla special that applies only to the Web site specified.
Browser-specific is okay here, because we’re working purely
on and in the client side.
You can get a lot done in these stylesheets, especially if you
know CSS well. You can hack the page’s layout to bits by
reordering, hiding or floating columns and other content. All of
these options are possible via GreaseMonkey as well, but
GreaseMonkey is better suited to bigger stuff. In other words,
don’t go to GreaseMonkey if page changes are easily solved
with a stylesheet; it’s overkill.
Lor this article, we’ll make one simple change. We’ll bring
some gravitas to the site by replacing some content with fancy
calligraphy drawn from another site.
Illuminated Drop-Caps for Paragraphs
The CSS :first-letter pseudo-selector lets you take an ordinary
paragraph of text and make the first letter big, so that several
lines of text flow around it. It’s a self-important feature and
what we’re looking for. We simply could apply that feature, but
most computers don’t have fancy medieval fonts installed. And,
a big Times Roman letter L isn’t that exciting. It would be bet¬
ter if we could get the LinuxJournal.com Web site properly illu¬
minated, like the Book of Kells, with extra fancy calligraphy.
Here are a couple of screenshots showing the before-and-
after looks, taken on Windows XP Professional. This is a time¬
ly reminder that the user experience is what’s important here. It
also emphasizes that open source means cross-platform when
stated in Mozilla terms. Everything described in this article
Figure 1. A Regular Linux Site
541 OCTOBER 2005 WWW.LINUXJOURNAL.COM
tiw t« Wo |HbfdH4 Ttm \mp
•#* - , & _ - 0® u.
Figure 2. That's better. We didn't need monks to illuminate this manuscript, sim¬
ply a GreaseMonkey script.
works identically on Windows, Macintosh, Linux and various
obscure Mozilla platforms, such as Solaris.
In the second screenshot, you can see that the first letter of
each paragraph has been replaced with a fancy illuminated letter.
Because I don’t have access to the back end of the LJ Web site,
that’s something of a feat. In fact, these images come from the
Australian National University’s Medieval Studies image server.
I’ve used the thumbnail images only. It’s a bit naughty to
serve up some other Web site’s images, and these images aren’t
perfectly cropped, registered scans, but for the purposes of, well,
illustrating a technique, they’ll do. Let’s hope some parsimonious
old sod doesn’t take them down by the time you read this.
Spinning Up the Script
To make this embellishment work, you need a GreaseMonkey
JavaScript script. To make such a script, proceed as though this
were any other Web page project. I saved to local disk the LJ
home page and then added this to the end of the section: