LINUX JOURNAL PERSONAL DESKTOP • GreaseMonkey • Project Utopia • Ruby on Rails • 3-D Audio • she OCTOBER 2005 ISSUE 138 AN _ ssc Take the money and exec(): bil reader driver PUBLICATION + WHERE'S MY FILE? Comparing file search tools Fight spam and win with THE BOOK OF POSTFIX l7 Managing your TODOs, from Evolution to Emacs An in-kernel FTP client Decrypting shc-encrypted shell scripts An 8-speaker f 3-D sound system Fixing other people's Web sites with GreaseMonkey USA $5.00 CAN $6.50 www.linuxjournal.com 1 0 o 71486 03102 JOURN Since 1994: The Original Magazine of the Li OCTOBER 2005 Community PLUG IT IN AND_ 1 illfi^D yjyj v Project Utopia makes desktop hardware play nice ov't-o-p-ba.ncl a. clm'mis-f rcL-/ior> chi/cl's p/ccy Systems Aclminis-fraJ-or AlterPath" Manager AlterPath" BladeManager AlterPath' OnSite The Next-Generation IT Infrastructure Cyclades AlterPath™ System is the industry's most comprehensive Out-of-Band Infrastructure (OOBI) system. The AlterPath System allows remote data center administration, eliminating the need for most time-consuming, remedial site visits. When fully deployed in your data center, Cyclades AlterPath System lowers the risks associated with outages, improves productivity and operational efficiency, and cuts costs. Each component of the AlterPath System is designed to seamlessly integrate into the enterprise, able to scale in any direction. Whether you need serial console management of networking equipment, KVM for access to Windows® servers, branch management, IPMI or HP iLO for service processor management or advanced power management, the AlterPath System delivers. Cyclades brings it all together, making OOBI administration seem like child’s play. Over 85% of Fortune 100 choose Cyclades. www.cyclades.com/lja 1.888.cyclades • sales@cyclades.com cyclades = 2005 Cyrlodes Corporation All right! reserved All other trademarks and product images are property of their respective owners Produo information subject to change without notice ■ 11 nmnntyt»'i tlMhlltllHlHI otlHUHlHttlllt ;/v>; 3 co V.wv.v • t *T 1 * f ' ( tj Er?e^CT 8x AMD Opteron Processor 940 sockets Supports 800 series Opteron CPUs with dual core tech. Up to 128GB DDR Registered ECC memory Support 4 Ranks memory module 1350W Redundant PSU 3+1 Support IPMI server management Industry 19" rack-mountable 5U chassis 4 x Gigabit Ethernet ports, and 4 PCI-X slots Up to 1 0 hot-swap HDDs with option HDD canister Modularization design, I/O may vary 8-Way AMD Opteron Server Benchmark Rating SPEC CPU2000: COMPILERS COMPARED SPECint_rate_base2000 IWILL H8501 ► Opteron sockets IWILL H8501 Barebone System = . V.V:.Vv;V.;>H AMD Opteron --It .. .. Mil *** . . .. ***** iMtumtimiiMMiti ‘V “HontiMUMimimimiKi,,, •m rv **•*•* •V rsV iVVAV '«> WAV* "I Accommodate 128GB RAM ( IWILL Other Outstanding^ Motherboards & Small Form Factor: ) ▼ DK8-HTX EQE3 TECHNOLOGY HTX allows access into the AMD64 Direct Connect Architecture through a variety of HTX connectors and slots, integrating outside devices directly resident on the HyperTransport technology bus. ▼ DK8ES ▼ DK88 ▼ ZMAX-DP / ZMAX-D2 Dual processors Small Form Factor • 2 AMD Opteron Processor 940 sockets • Supports 2xx Opteron CPUs with dual core tech. • Up to 16GB DDR Registered ECC memory • 1 x PCI-X 64bit 133/100/66MHz expansion slot • 2 x PCI 64bit 100/66MHz expansion slots • HTX-Pro support Pathscale InfiniPath HTX Adapter • 2 x Gigabit Ethernet ports (Intel chipset) • AGP 8MB on board & support IPMI • 2 AMD Opteron Processor 940 sockets • Supports 2xx Opteron CPUs with dual core tech. • Up to 16GB DDR Registered ECC memory • 3 x PCI-X 64bit 133/100/66MHz expansion slots • 2 x PCI-Express x16 expansion slots (one in PCI-Express x2 mode) • 2 x Gigabit Ethernet ports (Broadcom chipset) • AGP 8MB on board & support IPMI • 2 AMD Opteron Processor 940 sockets • Supports 2xx Opteron CPUs with dual core tech. • Up to 64GB DDR Registered ECC memory • 2 x PCI-Express x8 expansion slots • 1 x PCI-X 64bit 133/100/66MHz expansion slot • 1 x PCI-X 64bit 100/66MHz expansion slots • 2 x Gigabit Ethernet ports (Broadcom chipset) • AGP 8MB on board & support IPMI • 2 AMD Opteron Processor 940 sockets • Supports 2xx Opteron CPUs with dual core tech. • Up to 4GB DDR Registered ECC memory • lx AGP 8X, lx PCI and lx mini PCI slot (ZMAX-DP) • 2x PCI-Express support SLI Tech.(ZMAX-D2) • 3x 3.5" HDD bays, and lx 5.25" CD-ROM bay • lx GbE, lx IEEE1394, 8x USB 2.0 ports • 300W Power supply IWILL USA Corp. 9004 Research Drive Irvine, CA92618 Tel: +1 949 753-5488 Fax: +1 949 753-5499 Visit www.iwill.net for more information. Or contact us: sales @ iwillusa.com, oem@iwHlusa.com IWILL reserves the right to change specifications or other product information without notice. This publication could include technical inaccuracies or photographical errors. IWILL provides this publication as is without warranty of any kind, either express or implied, including the implied warranties of merchantability or fitness for a particular purpose. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions; therefore, this disclaimer may not apply to you. Iwf 11 COVER STORY 66 PROJECT UTOPIA Linux's long-standing tradition of isolating the user from the hardware might be great for security, but it can be a real pain when you just want to snag some photos from your camera or check for wireless access points you're allowed to use. But there is a plan. This month, Robert Love covers Project Utopia. LINUX JOURNAL OCTOBER 2005 ISSUE 138 FEATURES 52 FIXING WEB SITES WITH GREASEMONKEY This Web site is fine, but it could really use....Redesign other people's Web sites to your liking, on the fly. NIGEL MCFARLAN E 60 THE LINUX FOR KIDS EXPERIMENT Can a Linux dad get his family moved to a secure, easy-administration box without giving up the fun and education? PAUL BARRY 66 PROJECT UTOPIA Traditionally, Linux has protected the hardware from the user for security. When apps need to understand the hardware, new modes of communication are arising. ROBERT LOVE INDEPTH 72 BUILDING A CALL CENTER WITH LTSP AND SOFT PHONES You don't need to put a phone and a computer at every desk. Use a soft phone on an almost-thin client. MICHAEL GEORGE 78 DIRT CHEAP 3-D SPATIAL AUDIO Look out! Bogey at 10 o'clock high! Your next simulator project can have realistic sound above, below and on all sides of the user. ERIC KLEIN, GREG S. SCHMIDT, ERIK B. TOMLIN AND DENNIS G. BROWN 88 TAMING THE TODO Is your computer helping you get work done, or making more work for you? Try these software options to get your act together. SACHA CHUA COVER PHOTO: BETHANY PASEMAN EMBEDDED 42 DEVELOPMENT OF A USER-SPACE APPLICATION FOR AN HID DEVICE, USING LIBHD We won't show you the money, but we'll show you the code for the device that shows you the money. EOIN VERLING TOOLBOX 14 AT THE FORGE Ruby on Rails REUVEN M. LERNER 22 KERNEL KORNER Network Programming in the Kernel PRADEEP PADALA AND RAVI PARIMI 34 COOKING WITH LINUX Trekking through the Desktop Jungle MARCEL GAGNE 38 PARANOID PENGUIN Limitations of she, a Shell Encryption Utility NALNEESH GUAR COLUMNS 47 LINUX FOR SUITS The Only Silo DOC SEARLS 96 EOF The Universal Internet Time Source ADRIAN VON BIDDER REVIEW 71 THE BOOK OF POSTFIX DON MARTI DEPARTMENTS 4 FROM THE EDITOR 6 LETTERS 10 UPFRONT 70 NEW PRODUCTS 81 ADVERTISERS INDEX 95 MARKETPLACE NetworkManager gets notifications of new network hardware and available access points, so all you have to do is find a hotspot (page 66). NEXT MONTH HACK ANYTHING Internet radio doesn't have to tie you to the computer. Dan Rasmussen, Jon Morgan and Paul D. Norton have updated a classic radio design with the ability to tune in Internet streams. Stuart Brorson covers the electronic design automation tools needed to work with schematics and crank out professional-looking board designs that you can build yourself or order from a PCB house. If you think your favorite pinball machine is complicated now, try interfacing it to a Linux box. John Bork covers digital I/O technigues to help you connect to useful devices such as solenoids and switches. WWW.LINUXJOURNAL.COM OCTOBER 200513 rm -rf /opt/bs A Linux desktop shouldn't be a kick in the teeth. BY DON MARTI T o understand the IT indus¬ try, start with On Bullshit by Harry G. Frankfurt. Prof. Frankfurt poses, but doesn’t answer, the question of why there is so much B.S. in our society. He compares his subject to shoddy con¬ struction, and that’s an analogy we can work with, because in software we’re working at the thrilling edge of language and craftsmanship. We have the tools for dealing with B.S. in computer languages. Try to B.S. a compiler and that’s a bug. It’s time to tackle the B.S. problem head-on and start reporting bugs in human communications too. Consider this filler, I mean essay, to be a bug report on the big companies that are doing Linux for the desktop. “Let’s ‘position’ Linux as a simplified desktop for ‘transactional users’”, they say. That’s right—employees, if your company gives you Linux, that means Management thinks you’re a human servlet. Decision¬ makers and content creators get a proprietary desktop OS. Of course, offending the employees’ pride might not show up on a TCO spreadsheet. But no executive would want to admit to running a division full of transac¬ tional, replaceable, outsourceable “human resources”. But what about Clayton Christensen, disruptive innovation and The Innovator’s Dilemmal Doesn’t the cheap, good-enough contender always grow the fea¬ tures and stability it needs to win? Yes, when it lets in the customers left pressing their noses against the Expensive Stuff Store window. In the 1980s Macintosh let you do layouts even if you couldn’t afford phototypesetting. In the 1990s Linux let you put up a Web server without blowing the price of a Coupe de Ville on a UNIX box. But selling less-capable prod¬ ucts to customers who can get the good stuff doesn’t fly. Seen an F-20 at an air show lately? It was a capable airplane, but it was posi¬ tioned as an “export fighter” for air forces that weren’t allowed to have, or couldn’t afford, the F-16. Naturally, countries held out for the “real” fighter. Information free¬ dom ideals can go only so far when vendors patronize Linux cus¬ tomers. “Aww, the little transaction worker filled out a Web form! Isn’t that cute?” Desktop Linux marketing is doing more harm than good, but work is under way to make Linux out-perform the other OSes. Robert Love’s Project Utopia is bringing together the desktop interface and the necessary tweaking of hardware to make things work smoothly, not just securely (page 66). Michael George has an example of how a thin-client environment almost works to solve a problem, but the project needed one key local app, the soft phone. See a hybrid approach to a VoIP station that works as a phone and a PC on page 72. One of the projects where soft¬ ware excellence, not transaction- workerism, has triumphed, is Mozilla Firefox. Mozilla expert and author Nigel McFarlane died last month, leaving us with one last article (page 52). Let Firefox serve as an example for the standards the desktop is coming to meet because all B.S. aside, it has to@ Don Marti is editor in chief of Linux Journal. OCTOBER 2005 ISSUE 138 EDITOR IN CHIEF Don Marti, ljeditor@ssc.com EXECUTIVE EDITOR Jill Franklin, jill@ssc.com SENIOR EDITOR Doc Searls, doc@ssc.com SENIOR EDITOR Heather Mead, heather@ssc.com ART DIRECTOR Garrick Antikajian, garrick@ssc.com TECHNICAL EDITOR Michael Baxter, mab@cruzio.com SENIOR COLUMNIST Reuven Lerner, reuven@lerner.co.il CHEF FRANCAIS Marcel Gagne, mggagne@salmar.com SECURITY EDITOR Mick Bauer, mick@visi.com CONTRIBUTING EDITORS David A. Bandel • Greg Kroah-Hartman • Ibrahim Haddad • Robert Love • Zack Brown • Dave Phillips • Marco Fioretti • Ludovic Marcotte • Paul Barry • Paul McKenney PROOFREADER Geri Gale VP OF SALES AND MARKETING Carlie Fairchild, carlie@ssc.com MARKETING MANAGER Rebecca Cassity, rebecca@ssc.com INTERNATIONAL MARKET ANALYST James Gray, jgray@ssc.com REGIONAL ADVERTISING SALES NORTHERN USA: Joseph Krack, +1 866-423-7722 (toll-free) EASTERN USA: Martin Seto, +1 905-947-8846 SOUTHERN USA: Laura Whiteman, + 1 206-782-7733 x119 INTERNATIONAL: Annie Tiemann, +1 866-965-6646 (toll-free) ADVERTISING INQUIRIES ads@ssc.com PUBLISHER Phil Hughes, phil@ssc.com ACCOUNTANT Candy Beauchamp, acct@ssc.com LINUX JOURNAL IS PUBLISHED BY, AND IS A REGISTERED TRADE NAME OF, SSC PUBLISHING, LTD. PO Box 55549, Seattle, WA 98155-0549 USA • linux@ssc.com EDITORIAL ADVISORY BOARD Daniel Frye, Director, IBM Linux Technology Center Jon "maddog" Hall, President, Linux International Lawrence Lessig, Professor of Law, Stanford University Ransom Love, Director of Strategic Relationships, Family and Church History Department, Church of Jesus Christ of Latter-day Saints Sam Ockman, CEO, Penguin Computing Bruce Perens Bdale Garbee, Linux CTO, HP Danese Cooper, Open Source Diva, Intel Corporation SUBSCRIPTIONS E-MAIL: subs@ssc.com • URL: www.linuxjournal.com PHONE: +1 206-297-7514 • FAX: +1 206-297-7515 TOLL-FREE: 1-888-66-LINUX • MAIL: PO Box 55549, Seattle, WA 98155-0549 USA • Please allow 4-6 weeks for processing address changes and orders • PRINTED IN USA USPS LINUX JOURNAL (ISSN 1075-3583) is published monthly by SSC Publishing, Ltd., 2825 NW Market Street #208, Seattle, WA 98107. Periodicals postage paid at Seattle, Washington and at additional mailing offices. Cover price is $5 US. Subscription rate is $25/year in the United States, $32 in Canada and Mexico, $62 elsewhere. POSTMASTER: Please send address changes to Linux Journal, PO Box 55549, Seattle, WA 98155-0549. Subscriptions start with the next issue. Back issues, if available, may be ordered from the Linux Journal Store: store.linuxjournal.com. LINUX is a registered trademark of Linus Torvalds. LINUX JOURNAL 41 OCTOBER 2005 WWW.LINUXJOURNAL.COM The Power of Choice Command the game with your next I/O move. Modularity. Scalability. Reliability. Cost-effectiveness. These represent the solid foundations that SBE delivers to OEMs for building innovative end solutions. Partnering with SBE for networking and communications I/O solutions allows you to take advantage of proven technology and field-tested products designed to optimize performance for your unique application needs. SBE offers a full spectrum of interface cards, ranging from It and T3 to Gigabit Ethernet and IPsec/SSL acceleration. These boards are available in multiple form factors, including PCI, PMC, and PTMC. Customers have the choice of buying these boards individually or bundling any of the PMC/PTMC modules with our intelligent core processing platforms to create a flexible, cost- efficient blade solution ideal for serving demanding telecom applications. Full Linux support is available on every board. ► Channelized T3 ► 24-port T1/E1 ► LAN/Ethernet ► Storage ► IPsec/SSL Encryption ► Blade platforms ► I/O and beyond... Linux 0 * tWw' flexibility on demand I 925-355-2000 I info@sbei.com I www.sbei.com Ultimate Linux Box Cooling? I was impressed by the desire to make a quiet PC—more people should complain to their OEMs/System integrators about this—it is the only way it will be fixed [“Ultimate Linux Box”, August 2005]. But I am not sure that removing 100% of the airflow is a positive thing for overall sys¬ tem performance or stability. I’m used to reading the annual Ultimate PC article in Maximum PC magazine each year, and they give a great deal of detail and many more pictures than the Ultimate Linux Box article [August 2005]. However, the big difference between the two articles is that your Ultimate Linux Box does much more customization. More on the ULB I have seen motherboards designed where the processor power supply components can exceed the design rating from the suppliers without airflow. Even if things don’t go bad enough to cause system stability issues, it can damage the processor by allowing the CPU voltage to go out of specification. Intel is so concerned about this—they are telling moth¬ erboard manufacturers to add circuits to the motherboard to monitor the temperature of the processor power supply and modulate the clock of the processor if things get too hot. See Section 9.4 in download.intel.com/ design/Pentium4/guides/30235604.pdf. Thanks—and looking forward to more sub¬ minute kernel compiles. Robin Whenever you experiment with any alternate cooling method, always measure and log temperatures. — Ed. Linux/BSD Confusion My child is almost two in the picture. The laptop is running KDE on LreeBSD. He quite plainly CALLED it “Linux”. My wife and I both looked at each other and at him, and he said it again. He has also said “Ethernet”. We are afraid, very afraid. Thanks for a (decade of a) great publication! kurtseel The big question is, if I want to replicate the Ultimate Linux Box, where would I get details on the custom-made power supply cooler modifications? I went to the Resources page, but many of the things I would need to access require that I’m a Linux Journal subscriber. I have sub¬ scribed to the Linux Journal in the past, but for the last year, I simply run out and pur¬ chase Linux Journal. Dean Articles from that issue will he openly avail¬ able soon. We’ll look for more info on the power supply mods. — Ed. Ten Years of Progress The shock has worn off from seeing my name in Linux Journal , and I’m able to write again. [See “Ten Years Ago in Linux Journal ”, July 2005, page 14.] Has it been ten years since “Novice to Novice” appeared in Linux Journal ? It must be. I stopped writing the column after my first child was born and—bless it!—if it’s not his tenth birthday already. And how much has Linux changed (and stayed the same) in ten years. Though I haven’t quite made the switch away from Microsoft, I did recently install Ledora Core. In ten years, installation has vastly improved since when I used version 0.99 of Slackware. The Ledora installation was rela¬ tively fast and idiot-proof. Everything worked except the modem, and although I’m finally getting DSL installed, I wanted modem access as backup. Turns out I have a PCTel modem, which seems unsupported by the 2.6+ kernel and by the drivers currently out there in Webland. (Hmmm, could be another “Novice to Novice” here.) But what’s blown me away about Linux are the live CDs. Knoppix and the variations are fantastic not only for emergencies but also just to learn *nix, which is why I got involved with Linux originally. Ah well! Thanks for remembering me after all these years. Yes, I still have the 0.99 Slackware CD with the grinning Bob. It just seems right to keep it. Dean Become Boring and Pigeonholed, Please Hi. I’ve been a subscriber to LJ for several years, and I’ve never figured out your niche. Servers, or desktops? Sysadmins, application programmers, system develop¬ ers or home users? Your intention seems to be: appeal to everyone. Unfortunately for me, you cover so many different topics that there is very little in each issue for me. If you can’t figure out your niche, I’ll let my subscription expire in January. jh When different areas of information technol¬ ogy can stop learning from each other, we’ll pick a “niche”. — Ed. Ergonomic Comments on Ultimate Linux Box The case on the cover for your Ultimate Linux Box is gorgeous [August 2005]. Beautiful. Amazing. It also blows. It’s a terrible design. Ghastly. Ideal for a computer show, but awful to use. There’s no leg room on the box. Lor many folks, that would mean sitting obliquely or too far from the keyboard—both would cause strain. 610CTOBER 2005 WWW.LINUXJOURNAL.COM Reduce Your Deployment and Support Costs MBX is the leader for your server and appliance manufacturing needs Supermicro 5013G-MB • Intel® Pentium 4 Processor® at 3.0EGHz • 1U Rackmount Chassis • 512MB PC3200 DDR • Maxtor 80GB Serial ATA Hard Drive • Dual Onboard Gigabit NIC’s Includes CDROM, Floppy and Video Lifetime toll free tech support 3 Year Warranty $959 or lease for $33/mo. Or Promote Your Brand • Same Configuration as Above • Custom Branded With Your Logo • Worldwide Deployment and Support • Custom Branded Packaging Available • Configurations in 2U and 4U Available • Custom OS and Software Install • Custom Chassis Color Available • No Minimum Quantity Required $999 or lease for $38/mo. MBX is the leader in custom appliances. Many premier application developers have chosen MBX as their manufacturing partner because of our experience, flexibility and accessibility. Visit our website or better yet, give us a call. Our phones are personally answered by experts ready to serve you. MBX" MOTHERBOARDEXPRESS www.mbx.com 1.800.688.2347 Intel, Intel Inside, Pentium and Xeon are trademarks and registered trademark of Intel Corporation or its subsidiaries in the United States and other countries. Lease calculated for 36 months, to approved business customers. Prices and specifications subject to change without notice. Setup fee may apply to certain branding options. Motherboard Express Company. 1101 Brown Street Wauconda, IL. 60084. Tactile response acts as a brake reducing the amount of impact on the end of the fingers (while, agreed, increasing the finger travel), so the keyboard used may actually increase hand stress for some. Its placement isn’t adjustable for height, and that can be catas¬ trophic for arm stress. On a much less important note, I’d point out that the case puts the cooling up very high indeed, into airspace often several degrees hotter than the rest of the room. Paul Pomerleau The coolant loops need to run well above the motherboard in order to get adequate con¬ vective cooling. For daily use, you can build a tall case without the monitor mounts or keyboard shelf. — Ed. Another Happy Reader Here’s a photo of my son Merit (about 26- months old) sitting on his trusty fire truck checking out my July 2005 issue of LJ. When he was done, he went back into the office and picked up an Advanced C Programming book! tim Pipe Tip “Text Manipulation with sed” by Larry Richardson had some useful hints [July 2005]. For instance, I wasn’t aware of the ! modification to the range field. But writing to a file at the same time you are reading it is decidedly dangerous. You are depending on the pipe buffer between cat and sed to hold the entire contents of the file. You are also banking on the assumption that cat will be started and allowed to fill its buffer before the file is written and, there¬ fore, truncated at zero length. A far better way to do this is in two steps: sed -e ' s/$/ mycomputer/' < \ /etc/exports > tempfile \ && mv tempfile /etc/exports The mv command is executed only if sed returns without problems. You don’t want to be overwriting important files with the wrong data! Allen Brown Her First Computer T-Shirt My daughter (2.2-years old) made the transi¬ tion. She was introduced to Potato Guy and Tux Paint on my Linux box. On this day, she learned everything she needed to know about the mouse operations. A very proud moment for me, she is on the road to becom¬ ing a geek just like her daddy. Elizabeth Su WOHID Certified (Wireless Optical Human Interface Device), T-shirt awarded shortly after. You can’t start them too young. This is only the beginning for them. I am just happy to be there. Photo of the Month: a Linux Father's Day I had a great Father’s Day and hope all the other open-source dads did too. Of course, the best gift was Paige, my going-to-be- seven-in-September daughter—and you will see she painted Tux on a rock for me this Father’s Day. She always has been a good drawer, and her favorite program is Tux Paint with Tux Racer a close second. James M. Susanka Photo of the month gets you a one-year subscription or extension. Send photos to ljeditor@ ssc.com. Jesse Apple LETTERS CONTINUED ON PAGE 94 8IOCTOBER 2005 WWW.LINUXJOURNAL.COM The Stmight MkPeople SINCE 1991 ABERDEEN YOU CAN BUY THESE NOW OR WAIT TILL DELL FREEZES OVER iBERDEEN STONEHAVEN A141 1U Dual Opteron™ 4 SATA/SCSI High performance dual server for top-of-the-line processing power with ultra-dense storage capacity. • Dual AMD Opteron™ Processors w/HyperTransport and 1MB Cache • AMD 8000 Series Chipset w/64-bit Support • Up to 16GB DDR-400 Reg. ECC Memory • Up to 4 x 400GB (1,6TB) Hot-Swap SATA or 4 x 300GB (1.2TB) Hot-Swap SCSI Drives • 400W AC Power Supply w/PFC • 5-Year Limited Warranty iBERDEEN STONEHAVEN A261 2U Dual Opteron™ 6 SATA/SCSI The highest performing 2U server available for the money. “Staggering ... Powerhouse Performance ... Highest Webbench numbers we've seen to date” - PC Magazine, December 27, 2004. • Dual AMD Opteron™ Processors w/HyperTransport and 1MB Cache • AMD 8000 Series Chipset w/64-bit Support • Up to 16GB DDR-400 Reg. ECC Memory • Up to 6 x 400GB (2.4TB) Hot-Swap SATA or 6 x 300GB (1.8TB) Hot-Swap SCSI Drives • 460W Hot-Swap Redundant Power Supply • 5-Year Limited Warranty 3U Dual Opteron™ 8 SATA/SCSI Gargantuan storage beast with a capacity of up to 3.2TB, room for a dual-height tape drive, at an incomparable cost/TB ratio. • Dual AMD Opteron™ Processors w/HyperTransport and 1MB Cache • AMD 8000 Series Chipset w/64-bit Support • Up to 16GB DDR-400 Reg. ECC Memory • Up to 8 x 400GB (3.2TB) Hot-Swap SATA or 8 x 300GB (2.4TB) Hot-Swap SCSI Drives • 760W Hot-Swap Redundant Power Supply • 5-Year Limited Warranty Starting at $ 1,895 Starting at $ 2,875 Starting at $ 2,975 ABERDEEN STONEHAVEN A124 1U Quad Opteron™ HPC 64-bit HPC environment workhorse server/cluster node. Superior cooling with plenty of power to handle any project. • Cuad AMD Opteron™ 800 Series Processors • AMD 8000 Series Chipset w/64-bit Support • Up to 32GB DDR-400 Reg. ECC Memory • Up to 2 x 300GB (600GB) SCSI Hard Drives • 500W Power Supply • Ultra Cool with Superb Air Flow • 5-Year Limited Warranty 2U Quad Opteron™ 3 SATA/SCSI Robust 64-bit server ideal for the HPC environment as a high performance server. Able to provide all the power and I/O for large databases and memory intensive projects. • Cuad AMD Opteron™ Processors w/HyperTransport and 1MB Cache • AMD 8000 Series Chipset w/64-bit Support • Up to 32GB DDR-400 Reg. ECC Memory • Up to 3 x 400GB (1,2TB) Hot-Swap SATA or 3 x 300GB (900GB) Hot-Swap SCSI Drives • 700W Power Supply • Ultra Cool with Superb Air Flow • 5-Year Limited Warranty 4U Quad Opteron™ 8 SATA/SCSI Best of both worlds, all-inclusive server with enterprise-class 64-bit HPC Quad power along with maximum storage capacity. • Quad AMD Opteron™ Processors w/HyperTransport and 1MB Cache • AMD 8000 Series Chipset w/64-bit Support • Up to 32GB DDR-400 Reg. ECC Memory • Up to 8 x 400GB (3.2TB) Hot-Swap SATA or 8 x 300GB (2.4TB) Hot-Swap SCSI Drives • 950W3+1 Hot Swap Redundant Power Supply • Ultra Cool with Superb Air Flow • 5-Year Limited Warranty Quads Starting at $ 8,265 Quads Starting at $ 8,125 Quads Starting at $ 9,625 Trademarks are of their respective owners. IjOlO 888-297-7409 www flhprdppnino pom/linux W W vvtCLL/V/X 1111LLA To go along with this month's theme of Personal Desktop, here are some articles from the Linux Journal Web site that will help you find your way through OpenOffice.org, try out some Linux audio software and rescue data from a hosed USB device: » Do you want to move to OpenOffice.org but aren't sure what to expect? Are you trying to convince friends and/or family members to give OOo a try, but they want to know about the learning curve? If so, Bruce Byfield's article "OOo Off the Wall: What New Users Need to Know About OpenOffice.org" (www.linuxjournal.com/article/ 8443) is suggested reading. Bruce sheds some light on OOo's "interface shortcomings" and "the limits of its on-line help", as well as the "logic of its inter¬ face design and the importance of styles and templates in an efficient work flow". » Audio for Linux has come a long way in the past couple of years, and Dave Phillips continues his tour of what's new for musicians and engineers, whether full-time or part-time. In recent months, he's introduced us to Freewheeling, "a powerful loop-based performance tool" (www.linuxjournal.com/ article/8445), as well as QSynth and QJackCtl, GUI front ends that "make Linux audio tasks easier and faster, letting you get straight to the music" (www.linuxjournal.com/ article/8354). » Finally, Collin Park shares his story of "Flow a Corrupted USB Drive Was Saved by GNU/Linux" (www.linuxjournal.com/article/ 8366), offering hope to those of us who have lost important data and will lose it again. What's New in Kernel Development After a long and difficult life, DevFS is final¬ ly being removed from the Linux kernel. Created by Richard Gooch, DevFS has been around for years, and it represented a serious attempt to cure the runaway /dev directory. Developing DevFS was an uphill battle against many detractors, but Richard did suc¬ ceed in creating a very useful tool. In the end, however, critics of DevFS won out, citing “unfixable races” and other problems, and Richard vanished from kernel development completely. Greg Kroah-Hartman and others then developed udev as a replacement for DevFS. Some lingering sense of the 2.6 kernel as a stable tree has made this decision slightly controversial even now, but almost certainly it’s not enough to influence the outcome. Farewell DevFS—it was a valiant effort. Recently, various folks have reported com¬ pilation problems when trying to compile the 2.4 kernel with GCC version 4, and some developers have posted patches to address these issues; however, Marcelo Tosatti has stated that it is simply too late in the day for these sorts of patches to make it into the 2.4 tree. Unlike 2.6 development, the maintainers of 2.4, 2.2 and 2.0 have not decided to follow suit and abandon the idea that their trees must aim for stability. Marcelo has been trying to rein in 2.4 develop¬ ment ever since the first 2.6 kernel came out, but he has still allowed large IDE changes, new hardware support and other patches whose inva¬ siveness would typically fly in the face of a push for stability. And with 2.6 development showing no sign of slowing down, Marcelo has been under constant pressure to incorporate new features into 2.4 to be available to folks who needed 2.4’s stability. With the advent of the w.x.y.z tree, however, some of this pressure has undoubtedly flagged, and Marcelo has been able to tighten up the restrictions on what can and cannot get into 2.4 at this late date. The git versioning system continues to grow and strengthen. Andrew Morton’s -mm tree will be available as a git repository, although Andrew himself has no plans to use any versioning tool for actual development. The ALSA Project has migrated development to git, as has iibata. Marcelo Tosatti’s 2.4 tree also will use git for ongoing development. Linus Torvaids is still very strongly involved with the project, and although mailing-list traffic has tapered off somewhat from its fran¬ tic early weeks, much of this is explained by the fact that folks now understand the basics of the tool, and the fundamental concepts no longer need to be explained to newcomers. In the midst of all the version-control upheaval, it’s hard to know for certain if the new w.x.y.z stable kernels are working out. But several kernel folks, including Jeff Garzik and Alan Cox, feel that this tree successfully pro¬ vides a stable kernel to supplement the 2.6 tree’s ongoing large-scale development. Greg Kroah-Hartman and Chris Wright, the primary maintainers of the w.x.y.z tree, do seem to be doing a rigorous job, not only collecting and applying patches, but adhering to Linus Torvaids’ strict guidelines on what patches may be applied, and how and when they may be accepted. A number of aspects make this project less appealing than doing real development work, but Chris and Greg seem to be bearing up nicely, and the rest of us are the beneficiaries. Martin J. Bligh has put together a set of automatic testing scripts that compile and boot all official kernel releases (including the w.x.y.z kernels) and several prominent branches like the -mm tree, within 15 minutes of their release. If a kernel boots successfully, Martin’s scripts hit it with a variety of benchmarks. Compilation and boot results are recorded, benchmark results are graphed and everything is made available as a set of ongoing kernel.org Web pages. This is the sort of project that will not solve all bugs, but it will identify many trivial bugs, track per¬ formance problems across multiple kernel releases and may identify hard-to-fmd bugs that regular users would not normally see. The relatively recent introduction of Signed-Off-By tags in kernel patch submis¬ sions has made a huge difference in providing a trail of authorship, so that if anything like the SCO lawsuit occurs again, it will be easy to prove who wrote any disputed source code. This was, in fact, Linus Torvaids’ stated pur¬ pose in introducing the Signed-Off-By header. When first introduced, the idea was quite amor¬ phous, with few details settled. Since then, var¬ ious wrinkles have been introduced to improve its usefulness. One of the most recent of these is the addition of a From header as the first line of the body of patch e-mails. This header iden¬ tifies the tme author of a given patch. Before this wrinkle, the tme author was assumed to be the person with the bottom-most Signed-Off-By header. This, however, became confusing and was not always adhered to. The From header is intended to leave no doubt as to the original authorship of a given patch. — ZACK BROWN 1 OB OCTOBER 2005 WWW.LINUXJOURNAL.COM Cyc/ade s AlterPath™OnSite makes branch c-f-fice administ ration child's p/&-y Systems Administrator l'iyPa.4h **»*S‘ f ( in^ernei r""'\ CDMA Office AlterPath OnSite > The Next-Generation IT Infrastructure Cyclades AlterPath™ OnSite is the most comprehensive remote site and branch office administration appliance available. This small, inexpensive solution for controlling network equipment, servers and other IT infrastructure devices can • Access, diagnose and restore remote IT devices quickly • Download software to multiple devices automatically and simultaneously • Configure user information, system settings and operating parameters • Send alerts of intrusions, equipment failures and alarms The AlterPath OnSite combines the functionality of both serial console and KVM over IP, allowing IT administrators to manage multiple servers and network devices through a single appliance. Cyclades brings it all together making remote site and branch office administration seem like child's play. Over 85% of Fortune 100 choose Cyclades. www.cyclades.com/ljb 1.888.cyclades ■ sales@cyclades.com cyclades C MOS Cptlndfli Corporamm AM nyhn raswwd AO crihnr trodaraarts and produfl images or* property o! far rwpedw* nmnn fodwt udnrmataa uAjkj id Nigel McFarlane With the sudden death of Nigel McFarlane, the Web Development and Open Source Software communities, both in Australia and around the world, have lost one of their most well-known authors, consultants and pundits. Although in many ways a very pri¬ vate person, Nigel had a professional and personal network that spanned the globe and included such on-line luminaries as the lead engineer for the open-source browser Firefox Ben Goodger, and countless others in the Open Source, Web Development and Linux communities. Since his passing, many community sites, in a number of languages, have expressed their sor¬ row, a testament to Nigel's influence. A real Melbourne boy, describing the city proudly as "the World's most liveable", Nigel had science degrees from both the University of Melbourne and LaTrobe University. Even when speaking in Sydney, he was always keen to get home as soon as possible, where he would bushwalk and ramble, swim and surf. Nigel forged a global reputation from his beloved Melbourne, in a way impossible until the 1990s. Many oth¬ ers have and will follow his lead, but as with much of what he did, here, too, Nigel was a pioneer. Since 1997, Nigel had become well known and respect¬ ed in the Web Development and more recently Open Source Technology com¬ munities through the publication of several successful books on JavaScript, Mozilla and most recently, the increas¬ ingly popular free open-source brows¬ er Firefox. Two earlier books on JavaScript, Instant JavaScript in 1997 and the co¬ authored Professional JavaScript in 2001, are still considered by many to be among the best books on the subject. More recently, the benchmark Rapid Application Development with Mozilla, and Firefox Hacks carved out a place in the increasingly important Open Source community. Nigel's writing extended to the columns "Searching for Substance" for InformIT, and articles for such publica¬ tions as Linux Journal, DevX, Builder.com, CNet, The Age and the Sydney Morning Herald. Nigel was an entertaining speak¬ er as well as a writer. I particularly recall chairing a conference session that Nigel presented late last year. Often confer¬ ence-goers are anxious to get early places in the meal queue, but although we had gone overtime for lunch, Nigel captivated the room. When offered the opportunity to break, the entire room turned it down, glued as they were to Nigel's presentation. Generous with his time, energies and knowledge, Nigel contributed to mailing lists, newsgroups and forums, as well as speaking to audiences large and small at conferences and for user groups. His reach went far beyond HLA Adventure - Version 3.ID : hp: 8407 g: 242 exp: 281 http://nembers.tr ipod.con/' v panks/hlaadv.htnl talk to dragon The dragon bellows,'I an the last of ny kind. And here you are, a dirty human, cone to slay me?! I breathe fire upon trees, rustling the bad kani out of the poison forests and valleys of this forgotten world. I keep ny peace inside these caves, only to be disturbed by the wretched hunans which pollute this landscape! I was once happy in ny silence here, yet you cone here to nake ne niserable! I won't allow it. Knave! Prepare to feel ny wrath!!!' The dragon noves about angrily, stomping the floor! examine dragon A large dragon with tough scales and powerful claws. Vou are carrying: lantern rope key sword longsword broadsword HLA Adventure members.tripod.com/~panks/hlaadv.html When Zork appeared on the scene in the late 1970s, computer enthusiasts from around the world were instantly hooked on the interactive fiction genre known fondly as the Text Adventure game. HLA Adventure is the latest in a long line of public domain and free software text adventures being released by people all over the world. It combines elements from MUDs, Advanced Dungeons & Dragons and J.R.R. Tolkien's famous The Lord of the Rings. Using verbs and nouns to communi¬ cate with the game world, the player moves about HLA Adventure with but a simple goal in mind: slay the menacing dragon at the end of a large expanse of caves. While solving this main quest, the player is also presented with nine other unique quests, which allow the player to find items and equip weapons, armor and a brightly lit lantern. Even a magical flute plays a role—useful in putting magical beasts to sleep. Players will encounter hellhounds, werewolves, vampires, hobbits, ghosts, barbarians and demigorgons. Talk to creatures in the game with the TALK TO command. Once you have acquired the necessary armament and passed the requisite number of quests, you can then enter into the cave and slay the dragon for good. Despite some bugs in the game, HLA Adventure is a solid, robust open- 121 OCTOBER 2005 WWW.LINUXJOURNAL.COM Plug the Levanta Intrepid™ into your network and perform the most important Linux management tasks in a fraction of the time you spend now. And gain power and flexibility that you've never had before: Levanta Intrepid™ ™ Fast & Portable: Provision servers or workstations practically anywhere, anytime - in minutes. Swap them around, mix it up. ™ Flexible: Supports commodity hardware, blades, virtual machines, and even mainframes. ™ Out of the Box: Includes pre-defined templates for servers, workstations, & software stacks. Or create your own. ™ Total Control: Track any file changes, by any means, at any time. And undo them at will. ™ Disaster Recovery: Bring dead machines quickly back to life, even if they're unbootable. 30-Day Money-Back Guarantee Order online by 10/30/05 Get $500 Off Enter PROMO CODE: LJ1005 Based upon technology that's already been proven in Fortune 500 enterprise data centers. Now available in a box, priced for smaller environments. Just plug it in and go. © 2005 Levanta, Inc. All rights reserved. Levanta and the Levanta logo are registered marks of Levanta, Inc. LEVANTA* www.levanta.com 1 . 877. LEVANTA Ruby on Rails Explore a Web development framework that comes with its own Web server, magically keeps track of details for you and integrates new code without restarting, by reuven m. lerner R uby, an interpreted programming language that looks and feels like a cross between Smalltalk and Perl, has been around for about ten years. Ruby has been gaining in popularity over the last few years, partly because of the release of English-language books and docu¬ mentation. In addition, programmers have become more inter¬ ested in finding an alternative to Perl and Python for their gen¬ eral-purpose programming needs. Ruby’s popularity might have continued to grow slowly were it not for Ruby on Rails, a Web development framework that has become the focus of enormous attention. Everyone in the Web development world seems to be talking about Rails; magazine articles, blog postings, conference tracks and even some new books all are dedicated to Rails. Rails is supposed to be elegant, easy to use and easy to modify. Even developers with no previous Ruby experience are switching to Rails. Does Rails live up to the hype surrounding it? To a large degree, I believe the answer is “yes”—it has a relatively shal¬ low learning curve, it connects easily and quickly to relational databases and it makes the creation of many small- and medi¬ um-sized sites faster and easier than I would have expected. But, of course, no framework is perfect, particularly one that was released publicly only one year ago. It remains to be seen whether Rails can hold up against more-established technologies on several different fronts. This month, we begin to look at several aspects of Ruby on Rails, so you can decide for yourself if my assessment is accurate. We begin by installing and configuring a basic Rails application. Over the next few installments of At the Forge, we will extend our application in several different ways, considering the ways in which Rails allows us to create and modify our applications. Installing Rails The first step in creating a Rails application is to install Ruby and then Rails itself. Most modern Linux distributions come with Ruby, although only the latest released version as of this writing (1.8.2) works with the most recent version of Rails (0.12.1). New versions of Rails have been coming out fre¬ quently, which means that one or both of these versions might have changed by the time you read this. Assuming you have installed Ruby, you next need to install Gems. It provides access to the Ruby Gems library, which is something of a cross between SourceForge and Perl’s CPAN (see the on-line Resources). Download and unpack the most recent .tar.gz file: tar -zxvf rubygems-0.8.10.tar.gz Enter the directory as the root user and type: ruby setup.rb all This installs the entire Gems package. Among other things, this installs the gem program in /usr/bin. You then can install Rails, which is distributed via Gems, with the following command: gem install --remote rails As with such systems as CPAN and Debian’s apt, the gem program is smart enough to identify and download any depen¬ dencies it might encounter. By default, you need to answer “y” explicitly when asked if you are interested in installing any dependencies. Because Rails depends on a number of other packages, you should be sure to answer “y” when prompted. When you are returned to the shell prompt, you can assume that Rails has been installed. However, this is not quite enough. If you are interested in working with a relational database, you also need to install a database interface library. Because I work with PostgreSQL, I installed the pure Ruby client, called postgres-pr: gem install --remote postgres-pr Somewhat confusingly, there also is a set of PostgreSQL client libraries (called postgresql) that can be used with Ruby. However, it seems as though most Rails developers are work¬ ing with the postgres-pr library, at least for now. Creating an Application Once Rails is installed, we can create a simple “Hello, world” program. To do this, we use the rails command, which is installed in /usr/bin/ by default. Because our example applica¬ tion is a Weblog, we call the application blog. For reference, the name of the application doesn’t have to be linked to the name of the URL under which it will appear. Type: rails blog Running this produces a fair amount of output, listing the files that have been created on our filesystem. When we give only a single name, blog, the application is created inside of a directory with that name. We can keep all of our applications inside of a single container directory, such as -/Rails, with: mkdir -/Rails rails -/Rails/blog If we look inside the newly created application directory, we see a number of directories and files. The script directory contains administrative programs, written in Ruby, of course. The public directory contains static HTML files, as well as images, stylesheets, JavaScript code and templates that you 1 41 OCTOBER 2005 WWW.LINUXJOURNAL.COM EmperorLinux ...where Linux & laptops converge You choose your laptop ••• from a wide selection of top tier laptops manufactured by IBM/Lenovo, Dell, Sharp, and Sony. They come in all sizes from two pound ultra-portables to eight pound desktop replacements; get exactly as much Linux laptop as you need. Need help deciding? Our experts will help you select a Linux laptop to meet your needs. The Meteor: 3lb Linux The SilverComet: 4 lb Linux • Sharp Actius MM20/MP30 • 10.4" XGA screen • 1.6 GHz Transmeta Efficeon • 20-40 GB hard drive • 512-1024 MB RAM • CDRW/DVD (MP30) • 802.1 lb/g wireless • ACPI hibernate • 1" thin • Ask about the 3D Molecule • Sony VAIO S380 • 13.3" WXGA+ screen •X@l 280x800 • 1.6-2.13 GHz Pentium-M • 40-100 GB hard drive • 256-1024 MB RAM •CDRW/DVD orDVD-RW • 802.1 lb/g wireless • ACPI hibernate • Ask about the 17" Gazelle You choose your distribution ••• from among the most popular Linux distributions available. We'll install the distribution you select, then we'll install our custom, laptop-specific kernel and configure your distribution for full hardware support, including: X at the native resolution, wireless ethernet, power management, 3-D graphics, optical drives, and more. The Toucan: 5 lb Linux The Rhino: 7 lb Linux • IBM/Lenovo ThinkPad T series • U.l“ SXGA+/15.0" UXGA • X@1400xl050/X@l 600x1200 • ATI FireGL graphics • 1.6-2.13 GHz Pentium-M 7xx • 40-80 GB hard drive • 512-2048 MB RAM • CDRW/DVD or DVD-RW • APM suspend/hibernate • Ask about the 3 lb Raven X41 • Dell Latitude D810/M70 • 15.4" WUXGA screen • X@1920xl200 • NVidia Quadra or ATI Radeon • 1.73-2.26 GHz Pentium-M 7xx • 30-100 GB hard drive (7200 rpm) • 256-2048 MB RAM •CDRW/DVD or DVD±RW • 802.1 la/b/g wireless, GigE • Ask about the tiny Koala XI To: ttuppurtCeinjKrorlinLOc.coi Frau: ciiXtoBenJhaBebase.net Subject: Configuration of Just bought a new uirelj to learn how to configj my PC and ay laptop anywhere in ay houM And I still nent it to Let EmperorLinux do the rest Since 1999, EmperorLinux has provided pre-installed Linux laptop solutions to universities, corporations, and individual Linux enthusiasts. We specialize in the installation and configuration of the Linux operating system on a wide range of the finest laptop and notebook computers made by IBM/Lenovo, Dell, Sharp, and Sony. We offer a range of the latest Linux distributions, as well as Windows dual boot options. All systems come with one year of Linux technical support by both phone and email, and full manufacturers' warranties apply. www.EmperorLinux.com 1-888-651-6686 Model prices, specifications, and availability may vary. All trademarks are the property of their respective owners. YOUR HIGH PERFORMANCE COMPUTING SOLUTION HAS ARRIVED. VXRACK™ with the Intel® Xeon™ processor helps you simplify computing operations, accelerate performance and accomplish more in less time. Choose i v I ) convene one of the 3 convenient rack sizes VXR-128 Rack accomodating up to 128 VXBJades/256 Processors 40TB of aggregated Storage 1.5TB of Global Memory Power Distribution Included Patented Architecture Advanced Cooling System Integrated InfiniBand Cable Mgnt. $ 2,190.00* VXR-96 Rack accomodating up to 96 VXBiades/192 Processors 36TB of aggregated Storage 1.1 STB of Global Memory Power Distribution Included Patented Architecture Advanced Cooling System Integrated InfiniBand Cable Mgnt. $ 1,750.00* VXR-72 Rack accomodating up to 72 VXBIades/144 Processors 27TB of aggregated Storage 864GB of Global Memory Power Distribution Included Patented Architecture Advanced Cooling System Integrated InfiniBand Cable Mgnt $ 1,590.00* TECHNOLOGIES For more Information call or visit us at inlei, Mai logo, Intel inside, Intel insoe logo, Intel Centura),, Intel Genuine logo, Celeron, Intel ?tecn, ntal SpeedStep, Itanum, Pentium, enc VXB-7221B Intel SE7221B Motherboard 800MHz Front Side Bus Intel® Pentium® 4 3.2GHz 1GB DDR2 400 Memory Single 40GB 7200RPM ATA Drive One PCI/Express Slot Available DuaM 0/100/1000 Intel Lan Port 35QW Power Supply $ 985.00 VXB-7501W Intel SE7501W Motherboard 533MHz Front Side Bus 2 x Intel® Xeon™ 3.06GHz 2GB DDR 333 ECO Reg.Mem Single 40GB 7200RPM ATA Drive One PCI/X Slot Available Dual 10/100/1000 Intel Lan Port 3SOW Power Supply VXB-7520J Intel SE7520J Motherboard 800MHz Front Side Bus 2 x Intel® EM64T Xeon™ 3.2GHz 2GB DDR2 400 ECO Reg.Mem Single 40GB 7200RPM ATA Drive One PCI/Express Slot Available Dual 10/100/1000 Intel Lan Port 500W Power Supply $ 2 , 950.00 Choose one or more type of VXBIade $ 2 , 355.00 Add, Mutiply,That’s it. Easy as 1, 2, 3... For example you choose the following: One VXR-96 with 48 Dual Intel® EM64T Xeon™ and 40 Single Intel® Pentium®4. You take 1 (VXR-96) + 48 (VXB-7520J) + 40 (VXB-7221 B)...That’s it THE FUTURE OF CLUSTER TECHNOLOGY CIARA TECHNOLOGIES...A GLOBAL SOLUTION PROVIDER. Ciara Technologies is a world-class computer systems manufacturer. Ciara designs, develops, manufactures, markets, services, and supports a variety of computer systems Including graphic workstations, rackmount and tower servers, networked storage and the newly acclaimed VXRACK™ Cluster Technology. The company's state of the art supercomputer cluster is based on the Intel IA32 and IA64 architectures and utilizes Linux operating systems. We are proud to be recognized by Intel as an “Intel Premier Provider". Choosing Ciara is choosing a single point of contact for all your rr requirements. All our products are built under the ISO 9001 standards and regulations. The growth of Ciara enabled the company to move its 300+ employees,in February 2003, to an ultra-modern plant of 576,000 sqft.. Ciara now has the capability of producing more than 500,000 units per year. 866-7VX-RACK (866-789-7225) WWW.VXRACK.COM I Pontfum III Xeon are trademarks or registered trademarks c< Intel Corporation or its stieldlanes in the Untied States and other countries Ja-fbr typo^Hf^iy grrarE, pfiotographes srrors, prfcirg srrara. AH pricteg fl LK cbtla. 3-^prig aid BpplicablB taxes are not relucted. may use in your application. The directory you are mostly likely to work with is app, which contains the application itself. The app directory contains subdirectories named models, views and con¬ trollers. This design reflects the fact that Rails uses the MVC (model/view/controller) style widely used in many modern desktop and Web applications. In an MVC architecture, we divide our work into three parts—the controller, which acts like a switchboard, invok¬ ing the appropriate model and view; the model, which con¬ tains the data and some of the logic; and the view, which displays information to the user. If you have ever built a database-backed site with PHP and Smarty templates or with Zope and its Page Templates or even with Java and JavaServer Pages (JSPs), you already are familiar with at least some of these ideas. Rails simply makes them more explicit with its prenamed directory structure. Although it can’t do much, we now can start our empty Rails application with: cd ~/Rails/blog ruby script/server This starts the WEBrick HTTP server on port 3000. To access this fairly empty Rails site, we point our browsers to an appropriate IP address or hostname. In my particular case, I started Rails on my test server, whose IP address is 192.168.2.3. I thus point my Web browser to http://l92.168.2.3:3000/. And sure enough, there I see a “Welcome on board” message, indicating I have set up Rails correctly. Customizing the Behavior Now that we know how to see the default message, let’s move toward a “Hello, world” program. In Rails, there are two basic ways to do this. We can create a controller that returns HTML to the user’s browser, or we can create a view that does the same. Let’s try it both ways, so that we can better understand the relationship between controllers and views. If all we want to do is include a simple, static HTML docu¬ ment, we can do so in the public directory. In other words, the file blog/public/foo.html is available under WEBrick—started by executing blog/script/server—at the URL/foo.html. Of course, we’re interested in doing something a bit more interesting than serving static HTML documents. We can do that by creating a controller class and then defining a method within that class to produce a basic “Hello, world” message. Admittedly, this is a violation of the MVC separation that Rails tries to enforce, but as a simple indication of how things work, it seems like a good next step. To generate a new controller class named MyBlog, we enter the blog directory and type: ruby script/generate controller MyBlog Each time we want to create a new component in our Rails application, we call upon script/generate to create a skele¬ ton. We then can modify that skeleton to suit our specific needs. As always, Rails tells us what it is doing as it creates the files and directories: exists app/controllers/ exists app/helpers/ create app/views/my_blog exists test/functional/ create app/controllers/my_blog_controller.rb create test/functional/my_blog_controller_test.rb create app/helpers/my_blog_helper.rb Also notice how our controller class name, MyBlog, has been turned into various Ruby filenames, such as app/views/my_blog and app/helpers/my_blog_helper.rb. Create several more controller classes, and you should see that all of the names, like LooBar, are implemented in files with names like foo_bar. This is part of the Rails convention of keeping names consistent. This consistency makes it possible for Rails to take care of many items almost magically, especially—as we will see next month—when it comes to databases. The controller that interests us is my_blog_controller.rb. If you open it up in an editor, you should see that it consists of two lines: class MyBlogController < ApplicationController end In other words, this file defines MyBlogController, a class that inherits from the ApplicationController class. As it stands, the definition is empty, which means that we have neither overridden any methods from the parent class nor written any new methods of our own. Let’s change that, using the built-in render_text method to produce some output: class MyBlogController < ApplicationController def hello_world render_text "Hello, world" end end After adding this method definition, we can see its results by going to http://192.168.2.3:3000/MyBlog/hello_world. Notice how the URL has changed: static items in the public directory, such as our file foo.html, sit just beneath the root URL, /. By contrast, our method hello_world is accessed by name, under the controller class that we generated. Also notice that we did not need to restart Rails in order to create and test this definition. As soon as a method is created or changed, it immediately is noticed and integrated into the current Rails system. If we define an index method for our controller class, we can indicate what should be displayed by default: class MyBlogController < ApplicationController def hello_world render_text "Hello, world" end def index render text "I am the index!" 1 81 OCTOBER 2005 WWW.LINUXJOURNAL.COM Let’s be ^ treme Introducing Appro Xtreme Servers & Workstations with 8 DIMM Sockets per CPU 2-way or 4-way, Single or Dual-Core AMD Opteron™ processors Largest memory capacity - 8 DIMM Sockets per CPU, up to 128GB PCI-Express technology to increase I/O bandwidth and reduce system latency Outstanding Remote Management - IPMI 2.0 compliant Cable-free design, ready to run, simple to install, service and maintain Support for Windows® or Linux OS Ideal for memory-intensive and l/O-intensive applications 1U/2UI3U Servers and Workstation AMD Opteron™ Processors - AMD64 dual-core technology reduces memory latency and increases data throughput - Dual-core processors with Direct Connect Architecture deliver the best performance per watt in the industry with little or no increase in power consumption or heat dissipation. HPC Cluster Solutions Appro delivers high-performance computing solutions to help you maximize productivity for a solid ROI. On-site maintenance and installation services are also available. For more information, please visit www.appro.com or call Appro Sales at 800.927.5464, 408.941.8100. end end Of course, it’s not that exciting to be able to produce static text. Therefore, let’s modify our index method such that it uses Ruby’s built-in Time object to show the current date and time: def index render_text "The time is now " + Time.now.to_s + "\n" end And voila! As soon as we save this modification to disk, the default URL (http://192.168.2.3:3000/MyBlog/, on my computer) displays the time and date at which the request was made, as opposed to a never-changing “Hello, world” message. Let’s conclude this introduction to Rails by separating the controller from its view once again. In other words, we want to have the controller handle the logic and the view handle the HTML output. Once again, Rails allows us to do this easily by taking advantage of its naming conventions. For example, let us modify our index method again, this time removing its entire body: def index end This might seem strange at first glance. It tells Rails that the MyBlog controller class has an index method. But it does¬ n’t generate any output. If you attempt to retrieve the same URL as before, Rails produces an error message indicating that it could not find an appropriate template. Because the template is a view, we can define it inside of the blog/app/views directory of our application. And because we are defining the index view for the MyBlog class, we modi¬ fy the index.rhtml file in the my_blog subdirectory of views. Notice how Rails turns ThisName into this_name when it comes to directories. Doing so saves users from having to think about capitalization in URLs, while staying consistent with traditional Ruby class naming conventions. .rhtml files are a Ruby version of the same kind of template that you might have seen before. It acts similarly to ASP and JSP syntax, with <% %> blocks containing code and <%= %> blocks containing expressions that should be interpolated into the template. However, nothing stops us from creating an .rhtml template that actually is static: Hello, again!

Hello, again!

Consider what happens now if you attempt to load MyBlog in your browser. The controller class MyBlog is handed the request. Because no method was named explicitly, the index method is invoked. And because index doesn’t produce any output, the my_blog/index.rhtml template is returned to the user. Finally, let’s take advantage of our template’s dynamic properties to set a value in the controller and pass that along to the template. We modify our index method to read: def index @now = Time.now.to_s end Notice how we have used an @ character at the beginning of the variable @now. I found this to be a little confusing at first, as @ normally is used as a prefix for instance variables in Ruby. But it becomes fairly natural and logical after a little time. Finally, we modify our template such that it incorporates the string value contained in @now: Hello, world!

Hello, world!

11 is now <%= @now %>.

Once again, you can retrieve the page even without restart¬ ing Ruby. You should see the date and time as kept on the serv¬ er, updated each time you refresh the page. Conclusion Ruby on Rails is, without a doubt, one of the most talked-about Web technologies to emerge in the past few years. This month, we saw how straightforward it is to create a new Rails applica¬ tion, to create a controller and a view and to integrate them using a combination of naming conventions and relatively stan¬ dard template syntax. However, we did not discuss views, par¬ ticularly those associated with a relational database. Next month, we will do exactly that, connecting Rails to the PostgreSQL database. I believe doing so will begin to show why people are so excited about Rails and why it might be a good tool for many Web developers to learn. Resources for this article: www.linuxjournal.com/article/ 8457.0 Reuven M. Lerner, a longtime Web/database con¬ sultant and developer, now is a graduate student in the Learning Sciences program at Northwestern University. His Weblog is at altneuland.lerner.co.il, and you can reach him at reuven@lerner.co.il. 201 OCTOBER 2005 WWW.LINUXJOURNAL.COM MORE SPACE. LESS MONEY. Unlimited Affordable Network Storage Everybody needs more space. And they need to spend less money. What if you can both have more space and spend less money? What if you could put IV 2 terabytes in only 3 rack units? What if that IV 2 terabytes cost less than $10,000? In¬ cluding the SATA disk drives. Imagine if you could glue it all together with a RAID appliance into one system. What if you could add as much storage as you wanted, one shelf at a time, and never have to 'fork-lift’ anything? Coraid’s new SATA EtherDrive Storage allows you to do just that. Using industry standard SATA disk drives, EtherDrive Storage connects disks directly to your Ethernet network. Each disk appears as a local drive to any Linux, FreeBSD or Solaris system using our open ATA-over-Eth- ernet (AoE) protocol. Since the disks just appear as local drives you already know how to use them. The EtherDrive® SATA Storage Shelf is a 3U rack- mount network appliance that contains 15 SATA drive slots. Its triple redundant power supply protects you from your most likely failure. Its dual Gb Ethernet interfaces allow your data to go fast; 200MB per second. And at a very af¬ fordable price. List price for the EtherDrive Storage Shelf, without disks, is only $3,995. Our companion product, the RAIDBlade RAID controller, allows a virtually unlimited number of Storage Shelves to be combined into a set of logical AoE storage de¬ vices. Now you can have unlimited storage at a very af¬ fordable price. For complete information, visit our website at www.coraid.com, or call, toll-free, 1-877-548-7200. And we’ll show how we’ve made network storage so affordable, you can have all the space you want. www.coraid.com info@coraid.com 1.706.548.7200 CO RAID Network Programming in the Kernel Take a tour of the kernel's networking functionality by writing a network client that runs in kernel space. BY PRADEEP PAOALA AND RAVI PARIMI 11 Linux distributions provide a wide range of network appli¬ cations—from daemons that provide a variety of services such as WWW, mail and SSH to client programs that access one or more of these services. These programs are written in user mode and use the system calls pro¬ vided by the kernel to perform various operations like network read and write. Although this is the traditional method of writing programs, there is another inter¬ esting way to develop these applications by implementing them in the kernel. The TUX Web server is a good example of an application that runs inside the kernel and serves static content. In this article, we explain the basics of writing network applications within the kernel and their advantages and disadvantages. As an example, we explain the implementation of an in-kemel FTP client. Advantages and Disadvantages of In-Kernel Implementations Why would one want to implement applications within the kernel? Here are a few advantages: ■ When a user-space program makes a system call, there is some overhead associated in the user-space/kernel- space transition. By programming all functionality in the kernel, we can make gains in performance. ■ The data corresponding to any appli¬ cation that sends or receives packets is copied from user mode to kernel mode and vice versa. By implement¬ ing network applications within the kernel, it is possible to reduce such overhead and increase efficiency by not copying data to user mode. ■ In specific research and high-perfor¬ mance computing environments, there is a need for achieving data transfers at great speeds. Kernel applications are useful in such situations. On the other hand, in-kernel imple¬ mentations have certain disadvantages: ■ Security is a primary concern within the kernel, and a large class of user¬ mode applications are not suitable to be run directly in the kernel. Consequently, special care needs to be taken while designing in-kernel applications. For example, reading and writing to files within the kernel is usually a bad idea, but most appli¬ cations require some kind of file I/O. ■ Large applications cannot be imple¬ mented in the kernel due to memory constraints. Network Programming Basics Network programming is usually done with sockets. A socket serves as a com¬ munication end point between two pro¬ cesses. In this article, we describe net¬ work programming with TCP/IP sockets. Server programs create sockets, bind to well-known ports, listen and accept connections from clients. Servers are usu¬ ally designed to accept multiple connec¬ tions from clients—they either fork a new process to serve each client request (con¬ current servers) or completely serve one request before accepting more connec¬ tions (iterative servers). Client programs, on the other hand, create sockets to con¬ nect to servers and exchange information. FTP Client-Server Interaction Let’s take a quick look at how an FTP client and server are implemented in user mode. We discuss only active FTP in this article. The differences between active and passive FTP are not relevant to our discussion of network programming here. Socket Programming Basics Here is a brief explanation of the design of an FTP client and server. The server program creates a socket using the socket () system call. It then binds on a well-known port using bi nd () and waits for connections from clients using the 1 i sten () system call. The server then accepts incoming requests from clients using accept () and forks a new process (or thread) to serve each incom¬ ing client request. The client program creates a control socket using socket () and next calls connect () to establish a connection with the server. It then creates a separate socket for data transfer using socket () and binds to an unprivileged port client server Figure 1. The FTP protocol uses two sockets: one for control messages and one for data. Flere's how the first connection, used for commands, gets set up. 221 OCTOBER 2005 WWW.LINUXJOURNAL.COM The bolA7 craiL Easy, high-performance clustering. For years, many searched, but none could find it. Some said it didn't exist. But not the Penguin. Penguin Computing® made easy, high-performance clustering a quest. Now you can find Linux hardware and software solutions, configured to order, driven by Scyld's commercially supported, industry-leading Linux clustering software. For the turnkey clusters you need to run even your most important applica¬ tions, come to Penguin Computing. Penguin Computing's dedicated experts, who are 100% focused on Linux, are waiting to serve. O d) Powerful, easy clustering. It's the once and future thing. Love what you do (^) CL E o .o www.penguin □□□□ mmnn SCYLD registered trademark of Penguin Computing, Inc. Scyld, Scyld Software, Scyld Beowulf, and the Scyld Block Logo are trademarks of Scyld Software, Inc. Linux is a registered trademark of Linus Torvalds. Other names are for informational purposes only and may be trademarks of their respective owners. (>1024) using bi nd (). The client now 1 i sten () s on this port for data transfer from the server. The server now has enough knowledge to honor a data transfer request from the client. Finally, the client uses accept () to accept connections from the server to send and receive data. For sending and receiving data, the client and server use the wri te () and read() or sendmsgQ and recvmsgO system calls. The client issues close () on all open sockets to tear down its connection to the server. Figure 1 sums it up. FTP Commands Here is a list of a few FTP commands we used. Because our program provides only a basic implementation of the protocol, we discuss only the relevant commands: a TYPE I\r\n command to the server to request this. Figure 2 is a diagram that shows a few FTP commands and their responses from the server. Socket Programming in the Kernel Writing programs in the kernel is different from doing the same in user space. We explain a few issues concerned with writing a network application in the kernel. Refer to Greg Kroah-Hartman’s arti¬ cle “Things You Never Should Do in the Kernel” (see the on¬ line Resources). First, let’s examine how a system call in user space completes its task. For example, look at the socket () system call: ■ The client sends a USER \r\n command to the server to begin the authentication process. To send the password, the client uses PASS password\r\n'. ■ In some cases, the client sends a PORT command to inform the server of its preferred port for data transfer. In such cases, the client sends PORT \r\n. The RFC for FTP requires that the al-a4 constitute the 32- bit IP address of the client, and pl-p2 constitute the 16-bit port number. For example, if the client’s IP address is 10.10.1.2 and it chooses port 12001 for data transfer, the client sends PORT 10,10,1,2,46,225. ■ Some FTP clients request, by default, that data be trans¬ ferred in binary format, while others explicitly ask the serv¬ er to enable data transfer in binary mode. Such clients send client 1 f server USER parimi\r\n 331 Password required for parimi\r\n PASS foobar\r\n 230 User parimi logged in\r\n PORT a1,a2,a3.a4.p1.p2\r\n 200 PORT command successful TYPE l\r\n 200 TYPE set to I Figure 2. The client issues FTP commands over the control connection to set up the file transfer. sockfd = socket(AF_INET,S0CK_STREAM,0); When a program executes a system call, it traps into the kernel via an interrupt and hands over control to the kernel. Among other things, the kernel performs various tasks, such as saving contents of registers, making changes to address space boundaries and checking for errors with system call parame¬ ters. Eventually, the sys_socket() function in the kernel is responsible for creating the socket of a specified address and family type, finding an unused file descriptor and returning this number back to user space. Browsing through the kernel’s code, we can trace the path followed by this function (Figure 3). User Space socketQ Kernel Space 1. Save registers 2. Change address space boundaries 3. Check system call parameters for errors 4. Miscellaneous other checks 1. sock_create() sys_socket()- +~ 2 . sock_map_fd() 3. Get socket descriptor Copy socket descriptor to user-space Figure 3. Behind the scenes of a system call: when user space executes socket(), the kernel does necessary housekeeping and then returns a new file descriptor. Design of an FTP Client We now explain the design and implementation of a kernel FTP client. Please follow through the code available at the Linux Journal FTP site (see Resources) as you read through this article. The main functionality of this client is written in the form of a kernel module that adds a system call dynamically that user-space programs can invoke to start the FTP client pro¬ cess. The module allows only the root user to read a file using FTP. The user-space program that calls the system call in this module should be used with extreme caution. For example, it is easy to imagine the catastrophic results when root runs: ./a.out 10.0.0.1 10.0.0.2 foo_file /dev/hdal/* and overwrites /dev/hdal with a downloaded file from 10.0.0.1. Exporting sys_call_table We first need to configure the Linux kernel to allow us to add new system calls via a kernel module dynamically. Starting 241 OCTOBER 2005 WWW.LINUXJOURNAL.COM DON'T BE SQUARE! GET CUBED! L Series Laptop - LS125Q-L Light &Thin Performance Starting at $1,562.54 G Series Laptop - GW1550-L Essential Technology on a Budget Starting at 31,302.54 X Series Laptop - XW1550-L Extreme Technology & Performance Starting at $1,608.04 R Cubed Technologies has provided pre-installed Linux laptops without OS tax since 2003 We customize the Fedora Linux distribution for each laptops configuration providing support for: PCMCIA, USB, FireWire, X, CD/DVD/CDRW/DVDRW, Sound, Power Management, Ethernet, Modem, Wireless, and more. Our laptops are equipped with Intel Centrino Mobile Technology. We also otter Windows dual boot options All of our laptops come with a one year parts and labor warranty. Visit us online at www.shoprcubed.com or call 309.34.CUBED for details. Technologies' HarrJ to Br/Ov TeeAJMhltylfoS 10 Life. 309.34.CUBED www.shoprcubed.com All models, prices, and availability may vary. All trademarks are the prop* ©Copyright 2005 with version 2.6, the symbol sys_call_table is no longer exported by the kernel. For our module to be able to add a sys¬ tem call dynamically, we need to add the following lines to arch/i386/kernel/i386_ksyms.c in the kernel source (assuming you are using a Pentium-class machine): extern void *sys_caH_table; EXPORT_SYMBOL(sys_call_table); After recompiling the kernel and booting the machine into it, we are all set to run the FTP client. Refer to the Kernel Rebuild HOWTO (see Resources) for details on compiling a kernel. Module Basics Let’s examine the code for the module first. In the code snip¬ pets in this article, we omit error checking and other irrelevant details for clarity. The complete code is available from the LJ FTP site (see Resources): #include #include #include /* For socket etc */ #include #include #include #include #include #include #include #include #include int ftp_init(void) { printk(KERN_INFO FTP_STRING "Starting ftp client module\n"); sys_call_table[SYSCALL_NUM] = my_sys_call; return 0; } void ftp_exit(void) { printk(KERN_INFO FTP_STRING "Cleaning up ftp client module, bye !\n"); sys_call_table[SYSCALL_NUM] = sys_ni_syscall; } The program begins with the customary include directives. Notable among the header files are linux/kernel.h for KERN_ALERT and linux/slab.h, which contains definitions for kmalloc() and linux/smp_lock.h that define kernel-locking rou¬ tines. System calls are handled in the kernel by functions with the same names in user space but are prefixed with sys_. For example, the sys_socket function in the kernel handles the task of the socket () system call. In this module, we are using system call number 223 for our new system call. This method is not foolproof and will not work on SMP machines. Upon unloading the module, we unregister our system call. The System Call The workhorse of the module is the new system call that per¬ forms an FTP read. The system call takes a structure as a parameter. The structure is self-explanatory and is given below: struct params { /* Destination IP address */ unsigned char destip[4]; /* Source IP address */ unsigned char srcip[4]; /* Source file - file to be downloaded from the server */ char src[64]; /* Destination file - local file where the downloaded file is copied */ char dst[64]; char user[16]; /* Username */ char pass[64]; /* Password */ }; The system call is given below. We explain the relevant details in next few paragraphs: asmlinkage int my_sys_call (struct params __user *pm) { struct sockaddr_in saddr, daddr; struct socket *control= NULL; struct socket *data = NULL; struct socket *new_sock = NULL; int r = -1; char ^response = kmalloc(SNDBUF, GFP_KERNEL); char *reply = kmalloc(RCVBUF, GFP_KERNEL); struct params pmk; if(unlikely(!access_ok(VERIFY_READ, pm, sizeof(pm)))) return -EFAULT; if(copy_from_user(&pmk, pm, sizeof(struct params))) return -EFAULT; if (current->uid != 0) return r; r = sock_create(PF_INET, SOCK_STREAM, IPPR0T0_TCP, &control) ; memset(&servaddr,0, sizeof(servaddr)); servaddr.sin_family = AF_INET; servaddr.sin_port = htons(PORT); servaddr.sin_addr.s_addr = htonl(create_address(128, 196, 40, 225)); 261 OCTOBER 2005 WWW.LINUXJOURNAL.COM XEON. Flexibility to power the enterprise. From mail servers to databases, ZT Servers powered by the 64-bit Intel® Xeon™ Processor can run the full range of 32-bit applications and offer extended flexibility for your 64-bit needs. So you can create powerful, all-purpose IT infrastructure that enhances business agility - and the bottom line. Intel®Xeon™ Processors3 GHz - Upgradable to Dual Intel® Xeon™ Processors 3.60 GHz, 1MB L2 Cache, 800MHz FSB ■ Intel® E7320 Chipset ServerBoard ■ 512MB ECC Registered DDR 333 SDRAM (Upto4GB) ■ 2xSeagate® 300GB 10,000rpm SCSI Hard Drive (Raid l) ■ 4x1" SCSI Hot-Swappable Drive Bays ■ Slim CD-ROM & Floppy Drive ■ 2 x Intel® 10/100/1000 Gigabit Network Controller ■ 1U RackmountChassisw/420WCold-Swappable PowerSupply ■ 3-Year Limited Warranty and First Year On-Site Service $ 3,999 Dual Intel® Xeon™ Processors 3GHz - Upgradable to Dual Intel® Xeon™ Processors 3.60 GHz, 1MB L2 Cache, 800MHz FSB ■ Intel® E7520 Chipset ServerBoard ■ 1GB ECC Registered DDR2 400 SDRAM (Upto 16GB) ■ 8 x Seagate® 300GB SATA Hard Drive (Total 2.4TB storage) ■8x1" Hot-Swap SATA Drive Bays ■ DVD±RW Burner & Floppy Drive ■ 8 Channel SATA Controller (RAID o, 1,5,10,50 Support) ■ 2xlntel® 10/100/1000 Gigabit Network Controller ■ 3U RackmountChassis w/550W PowerSupply ■ 3-Year Limited Warranty and First Year On-Site Service Dual Intel® Xeon™ Processors 3GHz - Upgradable to Dual Intel® Xeon™ Processors 3.60 GHz, 1MB L2 Cache, 800MHz FSB ■ Intel® E7520ChipsetServer Board ■ 1.0GB ECC Registered DDR 333 SDRAM (Upto 16GB) ■ 4 x Seagate® 250GB SATA Hard Drive (Total itb storage) ■ 6x1" Hot-Swap SATA Drive Bays ■ Slim CD-ROM & Floppy Drive ■ 8 Channel SATA Controller (RAID o,i,JBOD Support) ■ 2xIntel® lO/lOO/lOOOGigabitNetworkController ■ 2U RackmountChassisw/550W PowerSupply ■ 3-Year Limited Warranty and First Year On-Site Service $ 2,499 Intel® Xeon™ Processors 3GHz - Upgradable to Dual Intel® Xeon™ Processors 3.60 GHz, 1MB L2 Cache, 800MHz FSB ■ Intel® E7320 Chipset ServerBoard ■ 1.0GB ECC Registered DDR 333 SDRAM (Upto 16GB) ■ 4 x Seagate® 250GB SATA Hard Drive (Total itb storage) ■ 7x1" Hot-Swap SATA Drive Bays ■ DVD±RW Burner & Floppy Drive ■ 4 Channel Raid Controller (RAID o, 1,1/0,5, JBOD Support) ■ 2xlntel® 10/100/1000 Gigabit Network Controller ■ 4U Rackmount Chassis w/550W Redundant Cooling Power Supply ■ 3-Year Limited Warranty and First Year On-Site Service ■ 3 . New Powerful 64-bit Server Line Friendly Server Specialists Complete Solution Provider Flexibility and Variety: Customize platforms with leading-edge products We Build the Future ! $ 4,299 $ 2,199 # 1. OEM Computer Manufacturer New Accounts Receive Free Gift ■ 3 year warranty with lifetime tech support ■ Personal attention (Dedicated Technical Sales Team) ■ Reseller and volume pricing available. ■ Call now to customize using the latest technology Find out how ZT Insider Program can help maximize your Business Solution Goto ztgroup.com/go/linuxjoumal Call 866- ZTGROUP (866-984-7687) Prmotecode: Ij 1005 Purchaser is responsible for all freight costs on all returns of merchandise. Full credit will not be given for incomplete or damaged returns. Absolutely no refunds for merchandise returned after 30 days. All prices and configurations are subject to change without notice and obligation. Opened software is non-refundable. All returns have to be accompanied with an RMA number and must be in re-sellable condition including all original packaging. System’s picture may include some equipments and/or accessories, which are not standard features. Not responsible for errors in typography and/or photography. All rights reserved. All brands and product names, trademarks or registered trademarks are property of their respective companies. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, Pentium, and Pentium III Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. How do you make high-performance computing even better? Customize it. Using lntel®-based servers to maximize power and interoperability, Intel® Premier Providers build high-performance technology solutions customized to your enterprise’s needs. As members of the most elite Intel® channel program, Premier Providers deliver the latest IT solutions, and have priority access to parts and technical support. Insist on the Best. premier PROVIDER Find the Intel® Premier Provider that’s right for you. www.insistthebest.com r = control->ops->connect(control, (struct sockaddr *) &servaddr, sizeof (servaddr), 0_RDWR); read_response(control, response); sprintf(temp, "USER %s\r\n", pmk.user); send_reply(control, temp); read_response(control, response); sprintf(temp, "PASS %s\r\n", pmk.pass); send_reply(control, temp); read_response(control, response); We start out by declaring pointers to a few socket structures, kmalloc () is the ker¬ nel equivalent of malloc () and is used to allocate memory for our character array. The array’s response and reply will contain the responses to and replies from the server. The first step is to read the parameters from user mode to kernel mode. This is cus¬ tomarily done with access_ok and veri fy_read/veri fy_wri te calls. access_ok checks whether the user-space pointer is valid to be referenced, veri fy_ re ad is used to read data from user mode. For reading simple variables like char and i nt, use __get_user. Now that we have the user-specified parameters, the next step is to create a control socket and establish a connection with the FTP server. sock_create () does this for us—its arguments are similar to those we pass to the user-level socket () system call. The struct sockaddr_in variable servaddr is now filled in with all the necessary information—address family, destination port and IP address of the server. Each socket structure has a member that is a pointer to a structure of type struct prot o_o p s. This structure contains a list of function pointers to all the operations that can be per¬ formed on a socket. We use the connect () function of this structure to establish a connection to the server. Our functions read_response() and send_reply() transfer data between the client and server (these functions are explained later): r = sock_create(PF_INET, SOCK_STREAM, IPPROTO_TCP, &data); memset(&claddr,0, sizeof(claddr)); claddr.sin_family = AF_INET; claddr.sin_port = htons(EPH_PORT); clddr.sin_addr.s_addr= htonl( create_address(srci p)); r = data->ops->bind(data, (struct sockaddr *)&claddr, sizeof (claddr)); r = data->ops->listen(data, 1); Now, a data socket is created to transfer data between the client and server. We fill in another struct sockaddr_in variable claddr with information about the client— protocol family, local unprivileged port that our client would bind to and, of course, the IP address. Next, the socket is bound to the ephemeral port EPH_PORT. The func¬ tion 1 i sten () lets the kernel know that this socket can accept incoming connections: a = (char *)&claddr.sin_addr; p = (char *)&claddr.sin_port; send_reply(control, reply); read_response(control, response); strcpy(reply, "RETR ") ; strcat(reply, src); strcat(reply, "\r\n"); send_reply(control, reply); read_response(control, response); 28 OCTOBER 2005 WWW.LINUXJOURNAL.COM The Intel® Xeon™ processor now provides high availability for your 64-bit applications and clusters. The Intel® Xeon™ processor now works harder for your business than ever. With innovative features that enable power-saving options, flexible memory, I/O and storage configuration. And, of course, continued support for all your existing 32-bit applications. How can clusters featuring the Intel Xeon processor serve you? intel.com/go/xeon Atipa Technologies 4921 Legends Drive Lawrence, KS 66049 888-222-7822 www.atipa.com Bell Computers 187 Pacific Street Pomona, CA 91768 909-598-1006 www.bellcomputer.com Western Scientific 9445 Farnham Street San Diego, CA 92123 858-565-6699 www.westernscientific.com/intel ©2005 Intel Corporation. Intel, the Intel logo, Intel Inside and Intel Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. All rights reserved. Compilers are building the 64-bit applications infrastructure. C, C++, F77, F95 and HPF • 32-bit and 64-bit Linux Optimized for AMD64 and IA32/EM64T • Full 64-bit support Workstation, Server and Cluster configurations • Fast compile times Native OpenMP • Native SMP auto-parallelization • Cache tiling Function inlining • SSE/SSE2 Vectorization • Loop unrolling Interprocedural optimization • Profile-feedback optimization Large file support on 32-bit Linux • 64-bit integers and pointers F77 pointers • Byte-swapping I/O • VAX and IBM extensions OpenMP/MPI/threads debugging • OpenMP/MPI/threads profiling Interoperable with g77/gcc/gdb • PDF and printed documentation Electronic purchase, download and upgrades • Tech support Network-floating licenses • Academic and volume discounts Visit www.pgroup.com to download a free PGI evaluation package and see the latest tips and techniques for porting to 64-bit systems. The Portland Group www.pgroup.com ++01 (503) 682-2806 STMicroelectronics The registered trademarks and marks are the property of their respective owners. As explained previously, a PORT command is issued to the FTP server to let it know the port for data transfer. This command is sent over the control socket and not over the data socket: 64-bit GAUSSIAN new_sock = sock_alloc(); new_sock->type = data->type; new_sock->ops = data->ops; r = data->ops->accept(data, new_sock, 0); new_sock->ops->getname(new_sock, (struct sockaddr *)address, &len, 2); Now, the client is ready to accept data from the server. We create a new socket and assign it the same type and ops as our data socket. The accept () function pulls the first pending connection in the listen queue and creates a new socket with the same connection properties as data. The new socket thus created handles all data transfer between the client and server. The getnameO function gets the address at the other end of the socket. The last three lines in the above segment of code are useful only for printing information about the server: if((total_written = write_to_file(pmk.dst, new_sock, response)) < 0) goto err3; The function wri te_to_f i le deals with opening a file in the kernel and writing data from the socket back into the file. Writing to sockets works like this: void send_reply(struct socket *sock, char *str) { } send_sync_buf(sock, str, strlen(str), MSG_DONTWAIT); int send_sync_buf (struct socket *sock, const char *buf, const size_t length, unsigned long flags) { struct msghdr msg; struct iovec iov; int len, written = 0, left = length; mm_segment_t oldmm; msg.msg_name = 0; msg.msg_namelen = 0; msg.msg_iov = &iov; msg.msg_iovlen = 1; msg.msg_control = NULL; msg.msg_controllen = 0; msg.msg_flags = flags; oldmm = get_fs(); set_fs(KERNEL_DS) ; Gaussian 03 is the premier electronic structure program. Chemists and other scientists use it to study important molecules and reactions related to drug design, materials science, catalysis, and other areas of leading edge and commercial research interest. See www.gaussian.com to learn about the latest Gaussian 03 innovations that make it applicable to very large molecules previously out of reach of accurate models. Gaussian, Inc builds Gaussian 03 for 64-bit AMD64 and EM64T processor-based systems using PGI Compilers and Tools. repeat_send: msg.msg_iov->iov_len = left; msg.msg_iov->iov_base = (char *) but + written; len = sock_sendmsg(sock, &msg, left); return written ? written : len; } The send_reply() function calls send_sync_buf (), which does the real job of sending the message by calling sock_sendmsg() . The function sock_sendmsg() takes a pointer to struct socket, the message to be sent and the message length. The message is represented by the struture msghdr. One of the important members of this structure is i ov (io vector). The iovector has two members, iov_base and iov_len: struct iovec { /* Should point to message buffer */ void *iov_base; /* Message length */ _kernel_size_t iov_len; }; These members are filled with appropriate values, and sock_sendmsg() is called to send the message. The macro set_f s is used to set the FS register to point to the kernel data segment. This allows sock_sendmsg() to find the data in the kernel data segment instead of the user- space data segment. The macro get_f s saves the old value of FS. After a call to sock_sendmsg(), the saved value of FS is restored. Reading from the socket works similarly: int read_response(struct socket *sock, char *str) { len = sock_recvmsg(sock, &msg, max_size, 0); return len; } The read_response() function is similar to send_reply () . After filling the msghdr structure appropriately, it uses sock_recvmsg() to read data from a socket and returns the number of bytes read. A User-Space Program Now, let’s take a look at a user-space program that invokes our system call to transfer a file. We explain the relevant details for calling a new system call: #define_NR_my_sys_call 223 _syscalll(long long int, my_sys_call, struct params *, p); int main(int argc, char **argv) { struct params pm; /* fill pm with appropriate values */ r = my_sys_call(&pm); } #d e f i n e N R_my_sy s_c all 223 assigns a number to our system call. _syscalll() is a macro that creates a stub for the system call. It shows the type and number of arguments that our system call expects. With this in place, my_sys_call can be invoked just like any other system call. Upon running the pro¬ gram, with correct values for the source and destination files, a file from a remote FTP server is downloaded onto the client machine. Here is a transcript of a sample run: # make make -C /lib/modules/2.6.9/build SUBDIRS=/home/ppadala/ftp modules make[1]: Entering directory '/home/ppadala/linux-2.6.9’ CC [M] /home/ppadala/ftp/ftp.o Building modules, stage 2. MODPOST CC /home/ppadala/ftp/ftp.mod.o LD [M] /home/ppadala/ftp/ftp.ko make[1]: Leaving directory '/home/ppadala/linux-2.6.9 1 # gcc do_ftp.c # ./a.out 152.2.210.80 /README /tmp/README anonymous anon@cs.edu Connection from 152.2.210.80 return = 215 (length of file copied) Conclusions We have seen a basic implementation of an FTP client within the kernel. This article explains various issues of socket pro¬ gramming in the kernel. Interested readers can follow these ideas to write various network applications, such as an HTTP client or even a Web server in the kernel. Kernel applica¬ tions, such as the TUX Web server are used for high-perfor¬ mance content serving and are well suited for environments that demand data transfer at high rates. Careful attention has to be paid to the design, implementation and security issues of such applications. Resources for this article: www.linuxjournal.com/article/ 8453.0 Pradeep Padala is a PhD student at the University of Michigan. His general interests are in distributed systems with specific emphasis on scheduling and fault tolerance. He is the author of the NCurses Programming HOWTO and contributes to various open-source projects. More about him can be found on his Web site at www.eecs.umich.edu/~ppadala. Ravi Parimi has a Master's degree in Computer Engineering and currently works in Silicon Valley, California. His main interests are in operating systems, networking and Internet security. He has been using Linux since 1998 and aspires to be a kernel hacker. In his free time, he pursues Vedic studies and Chess. 321 OCTOBER 2005 WWW.LINUXJOURNAL.COM Where Open Minds Meet... October 5-6 • 2005 • Olympia 2 • London The UK's only event for business and the technical community Come along to LinuxWorld Expo and tackle IT business issues, gain real-time Linux and open source solutions, meet key suppliers, ask technical points and get answers, discover 'how-to', see new technologies, source products and network with the entire community; experts, colleagues and suppliers; Register NOW at www.linuxworldexpo.co.uk for FREE entry into the exhibition, featuring; • The Great Linux Debate • OSC/Open Source Academy • FREE Product Briefings and Demonstrations • FREE Showcases of Technology • FREE Case Study Presentations # FREE Open Forum Europe Advice Centre # .org village # Internet Cafe with Wireless Connection • Pre-registered visitors can take the LPI exam for ONLY £25 - Saving £100 CONFERENCE PROGRAMMES Technical and Enterprise: Two streams, two days of informative and in-depth sessions covering key issues and topics for technical and enterprise. Linux in Business: Practical applications, benefits and analysis of Linux and open source in business: SPEAKERS INCLUDE: Klaus Knopper 'Engineer, Consultant and Software Developer, KNOPPER.NET& Maker, Knoppix Mark Shuttleworth, President, Ubuntu Foundation Rasmus Lerdor [Engineer, Yahoo! Inc. Paul Everitt, Founder, Zope Europe Association Linux for the Technical team: What's here now; and what's coming next. SPEAKERS INCLUDE: Andrew Eddie, Senior Systems Integrator, Toowoomba City Council, Australia Kevin Carmony, President and CEO, Unspire Inc Michael Tiemann, Vice President Open Source Affairs, Red Hat Brian Green, Director of Solutions Management, Novell Delegate places are limited, so book today and also benefit from: • FREE LPI Examination - Saving £125 • GUARANTEED seating at The Great Linux Debate • FREE Entry to the exhibition View the full conference programme and book your place TODAY! - www.linuxworldexpo.co.uk Register NOW at www.linuxworldexpo.co.uk GOLD SPONSOR D0U PLATINUM SPONSORS NOVell s|sf ORACLE OWNED BY *IDG WORLD EXPO ORGANISED BY turret GROU P | All trademarks acknowledged. E&OE. Programme may be subject to change. Correct at time of press. Trekking through the Desktop Jungle Is it easier to find a document on a faraway Web server than one on your own hard drive? Try some search programs to dig up the files you need. BY MARCEL GAGNE T hat certainly does make it difficult, Francois. When I asked you to locate the wine order from last month and you told me it was somewhere on your disk, I didn’t expect that it was sitting “somewhere on your disk” in quite this way. This is possibly the most disorganized home direc¬ tory I’ve ever seen. Every document is in the same folder, and all the files are cryptically named. What were you thinking, mon amil Quoi? Well, of course there is a way to find it. If the docu¬ ment still exists somewhere on your disk, we’ll find it. We just need to use the right tools. Later, though—our guests will be here any moment and...too late, Francis, they are already here! Welcome, everyone, to Chez Marcel , home of fine Linux fare and exquisite wines. Please sit and make yourselves com¬ fortable. Francis will fetch your wine immediatement. Francis, head to the east wing of the wine cellar and bring back that 2001 Nuits Saint George Pinot Noir we’ve been tast¬ ing, er, I mean, subjecting to quality control. Vite! That wine, mes amis , just happens to represent part of an order lost in one of Francis’ documents on his computer. Trouble is, he doesn’t remember which document. What we need to do, is set him up with a desktop search engine. Luckily, this just happens to be the basis of tonight’s menu, so we all will profit from my faithful waiter’s lack of organization. The original desktop search engine, mes amis , is something that’s been around in Linux from the beginning, and that’s the find command. This is an amazingly powerful tool and one that is easily overlooked in this age of cutting-edge graphical desk¬ tops. In its most basic form, find is used like this: find starting_dir [options] One of those options is -print, which makes sense only if you want to see any kind of output from this command. You easily could get a listing of every file on the system by starting at the top and recursively listing the disk: find / -print Of course, it makes more sense to search for something, for instance, all the MP3-type music files sitting on your disk. Because you know that the files end in a .mp3 extension, you can use that to search: find / -name "*.mp3" -print This is also great for locating big files you haven’t looked at in forever. Maybe it’s time to do a little archiving of those old files, but how do you find only them? Say you want to look for anything that has not been modified (this is the -mtime parameter) or accessed (the -atime parameter) in the past 12 months. The -o option is the “or” in this equation: find /home/marcel -size +1024 \( -mtime +365 -o -atime +365 \) -Is In case you are curious, the back-slashes in front of the parentheses are escape characters; they are there to make sure the shell does not interpret them in ways you do not want it to—in this case, the open and close parentheses on the second line. The preceding command also searches for files that are greater than 500KB in size. That is what the - size +1024 means, because 1024 refers to 512-byte blocks. The - Is at the end of the command tells the system to do a long listing of any files it finds that fit the search criteria. So far so good? The find command is fairly simple to use on the surface, but it also has many command-line options and (as you can see) interesting ways of passing the results of a search to other commands, so that the results can be narrowed down or fine- tuned. Getting to know find is a great idea, but there are alter¬ natives that are a little friendlier. Many people out there have grown up in the graphical world of KDE or GNOME, so desktop tools have been created in each of these environments. Even so, my experience indi¬ cates that these excellent tools are, for many users, as equally overlooked as find. Let’s have a look at those now. Let’s begin our search for search tools under KDE. Click the application launcher and look for a submenu labeled Find. The Find menu has two options, one for files and one for Web search (which, by default, launches Konqueror on the Google Web site). You also can fire up the files search tool by using the Alt-F2 quick launch (program name: kf i nd). When the application starts, the Find Files/Folders dialog appears. It contains three different tabs, and each is designed to help you locate the information you need. They are labeled Name/Location, Contents and Properties. Under the Name/Location tab, specify the starting folder, either by entering it manually or by clicking the Browser but¬ ton and navigating over to it using the KDE file navigator. There’s also a field labeled Named where you enter part of a filename using Linux metacharacters. For instance, if I wanted to find all the files with Cooking anywhere in the title, I would enter *cooking*. By default, this is a case-insensitive search, so upper- and lowercase don’t matter in terms of the search results. You can, however, override this behavior by clicking the Case-sensitive search check box. Under the Contents tab, the real action takes place. Generally speaking, I don’t have a problem locating a file by name. It’s the content that is the real issue. Which of your sev¬ eral hundred documents contains a reference to a particular 341 OCTOBER 2005 WWW.LINUXJOURNAL.COM Name/Location ~~J Contents | Properties j File type: All Files & Folders Containing text [wine Q Case sensitive Q Regular expression □ Indude binary files Search metainfo sections: I* [ for: 1 Etnd j Save As... Name ▼ In Subfolder I s ”, J Modified Per- lir< ' Cooking June 2004 Democrazy.txt 15,119 18/03/04 10:02 am 1 [7 Cooking June2005 lntranets.txt 2005/ 13,749 11/03/05 07:39 pm 1 7] Cooking Mar2004Xnesttxt 12,133 09/12/03 12:22 pm 1 [J! Cooking May2004 Superkaramba.txt 11,581 17/02/04 04:54 pm 1 —I 0 Cooking May2005 Crossing Platforms.txt 2005/ 10,896 11/02/05 04:20 pm [7 Cooking Nov 2004 High Peformance.txt 2004/ 12,338 13/08/04 05:53 pm 71 Cooking Oct 2004 Security.txt 2004/ 12,753 08/07/04 04:50 pm 7 Cooking Sept 2004Wireless.txt 2004/ 12,157 11/06/0411:48 am I [7 Cooking Sept2005 Wireless.txt 2005/ 13,059 09/06/05 10:43 am 1 < 1 VLM Ready. 20 files found Figure 1. KFind makes it easy for Marcel to locate all those columns that mention "wine". Figure 2. The GNOME search tool allows you to search by name as well as text within a file. word or phrase is a more difficult search than which has a par¬ ticular word in the name. The Contents tab lets you enter your search text (again, case-insensitive by default), regular expres¬ sion searches and so on. You even can specify that Kfind search through binary files and not only documents (Figure 1). There’s also a meta-info search feature for things like MP3 files that contain embedded information, such as title and artist. Finally, the Properties tab provides a means of searching for files or folders based on creation or modification date, own¬ ership and more. Similarly, GNOME users have access to the GNOME search tool (program name: gnome-search-tool), a similar program that lets you search based on filename, file content (text search) and date. Choose Search for files in the GNOME Places menu (I’m running 2.10 in this example), and this brings up the file find dialog (Figure 2). When the dialog first appears, there isn’t much to see. The defaults are to search for a file by name, which you enter in the Name contains field. Below that is your starting folder for the search, the default being your home directory. To get the full power of the GNOME search tool, click on the arrow next to the label that says Show more options. A new field appears through which you can specify some text in the file itself. Finally, directly below the text search field, is one other option that can be quite complex. A drop-down box labeled Available options includes size, date and ownership search cri¬ teria that can be applied to narrow down your search results even further. If you’ve been following search technology in any way, you’ll know that there’s a lot of excitement concerning desktop search engines these days—think Google for your desktop. In fact, Google does provide such a tool, but alas, only for non- Linux operating systems. However, this is not to say that desk¬ top search tools don’t exist for Linux. One such tool is Roberto Cappuccio’s Kat, a desktop search engine and indexing tool that makes it easy and fast to do full- text searches in a variety of document formats (for example, PDF, OpenOffice.org, KWord and so on). You also can search for images using thumbnails and more. The Kat Web site (see the on-line Resources) provides bina¬ ry packages for a number of distributions, so you may not need to build from source. Should you need to, however, the process is nothing more than the classic extract-and-build five-step. In terms of prerequisites, you need the SQLite database and its development libraries. To use Kat, simply start the program (name: kat) and a plain three-pane window appears where you will do your work We’ve got problems with your name on them. At Google, we process the world’s information and make it accessible to the world's population. As you might imagine, this task poses considerable challenges. Maybe you can help. We’re looking for experienced software engineers with superb design and implementation skills and expertise in the following areas: • high-performance distributed systems • operating systems • data mining • information retrieval • machine learning • and/or related areas If you have a proven track record based on cutting-edge research and/or large-scale systems development in these areas, we have brain-bursting projects with your name on them in Mountain View, Santa Monica, New York, Bangalore, Hyderabad, Zurich and Tokyo. Ready for the challenge of a lifetime? Visit us at http://www.google.com/lj for information. EOE WWW.LINUXJOURNAL.COM OCTOBER 2005135 Figure 3. Using kfile hooks, Kat can index almost anything. Figure 4. As Kat creates the new catalog, the program reports statistics on the process. and your searching. The first step is to create a catalog. To do this, click File on the menu bar and select New. When creating a new catalog, a four-tabbed window appears. The first tab, labeled Catalog, is where you enter the starting directory, the name of the catalog and other identifying information. On the second tab, labeled Metadata, you’ll see a list of the various metadata engines that are available to Kat for indexing. You can remove different formats, but most likely, this will stay as is (Figure 3). Similarly, the Fulltext tab. Under Thumbnails, you can select the size of the thunmbnails created during the index process. A status window keeps you abreast of the number of files and folders scanned, as well as the size of the collection (Figure 4). This brings us to the one big drawback of a tool like this. If Figure 5. Although the initial indexing can take some time, Kat searches are blazingly fast. the folder for which you are creating a catalog is large, this can take an amazing amount of time. Be prepared or keep your cat¬ alogs confined to a reasonable collection of files. I tried to index my own home directory in its entirety at nearly 6.6GB of data—suffice it to say, that was a mistake. Once a catalog has been created, finding information is blazingly fast. Simply click on the search icon on the far right (the magnifying glass), enter your search term and Kat returns the results of the search almost instantly (Figure 5). According to the clock on the wall, it would appear, mes amis , that closing time has arrived. Before we leave this topic of desktop search engines, I’d like to mention another package with the friendly, puppy-dog name of Beagle. Beagle is built on Mono (the open-source .Net implementation) and requires an inotify-enabled kernel. Neither is uncommon in the more modern distributions. Beagle also shows promise in that it is very fast and works silently in the background, keeping an eye on what you tell it while automatically updating its catalog of information. Unfortunately, Beagle is very much alpha code and not quite ready for prime time, as they say (although it is included with the new SUSE Linux Professional 9.3). Nevertheless, Beagle is a tool to watch, and I’ve included the link in the on-line Resources. Please raise your glasses, mes amis , and let us all drink to one another’s health. A votre sante! Bon appetit! Resources for this article: www.linuxjournal.com/article/ 8456.0 Marcel Gagne is an award-winning writer living in Mississauga, Ontario. He is the author of Moving to the Linux Business Desktop (ISBN 0-131-42192-1), his third book from Addison- Wesley. He also makes regular television appear¬ ances as Call for Help's Linux guy. Marcel also is a pilot and a past Top-40 disc jockey. He writes science fiction and fantasy and folds a mean Origami T-Rex. He can be reached via e-mail at mggagne@salmar.com. You can discover a lot of other things (including great Wine links) from his Web site at www.marcelgagne.com. 3 6 H OCTOBER 2005 WWW.LINUXJOURNAL.COM Best Price Best Performance. Best Support Enterprise Router™ The Enterprise features four 8 Gbps buses, dual CPUs, and redundant AC or DC power supplies. It can route multiple 0C3 or 0C12 circuits at wire speed. The Transport " is ImageStream's best¬ selling router. With its small footprint, business-class features, and competitive price, the TransPort is an ideal rout er for T1 and El applications that demand low latency wire-speed performance. The TransPort includes 128 MB RAM, three 10/100 ethernet ports, and an expansion slot for add-on cards. It also features the ImageStream Linux™ router distribution, which supports most WAN protocols and advanced features including NAT firewall, peer-to-peer traffic control, bridging, bandwidth limiting, QoS, dynamic routing, VPN, and more. Like all ImageStream routers, the TransPort includes 12 months of free 24/7 technical support, a full 12-month warranty on parts and labor, free software upgrades for life, and the industry's only money back performance guarantee. Gateway Router™ ImageStream's Gateway is the industry's lowest cost 0C3 router. The upgraded dual bus Gateway 64™ can route multiple DS3/E3 or 0C3 circuits at wire speed. UJelL Connected The TransPort provides three 10/100 ethernet ports for flexible LAN connectivity. Faoless CPU The Transport's highly efficient CPU minimizes power consumption and cooling requirements. Best Software The ImageStream Linux™ router distribution has everything you need to deploy advanced network applications including secure shell, menu-based configuration, real-time monitoring, IP firewall, bridging, interface bonding, dynamic routing, QoS, IPsec VPN, free software upgrades for life, and more. Best Support ImageStream routers include a full year of free 24/7 support. When you consider total cost of ownership, ImageStream routers cost less up front and over time. Rebel Router™ The Rebel is the industry's lowest cost DS3/E3 router. The 1U Rebel can route one DS3/E3 or up to 16 T1/E1 circuits at wire speed. Rf Router The R1 is designed for extended tempera¬ ture applications including outdoor installa¬ tions. The R1 has a small footprint, and can be installed in set-top, wall-mount, and rackmount applications. Easy Indicators The TransPort provides front panel LEDs to show LAN connection status. -8.7 in. pi 0.0 in_ Sized Riyht The Transport's small footprint allows it to be installed just about anywhere. • Guaranteed Performaoce our 31-day performance guarantee ensures your router will function as promised. If your ImageStream router does not work as specified in writing, and our support team cannot correct the problem, you can return your router for a full refund of the purchase price. See our web site for details. ZM ImageStream. Routers for the Real World" 800.813.5123 www. imagestream. com 1 . 574 . 935.8484 ImageStream, Enterprise Router, Gateway Router, Gateway 64, Rebel Router, Rl, TransPort, ImageStream Linux and "Routers for the Real World" are trademarks of ImageStream Internet Solutions, Inc. Linux is a registered trademark of Linus Torvalds. Specifications are subject to change without prior notice. *Please refer to ImageStream's Web site for more information on wire-speed specifications and the performance guarantee. Limitations of she, a Shell Encryption Utility The shell script compiler, she, obfuscates shell scripts with encryption—but the password is in the encrypted file. Could an intruder recover the original script using objdump? by nalneesh guar he is a popular tool for protecting shell scripts that con¬ tain sensitive information such as passwords. Its popu¬ larity was driven partly by auditors’ concern over pass¬ words in scripts, she encrypts shell scripts using RC4, makes an executable binary out of the shell script and runs it as a normal shell script. Although the resulting binary contains the encryption password and the encrypted shell script, it is hidden from casual view. At first, I was intrigued by the she utility (www.datsi.fi.upm.es/~frosal/sources/shc.html) and consid¬ ered it as a valuable tool in maintaining security of sensitive shell scripts. However, upon further inspection, I was able to extract the original shell script from the shc-generated exe¬ cutable for version 3.7. Because the encryption key is stored in the binary executable, it is possible for anyone with read access to the executable to recover the original shell script. This arti¬ cle details the process of extracting the original shell exe¬ cutable from the binary generated by she. she Overview she is a generic shell script compiler. Fundamentally, she takes as its input a shell script, converts it to a C program and runs the compiler to compile the C code. The C program contains the original script encrypted by an arbitrary key using RC4 encryption. RC4 is a stream cipher designed in RS A laborato¬ ries by Ron Rivest in 1987. This cipher is used widely in com¬ mercial applications, including Oracle SQL and SSL. Listing 1 demonstrates running she. The two new files, named with the .x and .x.c extensions to the name of the source shell script, are the executable and an intermediate C version. Upon executing pub.sh.x, the original shell source is executed, she also specifies a relax option, -r. The relax option is used to make the executable portable. Basically, she uses the contents of the shell interpreter itself, Listing 1. Running she [userl@shiraz test]# cat pub.sh #!/bin/sh echo "Hello World" userl@shiraz test]# ./pub.sh Hello World [userl@shiraz test]# she -v -r -f pub.sh she shll=sh she [-i]=-c she [-x]=exec '%s' she [-1]= she opts= she: cc pub.sh.x.c -o pub.sh.x she: strip pub.sh.x [userl@shiraz test]# Is pub.sh pub.sh.x pub.sh.x.c [userl@shiraz test]# ./pub.sh.x Hello World such as /bin/sh, as a key. If the shell binary were to change, for example, due to system patching or by moving the binary to another system, the shc-generated binary does not decrypt or execute. I inspected the shell executable using strings and found no evidence of the original shell script. I also inspected the inter¬ mediate C source code and noted that it stores the shell script in encrypted octal characters, as depicted in Listing 2. Listing 2. The original shell script becomes an RC4-encrypted string in the C version. static char text[] = "\223\004\215\264\102\216\322\060\300\070\101\217\277\161\033\130" "\217\145\370\170\106\257\176\301\057\132\172\044\217\247\276\222" "\203\076\334\201\323\107\064\334\120\132\001\241\267\052\203\216" "\116\232\156\337\121\145\235\003\156\244\142\246\117\200\206\014" "\004\153\372\152\030\262\171\275\137\342\247\367\231\315\353\151" "\264\241\230\105\344\053\034\247\342\142\156\305\327\255\036\111" "\234\061\013\355\300\336\324\257\175\124\222\044\132\040\276\067" "\007\002\371\063\021\320\060"; The C source code also includes as arrays the password as well as other encrypted strings. Therefore, anyone with access to the source code easily can decrypt and view the contents of the original shell script. But what about the original shell binary executable generated by she? Is it possible to extract the original shell script from nothing but the binary executable? The answer to this question is explored in the next section. Extraction Approach I generated and reviewed the C source code for several shell scripts to better understand how the shell source is encrypt¬ ed and decrypted. Fundamentally, she uses an implementa¬ tion of RC4 that was posted to a Usenet newsgroup on September 13, 1994. I set off by first identifying the encryp- 38 OCTOBER 2005 WWW.LINUXJOURNAL.COM tion key and the encryption text. The objdump utility came in handy for this, bjdump, part of GNU binutils, displays information about object files. First, we use objdump to retrieve all static variables, for this is where the encryption key and the encrypted shell text are stored. Listing 3 pro¬ vides a brief overview of objdump. The first column of the output in listing 3 specifies the starting address¬ es in hexadecimal, followed by the stored data in the next four columns. The last column represents the stored data in printable characters. So some¬ where in the first four columns of the output is the array of characters that form the encryption key (password) and the encrypted shell script. Comparing the original C source code and Listing 3, you can see that the password most likely begins at address 0x804a540. After comparing other executables, I determined that the first address after the zeros leading the “Please contact your provider” text usually is the starting address. To retrieve these arrays, such as the one depicted in Listing 2, we also need to look at the disassembled code. We use objdump again here, except this time with the -d option, for disassemble, as shown in Listing 4. The last two columns represent assembly instructions. The movl instruction is used to move data— movl Source, Dest. The Source and Dest are prefixed with $ when referenc¬ ing a C constant. The push takes a single operand, the data source, and stores it at the top of stack. Now that we have the basics of objdump, we can proceed to extract the encryption password and eventually the shell code. In the intermediate C code pro¬ duced by she, about nine arrays are referenced by the variables pswd, shll, inlo, xecc, lsto, chkl, opts, txt and chk2. The pswd variable stores the encryption key, and the txt variable stores the encrypted shell text, she hides the useful information as smaller arrays within these variables. Thus, obtaining the actual array involves two steps. First, identify the length of the array. Second, identify the starting address of the array. The objdump output needs to be looked at in detail to obtain the actual I Need a sharper ^ development jBfek tool for your fljf application’s or database? AMD Opteron V SQL is only one of our options... C-TREE PLUS® DATABASE TECHNOLOGY I FEATURED I CUSTOMER OPENS UP YOUR OPTIONS HIGHLIGHTS TESTIMONIAL SQL offers a convenient and easy-to-use database interface. ISAM provides powerful performance with precision indexing control in a small footprint. With c-tree Plus you can simultaneously enjoy BOTH! Superior ISAM indexing technology PLUS an industry-standard SQL interface provide blazing fast data management for every environment. Break the limitations of a single solution and open up your database options. Experience the benefits c-tree Plus can deliver to your application! • Fast, reliable, and portable • Low deployment cost • No DBA required • Professional technical support • Source code • 64-bit support • 16-exabyte file support • Memory files • Embeddable database • Full OLTP support “We have reviewed Oracle and some of the other big relational databases and chose FairCom for our database development needs. With c-tree Plus, we see transactional volume that is 8 to 10 times faster than what we can get with other databases. I have been using c-tree based solutions since the 80 's and highly recommend it... ’’ Visit our Web site for more testimonials about c-tree! Database your way. See for yourself — download c-tree Plus® Today! Go to www.faircom.com/go/open for a FREE evaluation of c-tree Plus! Other company and product names are registered trademarks or trademarks of their respective owners. © 2005 FairCom Corporation WWW.LINUXJOURNAL.COM OCTOBER 20051 39 Listing 3. objdump browses the object file for interesting-looking strings. /usr/bin/objdump --section=.data -s pub.sh.x pub.sh.x: file format elf32-i386 Contents of section .data: 804a4e0 804a4f0 804a500 804a510 804a520 804a530 804a540 804a550 804a560 00000000 00000000 00000000 63742079 00000000 00000000 e554f49f 7a9beb67 eba28b7e 00000000 00000000 506c6561 6f757220 01000000 00000000 93dcd6dc 60277cb2 7e615a3a 3ca80408 00000000 73652063 70726f76 00000000 00000000 bb0bdc9b dd9e0886 6d37d51a 00000000 00000000 6f6e7461 69646572 00000000 00000000 ad60edd0 0797aeec 97c2eall .... Please conta ct your provider .T. ...—aZ:m7 804a68a, respectively. This way, we are able to obtain the start¬ ing addresses and lengths of all nine variables. Next, we need to be able to decrypt the original shell script using only the binary as input. In she, before the shell script itself is encrypted, many other pieces of information are encrypted. Furthermore, the RC4 implementation maintains state between encrypting and decrypting each individual piece of information. This means that the order in which she encrypts and decrypts information must be maintained. Failure to do so results in illegible text. To extract the original shell script, we need to perform several decryptions. For this step, I wrote a small program called deshc, using the existing code from one of the intermediate C files. The program reads two files as its input, the binary executable and an input file that specifies the array lengths and addresses, deshc executes the following four steps: Listing 4. The output of obj dump -d pub. sh . x shows information needed to find the encrypted script. Lines in parentheses were added. 8048e52: : 68 28 01 00 00 push $0x128 (Length of encryption key) 8048e57: : 68 40 a5 04 08 push $0x804a540 (Key address) 8048e5c e8 17 fb ff ff call 0x8048978 8048e61 83 c4 10 add $0x10,%esp 8048e64 83 ec 08 sub $0x8,%esp 8048e67 6a 08 push $0x8 (Length of shll) 8048e69: : 68 72 a6 04 08 push $0x804a672 (shll address) 8048e6e e8 a0 fb ff ff call 0x8048al3 8048e73 83 c4 10 add $0x10,%esp 8048e76 83 ec 08 sub $0x8,%esp 8048e79 6a 03 push $0x3 (length of inlo) 8048e7b: : 68 8a a6 04 08 push $0x 8048e80: : e8 8e fb ff ff call 0x8048al3 array length and the starting address. My first hint here is to look for all addresses that are within the data section (Listing 2) of the disassembled object code. Next, seek out all the push and mov commands in Listing 4. Addresses will be different for different scripts, but when you encrypt a few scripts and read the resulting C code, the patterns become familiar. The 804a540 address seems to correspond to the pswd vari¬ able, the encryption key. The length of the useful portion of the encryption key is represented by 0x128, or 296 in decimal form. Similarly, the next variables, shll and inlo, have useful lengths of 0x8 and 0x3 and starting addresses of 804a672 and Reads binary executable. ■ Extracts data section from the disassembled output. ■ Retrieves individual arrays based on input file. ■ Decrypts individual arrays in order, so that the RC4 state is maintained. Based on the objdump output, I have arrived at the follow¬ ing array lengths and addresses for the pub.sh.x executable: pswd 0x128 0x804a540 shll 0x8 0x804a672 inlo 0x3 0x804a68a xecc 0xf 0x804a68e Isto 0x1 0x804a6a4 chkl 0xf 0x804a6a6 opts 0x1 0x804a6be txt 0x76 0x804a6e0 All of these parameters are used in an input file to deshc, which then decrypts and prints the original shell script. Conclusion An approach to extract the shell source code successfully from she version 3.7 generated binary executable was demonstrated. The pub.sh script was used for illustrative purposes only. I have indeed tested the deshc program on executables that I did not create and without access to the source code or the original shell script. Francisco Garcia, the author of she, recently released ver¬ sion 3.8. It uses somewhat different data structures and improves upon the security of the previous version. Nevertheless, I believe that embedding the encryption pass¬ word within the binary executable is dangerous and prone to extraction as discussed in this article.@ Nalneesh Gaur, CISSP, ISAAR works at Diamond Cluster International as a BS7799 Lead Auditor. 401 OCTOBER 2005 WWW.LINUXJOURNAL.COM servers ► direct- More products, better service, guaranteed. GO STRAIGHT TO THE SOURCE! Introducing the Servers Direct Blade System with the power of Intel® Xeon™ Processor Increased computing power in a smaller footprint and simplified maintenance help you expand your enterprise solution to meet the most intense application demands. SDB-1100H Servers Direct Blade System Featuring a Server Direct Server Compute Blade powered by dual ® Xeon™ 800FSB Processors. Benefit of using Servers Direct Blade System: •More power, bandwidth, and processing performance to meet the demanding requirements of departmental workloads • Deliver world-class performance for peak server workloads •Future 64-bit-enabled applications •High performance small form factor SCSI hard drives (RAID 1 with 2 HDDs) •Ethernet I/O for demanding, data-intensive applications •Ability to easily add hotswappable SCSI hard drives, and additional Ethernet, or Fibre Channel I/O for increased application performance (RAID 1E requires use of the HDDs on the SBX82 as well as the HDDs on the SBESCSI) 1U Xeon Entry Level Server SDR-1300T Highest performing with Dual Xeon 800MHz. Excellent with general purpose applications and provide the most power. • Intel Xeon Processor 2.8Ghz with 800FSB1 MB Cache (Dual Processor Option) •Intel Extended Memory 64Technology • 1U Chassis with 420W power supply •Supermicro server board with Intel® E7320 (Lindenhurst VS) Chipset •Kingston 512MB DDR333 ECC Reg. RAM (2x256MB) • 1 pc x Seagate 80GB SATA 7200RPM hard drive • 2 x 1" Hot-swap SATA drive bays •Integrated ATI Rage XL SVGA PCI video controller •2x Intel® 82541 Gl Gigabit Ethernet Controllers •2x SATA Ports via 6300ESB SATA Controller RAID 0,1 Supported $999 2U Xeon Processing Server SDR-2103T High-density 2U platform optimized for performance and flexibility; ideal for Web hosting, data center, terminal services and High Performance Computing (HPC) • Intel Xeon Processor 3.0Ghz with 800FSB1 MB Cache (Dual Processor Option) •Intel Extended Memory 64Technology • 1U Intel Chassis with 700W PFC power supply •Intel® Server Board SE7520JR2 •Kingston 512MB DDR333 ECC Reg. RAM (2x256MB) •6pcs xSeagate 160GB SATA/150 W/ncq 7200rpm 8MB Cache •Intel SRCS16 6Channel SATA RAID Controller Card •Integrated ATI Rage XL SVGA PCI video controller •2x Intel® PRO/1000 MT Server Network Connections (Intel® 82546GB controller) $3,499 5U File Server SDR-5301S Outstanding performance, excellent data protection, and advanced management for departmental servers. • Intel Xeon Processor 3.0Ghz with 800FSB1 MB Cache (Dual Processor Option) •Intel Extended Memory 64Technology •Intel SC5300LX Chassis with Redundant 730W Power Supply •Intel server board w/lntel® E7520 (Lindenhurst) Chipset •Kingston 1024MB DDR400 ECC Reg. RAM (2x512MB) •Adaptec 2200S SCSI RAID Controller Card •Include 6-Drive SCSI Hot-Swap Cage Kit • 6 x Seagate 36GB SCS110K RPM U320 SCA hard drive •ATI Rage XL SVGA PCI video controller with 8MB of video memory •Dual Intel® PR0/1000 Server Network Connections $4,999 3U Clusterable SATA SAN Nodes SDR-3303T Provides a flexible, price and performance advantages to the storage needs of Small to Medium Business (SMB) market segment. •Intel 3U SAN Storage Enclosure w/700W Redundant Power Supply •Integrated Intel SE7501HG2 Server Board •2x Integrated SAN Intel Xeon 3.06Ghz/533FSB Processor •Integrated SAN 2x256MB Compact Flash Memory Cards w/SAN Mgt Software •3 X Integrated Intel SRCS16 6-Channel SATA RAID Cards •Kingston 1024MB DDR266 ECC Reg. RAM (2x512MB) • 16pcs x Western Digital WD2500SD RAID SATA •Integrated SAN Management Software & Storage System $13,599 Big business power, small business price tag Your business requires solid server solutions. With Servers Direct server systems based on the Intel® Xeon™ Processor, you can count on high availability, maximum efficiency and proven performance to help you meet your business reliability requirements. 1 . 877 . 727.7127 sales@serversdirect.com Development of a User-Space Application for an HID Device, Using libhid When it's time to get a new device working on Linux, every piece of information helps, whether it's reading the hardware documentation, snooping data, reading sample code or even running utilities on a non-Linux OS. by eoin verling T he Matrix is a USB bill validator, sometimes known as a note reader or bill acceptor, made by Validation Technologies International. The bundled software was developed for Microsoft Windows, but fortunately the device comes with low-level technical documentation that defines device-specific aspects, such as flow control, status bytes and local status LEDs. The device is a Human Interface Device (HID), as iden¬ tified by an enumeration process upon connection. The Windows device manager reports the device as such, as does usbfs on Linux. This article is specific to this particu¬ lar HID device, so including all of its code probably is unnecessary, but it should provide help for developing other HID-class devices. After some initial research, I decided to develop user-space code using an in-development library called libhid, which pro¬ vides a cross-platform way to access and interact with USB HID devices, libhid is implemented on top of libusb, so it does not depend directly on the kernel’s USB support. Another option for driving the Matrix is to use libusb directly, but doing so would be re-inventing the libhid wheel. A third option is to implement the Matrix driver as a kernel module, but it would incur the large overhead of learning kernel particulars. This option also would render the code platform-specific. Device descriptor 1 Siring descriptor f Report descriptor Physical descriptor Figure 1. A USB device's descriptors, stored in its ROM, hold information about it. sticks and force-feedback game controllers. Also included in the HID class are devices that may not require human interac¬ tion but do provide data in a similar format to HID-class devices, such as bar-code readers and, in my case, the Matrix note reader. Information about a USB device is stored in segments of its ROM called descriptors. A diagram of the descriptor structure is provided in Figure 1, where an overall view of the hierarchy can be seen. When a USB device is attached to a USB bus, an enumeration process takes place that equates to the descriptors on the device being read into memory. Information about an HID-class device is contained in its HID report descriptors. I plugged the device in to the Linux box in order to read the descriptors and monitor the device, the machine and the com¬ munications. I did this to try to get as much information as possible so I could have a better understanding of how to write code for the device. A key component of these report descriptors is the usage information, which is defined in the USB HID Usage Tables (see the on-line Resources). Usage values describe three basic types of information about the device: ■ Controls—information about the state of the device, such as on/off or enable/disable. ■ Data—all other information that passes between the device and the host. ■ Collections—groups of related controls and data. Investigation USB devices are categorized into device classes. A modem is in the communications class, and a speaker falls into the audio class. The HID class mainly consists of devices that people use to control computers. Examples of HID devices are mice, joy- Taken together, the usage page and usage number define a unique constant that describes a particular type of device or part of that device. For example, on the Generic Desktop usage page (page number 0x01), usage number 0x05 is a game pad, and usage number 0x39 is a hat switch. 421 OCTOBER 2005 WWW.LINUXJOURNAL.COM Because my device is unique—it isn’t a mouse, joystick or something commonly found in the examples of HID-class devices—the usage page is set to 65,440, which is a vendor- defined value. In comparing outputs of lsusb for other HID- class devices, they all had a defined usage page, such as Generic Desktop Controls or Game Controls. Because libhid still is in development, few previous examples of code are available to browse for reference. My work was much like an exploratory investigation. On Linux, with a standard Debian 2.6.9 kernel and usbutils, I was able to see that Linux recognises the device as a USB HID device, blnterfaceClass = HID, and loads the hiddev ker¬ nel module. This module, or piece of kernel code, is a generic driver for HID devices. It is not specific to our needs—it main¬ ly is used for mice, joysticks and the like—so it needs to be detached from the device or disabled (see the Communicating with the Device section). The device, like all USB devices, is enumerated upon connection to the USB bus. So looking at the output of lsusb -vvv, run as root, for more information is helpful in determining what the device capabilities are. lsusb parses the usbfs filesys¬ tem into a more readable format: [sample lsusb -vvv] Bus 001 Device 004: ID 0ce5:0003 Device Descriptor: idVendor 0x0ce5 idProduct 0x0003 Configuration Descriptor: Interface Descriptor: bNumEndpoints blnterfaceClass blnterfaceSubClass blnterfaceProtocol 1 3 Human Interface Devices 0 No Subclass 0 None HID Device Descriptor: Report Descriptor: (length is 32) Item(Global):Usage Page,data=[0xa0 0xff]65440 (null) Item(Local ):Usage, data= [ 0x01 ] 1 (null) Item(Main ):Collection, data= [ 0x01 ] 1 Application Item(Local ):Usage, data= [ 0x03 ] 3 (null) Item(Global):Logical Minimum,data=[ 0x00 ] 0 Item(Global):Logical Maxi mum,data=[ 0xff ]2 5 5 Item(Global): Report Size, data= [ 0x08 ] 8 Item(Global): Report Count, data= [ 0x05 ] 5 Item(Main ): Input, data= [ 0x02 ] 2 Data Variable Absolute No_Wrap Linear Preferred_State No_Null_Position Non Volatile Bitfield Item(Local ): Usage, data= [ 0x05 ] 5 (null) Item(Global):Logical Minimum,data=[ 0x00 ]0 Item(Global):Logical Maxi mum,data=[ 0xff ]255 Item(Global): Report Size, data= [ 0x08 ] 8 Item(Global): Report Count, data= [ 0x05 ] 5 Item(Main ): Output, data= [ 0x02 ] 2 Data Variable Absolute No_Wrap Linear Preferred_State No_Null_Position Nonvolatile Bitfield Item(Main ): End Collection, data=none The above output—some of the information has been omit¬ ted—follows the hierarchy depicted in Figure 1. Some values of note are: ■ idVendor and idProduct—unique identifiers for all USB devices, used for identifying and accessing the device in code. ■ bNumEndpoints—lists the number of endpoints available in a device. This value actually means the number of endpoints in addition to the default endpoint, endpoint 0, available in every USB device. 1 QU IER 42 nerator ride network O Free tech support LHj O Redundant UPS and ge O Nations 2U 256kbps -80GB $60/mo. 4U or Mid-tower 256kbps -80GB $80/mo. 1/4 Rack 512kbps (14U) -165GB $200/mo. 1/2 Rack 1 mbps (28U) -330GB $350/mo. www.layer42.net All prices include 100Mbps port, Firewall, 24x7 Monitoring and DNS hosting 408-450-5740 2336-F Walsh Ave„ Santa Clara, CA 95051 WWW.LINUXJOURNAL.COM OCTOBER 20051 43 ■ blnterfaceClass—the value that determines that a device is an HID-class device. ■ blnterfaceSubClass—the subclass of a device, in this case, HID. For example, the boot interface subclass of the device must be bootable or available to the BIOS, such as a mouse or keyboard. ■ blnterfaceProtocol—the protocol used. Possible values are 0 for none, 1 for keyboard or 2 for mouse; additional informa¬ tion is available in the HID spec. Communicating with the Device A block diagram depicting the flow of control of data is shown in Figure 2. It may help in picturing where your code fits in with respect to the libraries and the device. From my investiga¬ tion, I know that control messages periodically are written by way of the control pipe, and interrupt reads are made through endpoint 0. The control pipe is used for three tasks: receiving and responding to requests for USB control and class data; trans¬ mitting data when polled by the HID class driver, using the Get_Report request; and receiving data from the host. The Interrupt pipe is used for two tasks: receiving asynchronous, or unrequested, data from the device and transmitting low-latency data to the device. What is the 5-letter word for “an Inexpensive Compact Panel PC”? The PPC-E5 is the Solution that Fits ! Setting up a Linux Panel PC can be a Puzzling experience. However, the PPC-E5 comes ready to run with the Operating System installed on flash disk. Apply power and watch the Linux X Window Graphic User Interface appear on the vivid color LCD. Interact with the PPC-E5 using the responsive integrated touchscreen. Everything works out of the box, allowing you to concentrate on your application rather than building and configuring device drivers. Just Write-lt and Run-lt. Starting at $995. clilAC, inc. Phone 618 - 529 - 4525 Fax 618 -457 -0110 2390 EMAC Way, Carbondale, Illinois 62901 World Wide Web: http://www.emacinc.com myDri ver libhid I ibusb Ctrl pipe int pipe device Figure 2. The new driver uses libhid, which depends on libusb. The kernel has a DEBUG feature that can be activated in order to log extra information about what is happening when communicating with the device. To do this, a file in the kernel source needs to be modified. In the /usr/src/linux/ drivers/usr/input/hid-core.c file, these two lines need to be changed from: #undef DEBUG #undef DEBUG_DATA to: #define DEBUG #define DEBUG_DATA The module needs to be recompiled and installed. Once this is done, the modules should prove helpful in determining whether your code is working and doing what you expect. Sample code containing some helpful comments comes with libhid. The file test_libhid.c in the libhid/test directory is a good place to start writing code for the device. Below is a snippet of that code, along with some more explanation of the functions; details are omitted for brevity: HIDInterface* hid; hid_return ret; HIDInterfaceMatcher matcher = { 0x0ce5, 0x0003, NULL, NULL, 0 }; ret = hid_force_open(hid, 0, &matcher, 3); int const PATH_LEN = 2; int const PATH_IN[2] = { 0xffa00001, 0xffa00003 }; int const WRITE_PACKET_LEN = 8; char write_packet[8] = { 0x04,0x7f,0x7f,0x00,0x02,0x00,0x00,0x00 }; int const READ_PACKET_LEN = 5; char read_packet[5] ; ret = hid_set_output_report(hid, PATH_IN, 441 OCTOBER 2005 WWW.LINUXJOURNAL.COM PATH_LEN, write_packet, WRITE_PACKET_LEN); ret = hid_interrupt_read(hid, USB_ENDP0INT_IN+1, read_packet, READ_PACKET_LEN, 0 ); The first thing to do is identify the particular device we want to talk to. This is done with the HIDInterfaceMatcher call simply by entering the vendor ID and the product ID. These two identifiers are all that is required to identify any USB device. If you have more than one identical device, it is possi¬ ble to separate them by serial number, that is, two Matrix note readers would have the same vendor ID and product ID but different serial numbers. The HIDInterfaceMatcher call can do this; see the comments in the test_libhid.c file. After some variable setup, the next step is to detach the kernel driver from the HID device. Upon insertion of the HID device, the kernel usually loads the usbhid module, which we don’t want. We do have a few options, however, for unloading it or for not loading it in the first place. One such way is to enter this command: Figure 3. Understanding a device: one way to browse the available nodes of the HID tree is to use the SystemSoft HID Browser. 2. A Windows application available from Amaud, one of the libhid authors, also parses the usage tree and produces a nice GUI output, as shown in Figure 3. 3. By parsing the output oflsusb -vvv, run as root, it is possi¬ ble to parse the tree manually to determine the path. This process is explained in the comments of test_libhid.c code. root@localhost #> modprobe -r usbhid When the hid_force_open function runs, it attempts, n times, to detach the device before it fails. The device is now free from any control, so our code now “opens” the device. As with any USB device, it is necessary to send control information to the device to activate it. This information must be sent periodically in order for the device to remain active. If the control poll stops, the device deactivates after a cer¬ tain timeout. Writing to a device requires the HID usage path and its length, plus a packet and its length. To find this out, we need to parse the usage tree—the output of 1 s u s b - v v v—and obtain the path to the interface we want. As with everything else, there are various meth¬ ods for determining the path. At this stage, a lot of time was spent determin¬ ing what path to write to, and a number of tools are helpful here, such as: 1. The test_libhid.c code: when the correct vendor and product ID are entered in the code, the function hid_dump_tree, which uses the MGE hidparser (see Resources), which parses the HID usage tree and places the available usages at its leaves, outputs the available paths. $119 qty 100 200 MHz ARM9 10/100 Ethernet PC/104 bus TS-7200 ARM9 Single Board Computer $ 149 qty 1 * 32 MB SDRAM (64 MB optional) a 8 MB Flash (16 MB optional) * Compact Flash » 10/100 Ethernet * 2 USB ports * 20 Digital I/O » 2 Serial Ports Options: Shown with optional Compact Flash 9i Boots Debian stable from Compact Flash x Boots TS-Linux from on-board Flash " RS-485 " 8ch 12-bit A/D * RTC (battery-backed) We use our stuff. Visit our TS-7200 powered website at 9i Many x86 and ARM based SBCs and peripherals available 9i Call for custom designs (480) 837-5200 www.embeddedARM.com WWW.LINUXJOURNAL.COM OCTOBER 20051 45 From the above methods, we now have a path value we can use for the hid_set_output_report. Once we know where to write to, it’s a matter of what to send. This information should be in the technical documentation that comes with the device, and it can be verified with the USB-sniffing tools. As with the particular device I was using, verifying the format of a packet with the sniffing tools turned out to be important, as the infor¬ mation in the documentation didn’t match what the sniff log reported (see the Snooping section). Once the control message or output report is sent, we can start to read from the read pipe, endpointO. The function needed is an interrupt read function. It already exists in libusb, but a corre¬ sponding libhid function doesn’t. The developers of libhid simply hadn’t come across a device that required it yet, so I studied the format of the other functions and implemented it appropriately. I also added a new error code to the existing list. These additions are being considered for inclusion in the latest version of libhid. At this stage, once the interrupt read value is stored, I then parse this value, as per the Matrix documentation, to display the results to the user. For this device, that equates to informa¬ tion such as, “A ten-euro note has been inserted” or “The cash box is disconnected” and other such device-specific informa¬ tion. The details are unnecessary for the purposes of this arti¬ cle, but if anyone requires this detail, feel free to contact me. This process is repeated for as long as the driver is running. We must keep polling the device to keep it active. There is a status LED on the device that turns green when the device is active and remains orange when inactive. The goal for quite some time was to make the little light go green. Snooping Snooping can be done with a number of utilities. This is where I learned about the discrepancies between what the Matrix doc¬ umentation says and what actually happens: [5037 ms] <<< URB 647 coming back <<< -- URB_FUNCTION_CONTROL_TRANSFER: PipeHandle = 8180C814 TransferFlags = 00000002 (DIRECTI0N_0UT) TransferBufferLength = 00000005 TransferBuffer = 92al37ed T ransferBufferMDL = fe9876e8 UrbLink = 00000000 SetupPacket 00000000: 21 09 00 02 00 00 05 00 [5038 ms] <<< URB 645 coming back <<< -- URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER: PipeHandle = fe9876a0 [endpoint 0x81] TransferFlags = 00000003 (DIRECTION_IN) TransferBufferLength = 00000005 TransferBuffer = fefeef08 TransferBufferMDL = 81al8f48 00000000: 00 20 00 00 la UrbLink = 00000000 [5038 ms] >>> URB 648 going down >>> -- URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER: PipeHandle = fe9876a0 [endpoint 0x81] TransferFlags = 00000003 (DIRECTION_IN) TransferBufferLength TransferBuffer TransferBufferMDL UrbLink 00000005 fefeef08 00000000 00000000 From the snoop log, we see the control message sent to the device at the start, followed by a series of interrupt reads. According to the documentation, “The Host sends [a] poll to request information from Matrix at a periodic rate. Matrix answers to the poll and reports all the happening events.” So, my interpretation of this was to send periodic control write messages to the device and read the responses from the interrupt endpoint. Also according to the documentation, the format of the write message is five bytes in length, so with this information, I used the test_libhid.c file included with libhid to see what happens. I found that functions within libhid give error codes if they fail and that the /var/log/messages file, with the extra DEBUG informa¬ tion from the modified kernel file, reports useful errors. Upon closer inspection of the snoop log, I saw that the control write was, in fact, eight bytes in length. See SetupPacket in snoop log output. The five bytes described in the documentation seemed to represent the first five bytes of the packet, and the last three bytes seemed to be padding. That is, changing these last three bytes doesn’t seem to affect the operation. Subsequent error-free testing, with the packet set to eight bytes, confirmed that the documentation had been misleading. Conclusion In terms of where to start with this project, I found the mailing list for libhid to be helpful. The libusb mailing list also provid¬ ed guidelines. The Linux usbutils are quite useful in determin¬ ing what interfaces are available on the device and the meaning of the descriptors. The libhid source code, still in constant development, also is a source of help. Because the code constantly is being devel¬ oped, it is a good idea to keep an eye on the Subversion reposi¬ tory for changes, including documentation changes such as helpful comments in the code. Acknowledgements Special thanks to Charles Lepple and Arnaud Quette, the origi¬ nal authors of libhid, and also to Martin F. Krafft, who later joined and led the rewrite. They all provided me with a lot of help, and without them I certainly wouldn’t have gotten my lit¬ tle light to go green. Also, thanks to my supervisor, Dr Paul O’Leary, at WIT, for his encouragement and analytical skills. It always is good to have an experienced pair of eyes to guide me in the right direction. libhid uses the HIDParser framework made available by MGE. Resources for this article: www.linuxjournal.com/article/ 8275.«# of ceol agus Eoin Veiling (everling@theverlings.conn) gualified in 1998 and has been a sysadmin since. He cur¬ rently is undertaking a research Master's in parallel computing at Waterford Institute of Technology, Ireland. There's nothing he likes better than a bit craic! 461 OCTOBER 2005 WWW.LINUXJOURNAL.COM The Only Silo "Edison's light bulb was important not because he was the first with the idea; as many as ten others envisioned similar schemes. Rather it was significant because he conceived not just a bulb but a whole electrified world."—Teresa Riordan, US News BY DOC SEARLS W e’ve been fighting closed and proprietary soft¬ ware for a long time now. And we’ve had lots of success—enough, I think, that we need to move to the next stage: to the marketplace. We can see the problem when we look at how many closed systems have open foundations: Google and Amazon on Linux, Apple’s Mac OS X and Yahoo’s search infrastructure on BSD. Also, countless closed Web habitats served up by Apache. Am I being unfair here? Perhaps a little. You can’t be open in every possible respect, right? Some stuff needs to be locked down or closed off. Customer data, future product plans, trade secrets and “secret sauces” of one kind or another. But those aren’t the issue. The real issue is silos: closed habitats that serve as private marketplaces that lock customers in and competitors out. Dick Hardt of Sxip Networks gives a killer talk about “Identity 2.0”. As Dick puts it, Identity 1.0 is a province of walled gardens. Amazon, eBay, Flickr and Skype are all walled gardens. They may be lovely places to hang out, but they are also enclosed and private market spaces, as false in their own way as the faux Venetian canals and Parisian streets in Las Vegas hotels. What makes them most different from closed systems of the traditional sort is not a lack of interoperability—often they have that—but the lock-in of personal data. You can’t take and then it hits you:// LINUX ISN’T A POLITICAL STATEMENT. IT’S A PROFIT STATEMENT. Novell. find out more at novell.com ©2005 Novell, Inc. All rights reserved. Novell is a registered trademark of Novell, Inc. in the United States and other countries. WWW.LINUXJOURNAL.COM OCTOBER 2005147 your eBay reputation, or the business you’ve built inside eBay’s walled garden, over to Amazon. Even trivial data, such as your Skype contact list, isn’t portable. It’s locked inside a space that is not your own. To hear Dick describe it, Identity 1.0 is barely past medieval. It’s a country of duchies and city states. But since we’re so used to it, we can barely think outside its walls. Yet that’s where we belong, he says. The world we want—the Identity 2.0 world—is one of independent actors: free-range customers, conducting business and building relationships in ways that each individual controls and that work with many different vendors. The problem with the walled garden metaphor is that all the familiar examples are native to the Web. Silos, on the other hand, are everywhere, both on the Web and off. Nearly every familiar business category you can name—banking, hospitality, retailing, commercial aviation, car rental...even office equip¬ ment, such as copiers and printers—is a forest of silos. Take airlines. I am a registered frequent flyer with United, American, Delta, America West and Southwest. Yet the only common way I can relate to any of the five is money. None of my data in the United silo is available for my dealings with American or Delta. We’ve lived in a world packed with silos for so long that we now confuse them with a free marketplace. We left DG and formed a new systems integration company in Raleigh called BAS, for Business Application Systems. BAS’s goal was to produce what they called “machine- independent software”. To be independent, Earl explained, you needed software that was independent of every hard¬ ware vendor’s silo. One day, early in my company’s relationship with BAS, Earl explained the ideals of machine independence, all of which are familiar to anybody acquainted with open source today. (Although, naturally, BAS’s code was proprietary.) When he was done with his rap, my business partner asked the impolite question, “So how do you make your money?” “We’re whores”, Earl replied. “We walk the streets with the rest of them.” His point: they had no choice—except among silos. (BAS, it turned out, bet on the wrong silo: Texas Instruments’ DS990s.) Silo was just one container metaphor kicking around in those days. Others were smokestacks and stovepipes. Today those metaphors have fallen behind silo in popularity. I sus¬ pect that’s because silos are completely contained. Unlike smokestacks and stovepipes, they don’t have an opening at the top. The prototypical office building is a silo of sorts. With its security systems, its employee and visitor badges, it comprises what David Weinberger calls “Fort Business”: The world we want—the Identity 2.0 world—is one of independent actors: free-range customers, conducting business and building relationships in ways that each individual controls and that work with many different vendors. actually believe that a choice of silos comprises all the con¬ ditions required for a free market. We can see how limited this is when we look at the market category we call com¬ puters. A quarter century ago, we thought the category was free and open because we had a choice of silos from IBM, Digital, Data General, Wang and HR We thought the same way about networks when our choices were OmniNet, WangNet, IBM Token Ring, Sytek, Corvus, 3Com and Ungermann-Bass. I remember a long conversation I had with Ralph Ungermann about how his company’s goods were “open” because they interoperated with other networks. In a rela¬ tive sense, they may have been. But the market was essen¬ tially a field of silos. What he offered was inter-silo-oper- ability. Good for its day, but nothing like the Net that was to come—and which didn’t come from any one vendor at all. I remember Earl Gillmor talking about silos, way back in 1980. Earl enjoyed a small measure of notoriety as a member of a splinter group at Data General that lost “the shoot-out at the Holiday Inn” in Durham, North Carolina— an event immortalized in the early pages of Tracy Kidder’s book, The Soul of a New Machine. After the shoot-out, Earl This fort is, at its heart, a place apart. We report there every morning and spend the next eight, ten or twelve hours inacces¬ sible to the “real” world. The portcullis drops not only to keep out our enemies, but to separate us from distractions such as our families. As the drawbridge goes up behind us, we become businesspeople, different enough from our normal selves that when we first bring our children to the office, they’ve been known to hide under our desks, crying. Within this world, the Web looks like a medium that exists to allow Fort Business to publish on-line marketing materials and make credit-card sales easier than ever....The Web isn’t primari¬ ly a medium for information, marketing or sales. It’s a world in which people meet, talk, build, fight, love and play. In fact, the Web world is bigger than the business world and is swallowing the business world whole. The vague rumblings you’re hearing are the sounds of digestion. The change is so profound that it’s not merely a negation of the current situation. You can’t just put a big “not” in front of Fort Business and say, “Ah, the walls are coming down.” No, the true opposite of a fort isn’t an unwalled city. It’s a conversation. 4 8 ■ OCTOBER 2005 WWW.LINUXJOURNAL.COM David wrote that more than six years ago, in Chapter 6 of The Cluetrain Manifesto. We still aren’t having the con¬ versation required to bring the walls down. True, there are some significant conversations growing out of employee blogs. For example, nothing has done more to bring down Microsoft’s walls than interaction with outsiders by Robert Scoble, Kim Cameron (a subject of last month’s column) and about 2,000 other blogging Microsoft employees. But the problem isn’t communication. It’s the structure of markets themselves. I’m not talking about structure in an architectural sense, but in a deeper way that’s more like geology. Because the Internet is geological, not just archi¬ tectural. It has a nature that goes deeper than whatever structures private efforts can provide. But that nature is hard to see when your frames of reference are closed and proprietary. Like many in the Linux community (including my good friend Eric Raymond), I have strong Libertarian sensibili¬ ties. I understand the liberating advantages of private prop¬ erty to societies and their economies. Ownership matters, and ownership works. But we in the Free Software and Open Source communities also know there are some things that are beyond the scope of ownership and the control ownership naturally implies. Earth below the crust is as beyond the practical scope of ownership as the weather and the stars. Yet they provide us with services so fundamental we couldn’t live without them. One of those services is a deep and easily ignored context for property: gravity. Real estate would be meaningless without the gravitational pull provided by a mass we’ll never see. The Net’s geology is like that. I’ve written many times about the NEA nature of the Net, and of all free and open-source software: Nobody owns it, Everybody can use it and Anybody can improve it. The same applies to markets, and it’s time we started improving the ones we’ve got, by putting silos in a context that makes clear their limited advantages. The Supreme Court missed a chance to do that with the Brand X case. In a 6-3 decision that was handed down on June 27, 2005—the same day as Grokster, which is a big reason why not much of a fuss was made about it—the Supremes upheld a 2002 FCC ruling that classified cable broadband as a deregulated “information service” rather than a “telecommunications service”. Unpacked, that means the cable and telephone companies can (and will) be exclu¬ sive Internet service providers. Independent ISPs like Brand X and Earthlink, which don’t own physical connec¬ tions to homes and businesses, are out of luck if the cable and phone companies want to keep captive customers to themselves. More important, the FCC’s understanding of the Internet achieved the stature of law with the Brand X decision. That understanding is basically feature-rich broadcast. It’s a concept of service anchored on the supply side of the highly asymmet- and then it hits you:// YOU’RE WORKING WITH LINUX 24/7 IN EIGHT TIME ZONES. AND SO ARE WE. Novell. find out more at novell.com ©2005 Novell, Inc. All rights reserved. Novell is a registered trademark of Novell, Inc. in the United States and other countries. WWW.LINUXJOURNAL.COM OCTOBER 2005149 What makes Linux so different, and so successful, is that it's not designed as a silo. rical distribution system the FCC has governed for most of the last century. Former FCC Chairman Michael Powell, in a speech at the VON (Voice On the Net) conference one year ago, said, “To realize the innovation dream that IP communications promises, however, we must ensure that a willing provider can reach a willing consumer over the broadband connec¬ tion.” He generously acknowledged “the importance of consumer empowerment” and rights such as “Freedom to Access Content”, which he explained with “Consumers should have access to their choice of legal content.” Generous as that may have been, it was no less top-down than anything owned by Rupert Murdoch. Nowhere did he acknowledge the Net’s most profound commercial grace: supporting the ability of people to go into business, and to do business, with anybody they please, anywhere. Thomas Madsen-Mygdal, a young serial entrepreneur in Denmark who hosts the delightful annual reboot conference in Copenhagen, recently told me he likes and appreciates Flickr— the on-line photo gallery phenomenon that has taken the world by storm (and which was built on LAMP)—but that it has “lock-in” issues: I don’t mean “a total lock-in” in the traditional BigCo IT sense. More like that if open data standards existed, the col¬ lective value would be in the commons—not on one photo sharing site. The London bombings wouldn’t be about the “flickr tag”, but about the “photo tag” or just the tag —which in my book is much more aligned with our values and the society we want to create. Thomas’ higher-level concern is that “we’re selling out on values of open standard and decentralization”. What Tim O’Reilly calls the “architecture of participation”, Thomas says, is turning into something that is “based on silos” in practice. So, he adds, “I’m gonna try and push some open standards in the photo sharing space to level the competition.” He’ll do that, he says, by “dividing what is the commons from what is the product. That way, thousands of photo sharing products can create a collective value that’s a lot greater. Competition will be on the product side, rather than on who aggregates most of the commons.” Thomas’ site, 23people.com, is open for beta. In September 2005, O’Reilly put on its second Web 2.0 conference. (Shouldn’t they call it Web 2.1?) In September 2004, Tim O’Reilly described Web 2.0 as “the Internet as a platform”. Then he added: We heard about that idea back in the late 1990s, at the height of the browser wars, but that turned out to be a false alarm. But I believe we’re now starting the third age of the Internet—the first being the telnet-era command-line Internet, the second the Web—and the third, well, that tale grows in the telling. It’s about the way that open source and the open standards of the Web are commoditizing many categories of infrastructure software, driving value instead to the data and business processes layered on top of (or within) that software; it’s about the way that Web sites like eBay, Amazon and Google are becoming plat¬ forms with rich add-on developer communities; it’s about the way that network effects and data, rather than software APIs, are the new tools of customer lock-in; it’s about the way that to be successful, software today needs to work above the level of a single device; it’s about the way that the Microsofts and Intels of tomorrow are once again going to blindside established players because all the rules of business are changing. That was a lead-in to the Web 2.0 conference. After the conference, in an interview with Richard MacManus, Tim said: I actually ran a couple of panels on this at our Open Source convention, a year and a half or two years ago—called “Open Data—Do We Need a Bill of Rights for Web Services?” We had people from Amazon, eBay and others trying to answer that question: what does it mean when we’re investing our on-line data in these sites? Will we end up with something like the Open Source movement because the companies have ended up locking in their users? ....But the actual data ownership is maybe less important, in some areas, than people think. When we talk about user- contributed data, we’re not just talking about my data prop¬ er (as in having your mail stored on Gmail or Yahoo! Mail or whatever). We’re also talking about a kind of content that users are contributing to a collective work. So for example, Amazon Reviews—people don’t really care about that in the same way. They’re not saying, “Oh I created that review and I want to be able to export it to Barnes & Noble as well.” They’re creating it in a particular context of that community. ....Despite what I’ve said...data lock-in absolutely should be a concern. I believe that data lock-in of various kinds is going to be one of the key tools of business advantage in the Internet era. I think that as companies realize this, they will figure out how to be evil—so to speak (to use Google’s terminology)—and I predict that we will in fact have some major battles in that area. As I said last month, one answer is to create ways to do what Drummond Reed calls “Company Relationship Management” (or CoRM), which should look far more interesting and useful to companies than their own Customer Relation Management (CRM) systems, which by 501 OCTOBER 2005 WWW.LINUXJOURNAL.COM nature have no view outside the company’s own silo. In fact, CRMs are one of the main ways companies maintain their silos. Another is to pay more attention to where the Net’s deep, almost geological market-making infrastructure comes from. It’s not from the physical cables that run to homes or from the “services” available exclusively from cable and phone compa¬ nies, but from the open protocols that define the Net’s environ¬ ment. It’s also not from fancy private services inside corporate walled gardens but from the raw building materials that make deploying those services so free and easy. Which brings us back to the L in the LAMP suite that makes possible the last phrase above. What makes Linux so different, and so successful, is that it’s not designed as a silo. Linux didn’t come from a silo, and it had no ambitions to be a silo. At one point, Linus talked about “world domination”, but his tongue was in his cheek—even if he was indulging in prophesy that would prove out in the long run. Linux was never a business. It was, and remains, a great way to build anything, to support anything, for anybody. That’s the fundamental virtue we need to fight for when we go to battle. Our battle, however, is not with the companies that use open code to build walled gardens and silos. Our battle is with the closed, top-down silo-oriented value system that has been with us since the dawn of the Industrial Age. It’s that lame old value system that prevents us from imagining how we can improve markets that nobody owns and any¬ body can improve. The best way to shed the old mentality is to embrace our customers and not only their money. Today the preponderance of inventiveness and productivity is out in the free world, in the hands of free-range individuals. Linus Torvalds is the prototypical example of one of those individuals. There are countless more like him, producing all kinds of goods, expressing all kinds of demand—much of which they are able to supply for themselves, as Linus did, and with the help of others, as the Linux community has done. In fact, the only silo that matters is the most fundamen¬ tal and indivisible unit in the marketplace, the individual. What we need is to create and support independence, not dependence. Work to free individuals, and to take advantage of what they do with that freedom, and you’ll have a winning strategy in the new marketplace we’re all making together.0 Doc Searls is senior editor of Linux Journal. and then it hits you:// LINUX WORKS WITH ANY PLAN. ESPECIALLY THE FY ’06 BUDGET. Novell find out more at novell.com ©2005 Novell, Inc. All rights reserved. Novell is a registered trademark of Novell, Inc. in the United States and other countries. WWW.LINUXJOURNAL.COM OCTOBER 20051 51 Fixing Web Sites with GreaseMonkey Who says "View Source" on a Web page has to be a read-only proposition? Re-mix your favorite Web sites by changing styles, adding and removing elements, and more, by nigel mcfarlane H ere’s a strange thing: hacking open source isn’t done only at midnight, in the spare room, hunched over the protocol analyser, the breadboard, source code control and some helpless device. No, sometimes it’s done inside a different crucible entirely: a public world of shameless posturing and self-promotion. A lurid and neon habitation of signs, shops, styles and stuff populated by the babble of conver¬ sations both informed and banal. It’s a place of great joy and great angst; a place of towering conservatism and the last bastion of the radical voice. Within it, a good hairdo or a radically cut legline can get you as far as a symbolic debugger, possibly even further. Devices they may be, but of a different cut entirely from those of hardware. Its denizens slip hyperactively in and out of view like character actors with coffee addictions and inspired agents. Of course, I refer to the World Wide Web. In this article, you learn how to code in a new way, a way that’s about changing media, not about changing programs. To enter this nightclub and experience the beat, you need the right gear, and the right gear is Mozilla Firefox and GreaseMonkey. Alfred Bester and William Gibson are waiting, so ready your Mojo and prepare for cyberspace insertion. But first, a bit of background. Web Pages as Open Source We tend to forget that the Web is open source, in a way. Some of the Web’s infrastructure, browsers and servers, is traditional open-source software, but the idea also applies to Web page con¬ tent. Appropriation of code is an everyday occurrence. Every day, Web developers and designers use the View Source browser fea¬ ture to appropriate (industry term: steal) code and design from other people’s pages. It was ever thus, and it remains so. Ideas and code are shared freely and often; it’s an art design sensibility. Most technical people have dabbled with Web develop¬ ment, and dabbling is an easy way to have a bad experience. The big three technologies—HTML, CSS and JavaScript— were riddled with bugs for many years after their inception. That’s the experience that probably looms large for early adopters who first tried it out in the 1990s and walked away in disgust. Cross-browser code? No, thank you. Fortunately, matters have improved tremendously as of late, and the Web is reviving as a technology platform. Better stan¬ dards support, more standards support and the decline of hoary old misgivings, such as Netscape Communicator 4.x and Internet Explorer 5.0, have left Web developers with a nearly clear shot at real portability, a shot frustrated only by the once shiny but now fairly rusty Internet Explorer. In 2005, the buzz is about Modern DHTML, Layout without Tables, Semantic Markup and Asynchronous JavaScript and XML (AJAX). Client-side Web development is coming back, and these are the things of which it’s made. This time, the Web is backed by pro¬ fessionals with formal Web training and veterans with ten years of experience. These people have their acts together, and it’s possible to say things about Web technology that are no longer drowned out by the static of incompatibility issues. Supporting and colonizing this trend is the Mozilla Firefox Web browser, and Mozilla technology in general. Of course, Mozilla is fully open source, as open as a religious movement can be, and so there’s plenty of room for experimentation. The critical bit of Mozilla and Firefox is its interpreted nature. On top of a big, bad, networked C++ rendering engine is a thin skin of JavaScript scripts and XUL, an XML dialect. This makes Mozilla a distant cousin to Emacs or Tcl/Tk, as it provides the whole Firefox user interface by way of interpreted code. By writing an extension, you can enhance this user interface and drop it in to thousands of willing people’s daily experience. Go to update.mozilla.org to see the endless possibilities made real by this system. Every variant hardware device requires Linux kernel driver support; every variant human expectation about user inter¬ faces requires a Firefox extension. That’s a lot of extensions. Grabbing GreaseMonkey GreaseMonkey is a Firefox extension (see the on-line Resources). You have to click on the link twice, once to trust www.mozdev.org and once afterward to install the extension. GreaseMonkey differs from the other extensions because it provides no specific user-interface enhancements of its own other than a configuration dialog box. Instead, it creates a macro-like scripting environment into which you put JavaScript scripts. Those scripts operate on Web pages that you specify. When such a page loads, your script goes to work on the page content, no matter who provided it. You’re intercept¬ ing a content provider’s content and modifying it before it hits you. No wonder GreaseMonkey’s been called “TiVo for the Web”. I wrote about page modification tactics in Rapid Application Development with Mozilla (Prentice Hall, 2004), but GreaseMonkey has moved that idea into the mainstream by supporting traditional Web-scripting techniques and by packag- 521 OCTOBER 2005 WWW.LINUXJOURNAL.COM i i Take charge. Win the battle and take control, right from your comfy chair. It’s easy to conquer La*/ '■"J. 4'' the challenges of managing serial devices in the data center with the CCM serial console manager.* When used with DSView® 3, AVWorks®, or industry-standard SSH/Telnet client software, you can remotely control servers, network gear, telco and power devices from a single interface. With proactive alerts and offline buffering you can remotely diagnose failed devices and reduce downtime without setting foot in the data center. Visit us at Avocent www.avocent.com/serialcontrol. And start looking for a new chair. The Power of Being There® * Recliner recommended, but not included. Avocent, the Avocent logo, DSView, AVWorks and The Power of Being There are registered trademarks of Avocent Corporation. © 2005 Avocent Corporation. ing it all up into a digestible product. For all Firefox extensions, you must restart Firefox com¬ pletely to finish the install. Use File—>Exit to do that safely. Bucket-loads of pre-existing GreaseMonkey scripts are available (see Resources). Before you get too excited though, note that such scripts are tied to one Firefox installation and have no effect on any server. On a Linux or UNIX box, such scripts might affect a large user population, but they’re primar¬ ily a personal thing. For those readers switched on to people problems, the broader implications should be obvious. Spotlight on LinuxJournal.com To see all this at work, in this article I hack the Linux Journal Web site with GreaseMonkey. My esteemed editor, Don Marti, even asked me to do this. A brave man indeed. [Maybe next time they'll invite me to the Web site meeting. — Ed.] Give me a hill and I’ll climb it. First up is a bit of scrutiny of the site due for surgery. Recall it’s www.linuxjournal.com, if you’re reading this in print. This is also the fun part; personal tastes differ, and for user-side drivers—which effectively is what GreaseMonkey scripts are—it’s entirely valid and professional to be picky and subjective. In Mozilla-land, dogfood means testing your fixed bugs for technical correctness, and catfood means test¬ ing your inventions against unreliable and subjective people who might spring in any direction. It’s all catfood here, and there’s no right or wrong. After reading this article, ZJ’s long-suffering site maintainer will likely glare at me venomously or perhaps change the site before this sees print. Design sensibilities, you see. Sorry mate, they made me do it. Hard-core engineers should look away; you might find this analysis distressing. On to the site. Here’s a handful of observations. ■ The site icon, which appears in the location bar and on the current tab if you use tabbed browsing is dinky and unin¬ spired. Oh well. ■ There’s advertising everywhere. ■ Linux Journal's supposed to be the granddaddy of technical journals in open source, excluding academia and profession¬ al bodies. Where’s that indicated? ■ The headings are red. What’s with red? I’m not in a hurry. On the plus side—my survival as a critic is at stake—the site has a robust three-column layout and is clean overall. Someone knows his or her stuff. Viewing the source, the layout is all done with CSS, so that’s relatively modern; many indus¬ try sites still pump out the worst HTML you can imagine. The excessive use of
tags shows that LJ is halfway through modernisation; there’s still some Semantic Markup work to go, where meaningful tags are used as content descriptors instead of the meaningless
. That update might improve the site’s search engine performance or so it’s claimed. The Right Tool for the Right Job Now, of the above personal observations, some are simple to rectify and do not require GreaseMonkey. If you dislike adver¬ tising, then the AdBlock extension is for you; there’s nothing, or at least little, to code. Similarly, for a long time, all browsers have supported user-specified stylesheets. If you install the ChromEdit extension, you can get at that stylesheet without having to grovel through the filesystem looking for it. Bring it up via Tools^Edit User Liles, click the userContent.css tab and start typing. To make headings blue, you might add: hi.title a { color : blue Mmportant; } @-moz-document domain(linuxjournal.com) { hi.title a { color : blue Mmportant; } } The first rule applies to all Web sites; the second is a Mozilla special that applies only to the Web site specified. Browser-specific is okay here, because we’re working purely on and in the client side. You can get a lot done in these stylesheets, especially if you know CSS well. You can hack the page’s layout to bits by reordering, hiding or floating columns and other content. All of these options are possible via GreaseMonkey as well, but GreaseMonkey is better suited to bigger stuff. In other words, don’t go to GreaseMonkey if page changes are easily solved with a stylesheet; it’s overkill. Lor this article, we’ll make one simple change. We’ll bring some gravitas to the site by replacing some content with fancy calligraphy drawn from another site. Illuminated Drop-Caps for Paragraphs The CSS :first-letter pseudo-selector lets you take an ordinary paragraph of text and make the first letter big, so that several lines of text flow around it. It’s a self-important feature and what we’re looking for. We simply could apply that feature, but most computers don’t have fancy medieval fonts installed. And, a big Times Roman letter L isn’t that exciting. It would be bet¬ ter if we could get the LinuxJournal.com Web site properly illu¬ minated, like the Book of Kells, with extra fancy calligraphy. Here are a couple of screenshots showing the before-and- after looks, taken on Windows XP Professional. This is a time¬ ly reminder that the user experience is what’s important here. It also emphasizes that open source means cross-platform when stated in Mozilla terms. Everything described in this article Figure 1. A Regular Linux Site 541 OCTOBER 2005 WWW.LINUXJOURNAL.COM tiw t« Wo |HbfdH4 Ttm \mp •#* - , & _ - 0® u. Figure 2. That's better. We didn't need monks to illuminate this manuscript, sim¬ ply a GreaseMonkey script. works identically on Windows, Macintosh, Linux and various obscure Mozilla platforms, such as Solaris. In the second screenshot, you can see that the first letter of each paragraph has been replaced with a fancy illuminated letter. Because I don’t have access to the back end of the LJ Web site, that’s something of a feat. In fact, these images come from the Australian National University’s Medieval Studies image server. I’ve used the thumbnail images only. It’s a bit naughty to serve up some other Web site’s images, and these images aren’t perfectly cropped, registered scans, but for the purposes of, well, illustrating a technique, they’ll do. Let’s hope some parsimonious old sod doesn’t take them down by the time you read this. Spinning Up the Script To make this embellishment work, you need a GreaseMonkey JavaScript script. To make such a script, proceed as though this were any other Web page project. I saved to local disk the LJ home page and then added this to the end of the section: