Tenor, The Context Link Engine

By: Kurt Pfeifle
Wednesday, April 13, 2005 12:05:56 PM EST
URL: http://www.linuxplanet.com/linuxplanet/reviews/5816/1/

Traditional File Managers--Good, Not-So-Good, and Best-of-Breed

Yesterday I updated the SuperDAT file, which the virus scanner needs, on my company notebook running Windows XP. I do this twice a week. The software does a complete HD scan overnight and in the morning I normally only check to see if it found some malware or if I still have a supposedly "clean" system. This time, I ran the scan during the day because I wanted to know the number of files it scanned. The virus scanner's result: 418,769 files on a 40 GByte disk, which is 88% full.

So, how many files do my Linux machines host?

Curious, I ran ls -lR / | wc -l on my main workstation running SUSE 9.1. Result: 1,635,315 files and directories. Those files take up about 50% of the 80 GB available on the hard disk. Admittedly, with KDE 3.4++/HEAD checked out from CVS and compiled from sources, this is not the amount of data a "normal" user may own. But then again, I do not collect MP3s, digital photos, or what-have-you. My old workstation (SUSE-8.0) spawns these results: 1,493,166 files spread across four disks with a combined capacity of 40 GB, all of which are filled up to over 90%.

The sheer number of files to handle poses many problems, slowdowns, and delays for me. I need to search way too often for a particular file or info. I suppose most users have similar experiences.

Compared to Windows Explorer, Konqueror meets my needs as a file manager much better than Explorer. Konqueror really shines when it comes to file management, even if it has four times the number of files to handle than Explorer. Of course, Konqueror has a lot of help for that, thanks to some other KDE goodies. To name just a few:

• An extremely useful File Open... dialog.
• A nifty File Preview feature.
• The ability to Split Windows into 2 or more panes
• Tabs, just like those seen in most modern web browsers, which compliment the split windows.
• Network Transparency.

The "network transparency" function is especially convenient. Others have touted the idea of "the network is the computer" for servers for a long time and KDE has delivered it to the common desktop and all its applications. Where else can you type in a webdavs://, a fish//: or a http:// URI to open or upload a file directly via the "Open File" or "Save As" dialogs?

The point is that Konqueror is very innnovative today, even if some tasks are too difficult to grasp by inexperienced users. It implements many ideas and features seen nowhere else and has expanded the term "file manager" beyond its original meaning. I expect Microsoft to improve their own file manager (Explorer) a lot by the time they release their next incarnation of Windows. I would not be surprised at all if they even copied some of the best stuff from Konqueror and KDE's File Open dialogs.

Limits Hit: Too Many Files in the Hierarchy

But while Konqueror fulfills my requirements as a traditional file manager today, does the very concept of a "file manager" make it easy to keep track a millions files? Will it still be appropriate once I have 100 million files on my hard drives? Maybe I should learn the word "delete" before this limit is hit, but don't tell me it will never happen: remember how in the '50s IBM estimated the "world market for computers" to be "no more than 5"?

The number of files we keep has been growing at a furious pace and there is no sign this is going to stop.

Today Konqueror still works on the hierarchical structure our file systems represent. If you have ever had to teach someone new to computers, you've probably had to explain to them the concepts of directories, subdirectories, files, paths--all deeply nested on the disk, right? And then you would move on to the notions of drive letters and personal directories. It's not the easiest set of concepts to grasp for a computer newbie. If you've ever done it, you probably know what I am talking about.

That hierarchical, file system-derived concept is not very natural to human brains when they are desparately searching for a certain piece of information. Our "thinking" ticks differently than a computer's logic. Even after so many years of using computers, I'm still not perfectly trained in remembering exactly where I saved which document, under what name. Where is the OOo file containing the consulting quotation I sent last year to that potential customer in Hamburg? My mind doesn't think in straight logical hierarchical paths. It instead goes: "Wait... The IT manager said he preferred a PDF, but their financial department (who had to approve it) wanted a *.doc file, or a print out on paper. It was just before Christmas, wasn't it? Oh right, I sent a PDF by e-Mail (to speed things up), but kept the .doc as a future template for similar quotations..."

Hierarchical file systems with hierarchically organized user interfaces do not supply any intuitive and contextual search paradigm. Future deskops need to become much more easy ("intuitive") to work with, including the interfaces they offer to users for finding a certain piece of information or a file.

Human minds work with "contexts." A pure file manager navigating the file system, based on the harddisk hierarchy of directories does work okay--if the number of files are not too many and if we can remember where we put things in that hierarchy. As I recall, the first MS-DOS versions typically did not even have 1,000 files installed. Later that number grew to 10.000 files, at most. It was during this era that the concept of the classical file manager was born. For this number of files, it even worked well! Organized into a hierachical structure, files could be found again using that same structure.

Ever Growing Data

Today, standard hard disk sizes offered in shops are typically 160 GB and cost €100 [US$129] . Next year, it will be 400 GB for the same price. Where do all those data and files come from that fill them up? I don't know really, but what I do know is that on all my computers the complete available disk space tends to be eaten up within a year. If I put in a new, additional disk with double the volume, it is full after 12 months. If I replace the original disk with one that is five times the size, it doesn't last longer than 52 weeks.

Despite Konqueror's position as the best-of-breed file manager, it is painful for me to handle all this data. And it doesn't look like it will become a more pleasant experience as the numbers keep growing. Plus, I am not the person that easily deletes digital stuff. (I also do not throw away old hardware, or books, or paraphernalia that quickly, either.)

Now, I know about and love locate, and I run updatedb regularly. And it is extremely cool to type "locate:sylvi jpg" directly into the Konqueror location bar or even the File Open dialog; it finds that image which shows Sylvi's excellent handling of her "aircraft" when she starts up for a long paragliding flight. The KDE KIO slave for "locate" saves me the effort to first type into a Konsole window locate sylvi|grep jpg and then puts the result into the file manager. But even this is not enough.

Time for Tenor: Context Storage and Reuse

So the idea of finding a concept that handles "files" in a different way than every OS before sounded like something very compelling to me the first time I heard about it (and ever since).

KDE's Scott Wheeler has been brooding about the problem already for some time: "Why is it that Google can find some matching content on the Internet faster than I can find a document on my own harddisk?"

Scott has developed a concept for an entirely new way to access documents and data on your computers. The idea is to not throw away contextual information and not ignore meta data, as we do now. Instead, we should store these things for later reuse and retrieval.

If I save an image attached to a certain e-mail today, a lot of relevant context is lost forever (unless my brain accidentally remembers it): Who sent me the picture? Is the sender in my address book? Did she send mails before? Did she send more graphic files in her other mails? Where have I used this picture?

Scott's "Contextual Linkage Engine" is called Tenor. It is projected to collect all of this meaningful information from all programs we use. Of course, this isn't an easy task. First of all, the programs need to support this functionality. Tenor support must be deeply rooted in the environment it works within. KDE is an excellent research and development ground for this. After all, its individual components and applications are already integrated better than all its competitors.

Scott had this to say about Tenor: "The cool thing with Tenor is that it's asking bigger questions than the recent round of desktop search tools. It goes beyond 'Which of my files contains this information?', though it does do that too. But the point is, it also extends it to say, 'What's related to this?' and being able to search through those relationships too."

Not a Search Tool, But a Find Framework

Tenor will be unlike other "search tools" that currently exist. It is not a mere filename- and extension-indexing mechanism, married to a "locate"-like database, nor does it simply search the text of meta data or text inside of documents. Tenor does much more and reaches beyond anything else that has ever attempted to help users handle large amounts of personal files and data, finding the proverbial "needle in the haystack" piece of info by using stored contextual attributes. The recalled contextual info lends itself even to "Context Ranking" algorithms. (Google, are you listening?)

Recently Scott was invited to present his groundbraking concept to the German Research Center for Artificial Intelligence. One of their workgroups has been asking similar questions regarding the sheer mass of digital information to be handled by users. The group, working and researching on the Microsoft OS platform, is concentrating its efforts primarily onto the semantic desktop which draws its inspiration from W3C's RDF definitions.

Scott told me this about their resulting discussions: "It was interesting--I mean, we focused on the differences in our research, but it was encouraging to see how similar their solutions to similar problems were to the design of Tenor. There were quite a few jokes afterwards about the whole lab switching over to KDE (from Windows) once we get this working."

Scott's contextual linkage engine, itself nearly invisible to the user, will drive the "Content Browser" that goes one step beyond a traditional file manager. Content browsing will of course not replace Konqueror's functions, nor will Konqueror go away. Rather, it's legacy file manager skillset will be complemented by the content browser methodology. With the content browser, you will be able to find and use information spread out across your system using the same semantics you use when thinking about or describing that information in day-to-day life. It will also allow you to find and build relationships within your information set.

The effects of Tenor will also be visible everywhere--on the desktop itself, in groupware applications, and in content creation applications. Contextual navigation will make itself even felt when having to deal with the dreaded topic of "configuration" for individual applications as well as the complete desktop environment system. But perhaps it is in the Content Browser that its capabilities will be the most obvious.

Tenor scratches an itch that is present on all workstation OS platforms. Maybe the itch aches more on KDE, since KDE is more flexible, more versatile, and more configurable than any other environment. But the itch exists on mearly every platform.

The APPEAL Desktop Project, a newly formed workgroup inside the KDE community, explores ways to take the next generation of KDE onto a new level. Tenor has been identified by APPEAL as one of the strategically important technologies emerging inside KDE and is therefore taking an active role in supporting its further development. Aaron J. Seigo, KDE core developer and one of APPEAL's members, recently haiku-ed the central ideas behind APPEAL into three items:

"Breathtaking beauty
clarity in interface and
creativity."

With Tenor, creativity is certainly there. And in a not too distant future it will hopefully help me find the "attomic information" needles inside my 1,635,315 file-hard disk haystack.

Kurt Pfeifle has been involved with the KDE project since version 2.2, mainly with KDEPrint. A contributor to the APPEAL Desktop Project and Linuxprinting.org, Kurt is the author of several printing and CUPS-related documentents and HOWTOs and the main author of the printing chapters of "The Official Samba HOWTO Collection". In his spare time, Kurt doesn't like to herd cats and loves to go paragliding (and daydream about it).