What's New from 2005

What's New from 2005

What's New - 31 December 2005

Year End Review This is the traditional time of year to summarize the previous year and to preview the coming year. Items of interest from 2005:

• ISFDB - Almost all of the time spent on the ISFDB this year was involved in getting online editing facilities up and running. Accomplishments include:

• • Converting the ISFDB to MySQL database.
  • Creating the fundamental ISFDB editing tools.
  • Changing the website layout to support the editing tools.
  • Adding rudementary user account support.
  • Adding a wiki to support the more static aspects of the ISFDB.

• The major non-accomplishment: the editing facilities aren't public yet.

• Wikipedia - There was a huge flap this past year with regards to Wikipedia and article accuracy. Obviously, this controversy is directly applicable to the ISFDB as well, as it is one of the most direct criticisms (see msg 206) of the ISFDB. We also experienced a major problem with porn spam in the ISFDB wiki, and blog spam here in the ISFDB blog, forcing page protections on many wiki pages and moderation on blog comments. Bibliographic data tends to be far more objective than encyclopedia articles, but detailed verification of each and every submission erodes the power of large numbers of people performing submissions.
• Contributions - There were many offers of help throughout 2005, but most of them were related to editing and not to application development, and app development is a prerequisite to have people help with editing.
• Personal - As a rule I don't usually discuss the day job here, but as it tends to consume an increasing number of hours per week, it does have some relevance. I really only have 1, occasionally 2 hours per day of discretionary time to devote to... anything. Sometimes (as during most of the later half of 2005) I have no time whatsoever. As such, progress is, and will continue to be, slower than some would prefer.

Upcoming in 2006:

• For numerous reasons, work on the ISFDB historically drops off significantly in the later half of the year. As such, I plan on wrapping up 2006 ISFDB work by the end of June.
• In order to make the ISFDB self-sufficient in 2006, and to work through editing and moderator issues that might arise, I plan on having public editing capabilities no later than the end of March.
• Items currently standing in the way of public editing capabilities:
  • Database architecture - The current schema works mostly okay for both display and editing purposes - except in the area of pseudonyms. As such such, some de-normalization needs to take place.
  • User management - I don't currently have a system in place to manage user logins, although I may piggyback on the ISFDB Wiki capabilities.
  • Editing tool fixes - There are still a couple of bugs to fix in the editing tools, and a major hole in the ability to edit entries with multiple content, like magazines, collections, and anthologies.

Finally, I've put up a new page of the most popular ISFDB authors in 2005.

The ISFDB was a Winner in the Third Annual Wooden Rocket Awards, for best science fiction and fantasy websites (sponsored by SF Crowsnest). The ISFDB won in the category of Best Directory Site.

Thanks to everyone who voted for the site.

Reading: Iain M. Banks - Inversions. Jared Diamond - Guns, Germs, and Steel.

What's New - 13 December 2005

1. TAMU did some server maintenance over the weekend, and at present the ISFDB http server won't execute any CGI scripts. This will be rectified soon, hopefully.

1. I cleaned out the blogger spam in the comments section of this blog, and set the comment policy to moderated. So any comments made here won't show up instantaniously. Additionally, I cleaned the porn out of the Wiki, and protected some of the pages.

1. Since I haven't had much time to work on the ISFDB for the last half of 2005, I'll continue to work on the tools until they are sufficiently mature that I can stop working on it.

What's New - 5 December 2005

Fixed a date calculation problem that resulted in no forthcoming books for the month of December.

There's a fine posting by Robert Reginald in the ISFDB forum lambasting the ISFDB as "almost wholly unreliable". It's both true and inevitable.

Reading: Iain M. Banks - The Algebraist

What's New - 18 October 2005

Needless to say, I returned back from Estonia over a month ago. I'm just now able to start working on the ISFDB again. I haven't made any changes to the tables yet to work out the pseudonym problems, but I have been making corrections (20,098 so far), making fixes to the editing apps as I run into them, and fixed a problem in the forthcoming books app that prevented the December books from being displayed.

Also finished the script that downloads the current image of the SQL database, strip out submissions, login information, and the wiki tables (it's stored in the same database as the ISFDB), and emit a cleaned up version of the SQL database. The current version is available online at http://www.isfdb.org/backup.gz

What's New - 28 August 2005

Just returned from Japan, and found that the isfdb domain expired this morning. My current registrar sucks, a situation I will also fix soon.

Reading: Neil Gaiman - American Gods

What's New - 21 Aug 2005

Technically, nothing's new - the day job has taken over, and very little has happened to the ISFDB since the beginning of July. And with a trip to Japan this week, followed by a trip to Estonia next week, it's unlikely that much will happen until September.

I did attempt some serious ISFDB data entry last month, and found that the whole pseudonym situation with the current database layout is completely unacceptable, so I'll be attacking that first. I'll also get a downloadable version of the database online. I have a week of vacation lined up in September, so things should start moving forward again.

Reading: Dan Simmons - Olympos. Soundtrack: Muse - Hullabaloo.

What's New - 29 Jun 2005

I began doing online editing experiments last Friday, and they have gone well enough that the update from the 23rd will be the last wholesale update - the online database is now the master copy of the database. I now have a rudimentary login system (although it doesn't register new users yet), and I also have a primitive webservice API that allows my remote tools to perform automated updates (the API needs a lot of work, so I'm not going to publicize it until a later date).

My target date to turn on public editing capabilities has been June 30th, but I think I'll be a bit late on that date. My 25th wedding anniversary is in a couple of weeks, and between the obligatory associated festivities and a week-long visit from my parents, I won't be able to devote a lot of time to the site for the next two weeks. Since it doesn't make a lot of sense to turn on a major feature and then walk away for a couple of weeks, I'm now shooting for July 15th as the target date. In the meantime some minor features may be firmed up, some known bugs fixed, some updates to the online documentation will be made, and I'll try to get the database and source code available for download again.

Reading: Edward E. Smith - Children of the Lens. Soundtrack: Muse - Showbiz; Foo Fighters - In Your Honor (Disk 1).

What's New - 23 Jun 2005

Yesterday's update contains a huge number of data fixes, most of them centered on the top 200 most-viewed authors over the last three weeks. It may also be the last wholesale update that's done - I'll be experimenting with the editing apps over the next few days, and if they work okay I'll turn on the editing of author data, later followed by titles and publications. Ability to add/edit anthologies, collections, and magazines may take another couple of weeks.

Jeff Bactel has put up the new ISFDB wiki. We'll be moving all of the remaining static HTML pages from the ISFDB into the wiki, so that they can be edited by multiple people. Anyone who want's to pitch in may do so - after all it's a wiki (just don't move the old FAQ over - it's out of date and we need to start a new one from scratch).

Reading: Edward E. Smith - Second-Stage Lensman. Soundtrack: Muse - Origin of Symmetry; Nine Inch Nails- With Teeth.

What's New - 23 May 2005

Target Date - 30 June

The target date for deploying ISFDB2 is 30 June. All of the bibliographic pages are now operational with MySQL; most of the editing tools are complete; edits are controlled and integrated via a moderator. Here's the remaining work that needs to be completed before putting the new database online:

• Search - I still need to finish the advance search features. Most of this is already complete, but need about 1 day to finish and cleanup.
• Content Editing - The editing tools for publications, titles, and authors are complete (as well as merging tools), but I still haven't started the content editors (for filling in magazines, collections, and books).
• REST API - There will be a small number of API's that will allow external apps to make direct queries via URI's - such as retrieving publication data using an ISBN key.
• Login - Access to the editing tools will be controlled via login/password, and the following two features also require user logins.
• Tags - I'm going to get rid of the concept of non-genre records, and open up the database to other genres as well. To support this, each title will support folksonomy-style tags, such that users can categorize each title with keywords, genre, whatever.
• Votes - Users will be able to rate titles on a scale of 1 to 10.

There are additional items remaining after the above are completed, but they won't require extensive changes to the database schema, so I won't hold up deployment for them:

• Author Directory - The directory will be automated from the online data, not statically created.
• Magazine Pages - The information on these pages (which account for most of the HTML pages on the site) will be pulled into the database so that they can be edited by users.

And finally, it's been awhile since I posted one of these milestones: Exceeded 170,000 titles.

What's New - 31 Mar 2005

Burning Off the Crazed Vegetation

We live out in the country; far enough out that our neighbors are interesting specs of color on the horizon. For the last week farmers around here have been burning off the weeds and grass that grew in alongside the roads and runoff channels since harvesting. If you like the small-town smell of burning leaves in the Fall, well... that's small-town stuff - these burnoffs are a meteorological phenomenon, directly reducing visibility with widespread smoke and haze, and providing enough airborne particles to induce morning fog. They make for lovely displays at night.

I've pretty much completed all of the major normalization of the database, and along the way I've found a lot of crazed data vegetation growing in its various nooks and crannies. It's funny how those database books whine about normalization as a prerequisite to reduce data entry error - turns out they're right. So I've been burning off a lot of the weeds that have been growing in the database over the last 10 years. And just like in the picture above, burning off the weeds doesn't necessarily mean that your land is now completely weed free. So don't expect to see a pristine, error-free database when it shows up online. But it is much better.

I've been through a couple of rewrites of the ISFDB applications in the last year - once to convert over to Python, and another recent rewrite to convert over to MySQL. The general content layout of the ISFDB however, hasn't really changed over the last 10 years. The recent normalization of the author names required a new approach to the summary bibliography application, which has lead to a fairly substantial change in how data is going to be presented across the different pages. Combine that with our first use of CSS, and you get a website that you are definitely going to notice is different. I figure another week to finish off the bibliographic apps, and another 3 to finish off the editing apps, and I should have the whole thing online by the end of April.

A lot of people have been entering author biographies into Wikipedia, and some people have brought up the fact that some duplication of effort will be going on, trying to maintain biographies in both the ISFDB and in Wikipedia. As such, I plan on moving author bios out of the ISFDB and into Wikipedia, and adding direct links from the author's summary bibliography to their Wikipedia entry. We should consider a similar linkage for book synopsis, as many books have entries on Wikipedia as well.

So, if you would like to contribute some help to the ISFDB, something that people could start to do while coding continues, is to make author entries at Wikipedia for authors that have biographies in the ISFDB. There are 618 of them. I've put together a rough list of the authors at http://www.isfdb.org/authors.txt

What's New - 7 Mar 2005

Here's a quick update on the current status of the conversion of the ISFDB.

1. The new version of the ISFDB is now completely operational with MySQL. Don't go looking at the ISFDB for evidence - it's only running on the home servers at present. The primary reason for this is that the MySQL database layout is still in flux, while normalization continues (the database has change A LOT since version 0 was put up in January). I've tried to put a test version of the database online at TAMU, but due to the size of the database the process is painful enough that I only want to do it once. My guess is that the new version will probably be up and running by the end of March - April at the latest. I know - you'll believe it when you see it.

To minimize maintenance and errors, I'm only updating the MySQL version of the database at this time, which essentially means that I'm not going to update the online site again until the new database is ready.

1. Submissions. When the MySQL database shows up, all sorts of data submission capabilites will show up as well. Although integration of the submissions will occur directly into the MySQL database, the submissions will be moderated - meaning that they can either be accepted or rejected. The author editing interface was completed over the weekend, and I've been using it during the normalization changes.

1. Web services. The final piece that will show up with the new ISFDB will be a set of web service API's. These will allow you to make direct queries to the ISFDB from external applications, receiving back XML data.

The current set of documentation concerning the new ISFDB2 can be found here.

What's New - 27 Feb 2005

• Updated all of the ISFDB2 documentation and uploaded a current version of the MySQL database and the isfdb2 tools.
• All the code and data for the ISFDB is now under the Creative Commons Attribution License. This is pretty equivalent to both the OpenContent License that covered the data, and the BSD license which covered the source code. The intention is that anyone can use the code and data for any purpose, so long as attribution is made.
• Removed the OSI Certified Open Source since the OpenContent License has not been approved by OSI. I don't think this really means much anyway.
• If you're worried about the applicability of the Creative Commons license to software, I'm not. Software licenses are generally more complex to handle the different aspects of source code, object code, and binaries. We don't release object code or binaries, only source code. And since we're moving to a completely scripted solution, there won't be any object code or binaries to worry about.

What's New - 29 Jan 2005

• Another SQL update. The schema has been updated, and a new version of the database has been uploaded. The python tools are now complete and ready to be given a trial run on the web site.
• Soundtrack: Blur - Leisure - Modern Life is Rubbish.

What's New - 22 Jan 2005

• A few minor mods, but mostly an SQL update.
• A new copy of the MySQL database was uploaded
• A first pass at documenting the MySQL database schema is available.

What's New - 18 Jan 2005

Progress Report

I almost have help on several fronts now, and while no one is actually pitching in yet, the side discussions we've been having have been fruitful.

The first front is the status of the port to MySQL. I uploaded a snapshot of the new isfdb apps and a MySQL dump to the website in late December so that some joint work could be done. The version of the tools that are currently online are pretty much limited to the editing tools (which mostly work, but are still incomplete and probably slightly buggy). The discussions conducted along this front basically converged on this strategy: pretend it's 1995 and the ISFDB doesn't exist and you want to build the site from scratch - what do you do? So instead of trying to figure out in advance how to replace each and every feature in the current ISFDB, we can instead focus on fundamentals, like just getting a MySQL-driven version of the database up and running, and worry about everything else later. This approach greatly simplifies matters, and in fact the online database is about 85% converted now. I've taken the port about as far as I can without adding a few new index tables to speed some of the apps up, but over all it's humming along. I'll put a new snapshot of the tools up in a couple of weeks.

The second front has been a discussion on data submissions and editing. There have been some interesting noises over the last couple of weeks concerning Wikipedia and anti-elitism, and how it hurts the encyclopedia's standing as an authoritative publication. The ISFDB has had similar well-deserved complaints of inaccuracies lobbed its way, so there's a double-edged sword here of trying to choose between rapid growth (and more inaccuracies), or better data (and slower growth rates). I think in this case we'll again try to follow the IMDB, where the public can submit data, but it is moderated by editors. This was a model that was tried in the ISFDB years ago with limited success, but the differences here would be: 1) the tools will do the work not the editors, 2) the editors will either approve or reject submissions, and not try to burn huge amounts of time correcting the submissions, and 3) there will be editors instead of an editor. The mechanism that I'm settling on is generating XML-based submissions, which will be converted into SQL statements upon approval. I'm whipping up a prototype for adding new publications (predominately magazines) , and Mark Watson (aka the editor of BestSF) has agreed to take it for a test drive. For now the app will drop XML files in the TAMU file system, and a moderator app on my home system will download and integrate into the MySQL database. This will be online in a couple of days for testing, and once it passes a shakedown cruise (and I finish the moderator app), I'll make it usable to the public - for new submissions only - I can't do tool-driven corrections until the MySQL database is online.

The third front has to do with a web page editing system. I had some help lined up on this one, which... has... well, I'm not sure what's happening there. I think we were leaning towards adopting WikiMedia (which is easy to set up and use), but I'm still thinking about a Zope/Plone solution since it can be moderated.

What's New - 9 Jan 2005

• Pass 1 of integrating a major awards update from David, plus some end of the year bookkeeping. The awards update won't be complete until final merging with the rest of the database has occured.
• I've been generating the online data files from a MySQL database for some time now. A new set of online applications able to render webpages directly from the MySQL database is about 85% complete. Once this is finished, I'll replace the online tools with the new ones. The only noticeable change should be the addition of power searching.
• I'm 100 percent focused on the database conversion, so I'm not addressing any data corrections or additions at this time.
• Reading: Neal Stephenson - The System of the World.
• Soundtrack: Kyuss - Blues for the Red Sun, Welcome to Sky Valley.

What's New - 6 Jan 2005

Anything Good Happen While I Was Out?

Anyone who's ever browsed through the What's New section of the ISFDB (or has tried to get me to answer email for that matter), has probably noticed that I completely drop off the face of the planet from time to time. For long periods of time. My single longest recorded dropoff was for 13 months in 1999/2000.

If my life was a science fiction story, I wouldn't be able to remember anything about these disappearances, and I would stumble across odd secret-agent gadgets hidden away in the closets and attic of my house. I would meet people at the post office who seem curiously familar, who glance back at me with a brief grimace of recognition. While picking up a fallen magnet, I would discover three Polaroids of a stranger in the dust under the fridge. I would find notes written in my hand, in languages I don't know, using mathematics unknown to our civilization. I would have miscolored dreams of extra limbs and foreign appendages, while floating in a pressure suit filled with liquid helium. These dreams would be strangely comforting to me.

Alas, my life isn't a science fiction story (well, at least not like the science fiction stories they print on this planet), and I remember all too well what happens during these dropoffs, as well as what causes them. The causes roughly boil down into three categories: inclination, competition from other interests, and time.

Once upon a time, like many avid science fiction readers, I thought that I would not just read stories, but write them. I was duped into this thinking by well-meaning college professors during classes shaped in the form of writing workshops. Each of these professors praised my submissions, and encouraged me to submit them for publication. Over time I began to believe them, and armed with dillusions of writing literate short fiction I read publications like Granta and the annual O. Henry Prize Stories . I, of course, eshewed science fiction, but over time came to realize that my stories were a tad too bizarre to be published in traditional literate publications. So I began a systematic study of the science fiction marketplace by creating lists of award-nominated and winning fiction, the publications they appeared in, and then collected and read them. Over time the act of creating and maintaining a point value system based on awards became an obsessive act, leading to one computer program after another, and eventually the ISFDB was born. After which, I spent all my time maintaining and updating the ISFDB, and no time whatsoever reading short fiction.

Recently, however, I've come to a simple realization: I'll never be a writer. After years of socializing with writers, it is patently obvious that writers write because they have a deeply embedded need to do so. They even say exactly that when you talk to them - every single one of them will tell you straight out "I write because I have to". I don't. I wrote to meet class deadlines, and whenever the deadline passed, I stopped writing. I haven't finished a story 15 years. I don't keep a journal. I don't have index cards of story ideas. I don't dream of attending Clarion. I'm not working on a trilogy. So I've stopped worrying about it.

The real problem then: if I'm not going to write, then I don't need to understand any publication marketplace, and I don't really need to engage in a systematic reading of the finest fiction in that marketplace, so I don't really need to build lists of fiction anymore, which means I've lost the primary incentive for maintaining the ISFDB. As such, the only reason at all that I stick with the ISFDB nowadays is that it has become an interesting technological and sociological playground.

I'm not big on New Year's resolutions, but this year I have one: I'm going to stop maintaining the ISFDB at the end of 2005. There are two ways this can play out. The first way is that the ISFDB is completely converted in 2005 into a self-sustaining, community-driven website. This will require the help and involvment of others to realize. The second way is that the ISFDB remains pretty much as it does today, maintained by a single editor - it just won't be me. And if no one else steps forward, then weeds will start to grow over the site, the concrete will start cracking, and the ISFDB will become just another footnote in someone's early history of SF on the Internet.

P.S. - This is a gasoline-driven post. We lost electricity yesterday around 4pm in the ice storm that languished over central Illinois. It's now 9:00pm the next day, power is still out, and I'm running the furnace, the router, and my WiFi connection to the nearest town with a trusty John Deere generator. Maybe my life is a science fiction story after all.