A little bit of New Year’s hustle and a few bug fixes later, pwnserver is ready for pre-alpha release. What does pre-alpha mean? It means it may take technical proficiency and perseverance to get pwnserver up and running. It’s worth noting, however, that once it’s up and running it’s very stable. I run it once an hour, from cron, and it reliably pwns both new and old tweets from my Twitter accounts. It also powers my coworking twitter chatter chart.
Pwnserver isn’t a brute force, bulk download tool. It you want to hammer Twitter and suck down every tweet as fast as you can, you are better off rolling your own tool. Pwnserver is designed to digest bite-sized chunks of data over a span of time, to avoid drawing attention to itself. It’s also designed to reliably keep on top of new data while simultaneously working backward in time to earlier data.
After 14 hours, I have first data on “coworking” on Twitter.
As the Twitter Search pwner runs against the query results, the pwner updates the data used to created the chart. This happens once an hour, so check back often.
Pwnserver is a tool I created to undo some of the damage that has been done by Web 2.0 business models that encourage us to share and store valuable data online, but don’t or won’t provide tools for easily getting that data back. You know who you are. (Note, Twitter is not, IMHO, such a company. However data is data, and Twitter provides a great, easy first case.) That pwnserver can be quickly repurposed for research projects like this is a wonderful added benefit.
Pwnserver will be released as Free Software once I clean it up.
It’s difficult for me to quantify the growth of coworking.
I’ve noticed, for example, that the volume of tweets on Twitter mentioning “coworking” has increased dramatically in the last few months. Or has it? It’s hard to tell.
While riding the subway into the city this morning, it occurred to me that the technology I’m using to recapture data I’ve set free on the Web might also be used to gather the data I’m interested in about coworking.
A few hours of hacking later, I’d extended my Twitter pwner to track and capture tweets from Twitter Search. The result is a growing historical record of tweets that mention “coworking” stretching back a few months.
I launched pwnserver over a week ago, pointed it at three Twitter accounts, let it loose… and got busy and forgot about it (the big project on my plate is The Work/Life Revolution–stories about work, coworking, and autonomy).
While I was away, pwnserver patiently pursued the specified accounts, pwned their contents, and kept the local copy up to date. Beautiful! It’s a rare occasion when software works like it’s supposed to work. Especially when my back is turned. I attribute this fact to my extensive and exhaustive use of RSpec stories during development.
I’d turned my back to work on the book and to start work on the pwnserver user-interface. More to come on the user-interface…
The first run completed successfully–797 tweets pulled down and steady synchronization thereafter. After a T-day worth of hacking, the second production run is underway–a cadre of pwners pulling down tweets from three Twitter accounts. I cleaned up the database, improved the logging, and found and fixed a few edge cases.
The point of this project is to build a Free (as in Free Software) toolkit for getting my data back off the various web-based applications/social networks/platforms/hacks on which I put it. Twitter’s a push-over. Twitpic is next… I’m working my way to Facebook.
I put pwnserver into production two days ago. It’s a pretty simple pwn–Twitter’s a pushover with an API and all, but start off easy, I say… I’ve also pulled way back on the throttle–it will take me days to swallow all of the historical data. But then, brute force is a chump’s solution.
The proof will come 1) when this pwner hits the end/beginning of my Twitter history, and 2) when I start tweeting on this account again (which will add data at the beginning/end of the history). At that point the pwner is really just synching my database to the Twitter database.
