Researchr

Note: There has been quite a bit of development since I made these pages, and some of the installation/tech information is obsolete. I hope to update this soon, making it much easier for others to install this (it will still not be very easy unfortunately, but perhaps go from hard to moderate). In the meantime, if you are interested, take a look at the code, and send me an e-mail.

• Making Open Access articles much more visible, automatically
• Integrating iPhone photography with wiki for note-taking during conference
• tag-extract: A tool to automatically restructure text/outline using tags
• Search 40+ edutech wikis
• Using web clipping and sidewiki to gather and synthesize information
• API to check if a publication is “Open Access”
• “Semantic” Researchr/DokuWiki search

All of these show actual functionality, but much of it is experimental, and some of it may not be available in the Github repository yet, or might have been removed

• Information extraction and synthesis using clip, clip-again and side-by-side editing
• tag-extract quick demo, sorting interview transcript or literature review
• Using researchr and tag-extract to create a literature review
• Automatic import of citations from Scrobblr and Researchr
• Automatically importing publications to BibDesk based on DOI and anystyle-parser
• Proof of concept automatic metadata retrieval based on file fingerprinting
• The original researchr demo, embedded below

I've been really interested in how I can better manage all my PDFs, citation metadata, clippings, notes and ideas, etc. This becomes even more relevant now that I have entered a PhD program, and presumably am collecting articles and ideas not just for my PhD, but for many years as a researcher.

I am also very interested in how I can be an ”open scholar”, and share not only the final products of my research, but also all the “ephemera” produced along the way. I was inspired by Cormac Lawler's PhD wiki, as well as Euboulides, who does crazy things with Skim, BibDesk, DevonThink and MediaWiki.

It's really a whole framework with a bunch of applications (the key ones being BibDesk, Skim, DokuWiki, Chrome) and lot's of small “glue scripts” tying them together, as well as Keyboard Maestro to provide shortcuts. Many of these scripts are really useful in their own right, but together they make up an “academic information management workflow”, which is at the same time individual and happens on the local computer, but also set up to be easily shared with others. I just call it “researchr” to have something to call it - it's not a product I am trying to market, but rather an attempt to share both some neat things that have proven useful in my work, as well as the philosophical idea of an integrated academic workflow.

(see screenshots for illustrations of key features)

researchr - demo of an academic workflow with DokuWiki and BibDesk from Stian Haklev on Vimeo.

(link to video on Tudou, for mainland China readers)

I download lot's of PDFs, which are all for some reason named fulltext.pdf. I end up with a Downloads directory with hundreds or thousands of files. I take notes on paper, in DevonThink, in Simple Note, and cannot find what I need. I feel like I am loosing control over what I am learning, and have problems “converging” - bringing different theoretical ideas together. And then I end up writing final papers, hand-formatting citations, and having no way of linking the final paper to my earlier notes. In all of this, I'd love for some simple way to share my notes and ideas with others.

The solution for me came in the form of a few programs that were already amazingly powerful and capable. BibDesk for managing citations (I tried pretty much every possible bibliographic manager, and then some), a very powerful open source citation manager based on the open BibTeX format (which might be a bit unwieldy, but is quite universally understood).

As the information management system, I chose DokuWiki, a really great wiki which does not use a database, but plain text files (this made it much easier both to interact with programmatically, and to sync between my local computer and a server). There's a vibrant community of developers and lot's of great plugins available (some of the important plugins for this project).

The realization that I could access Applescript in Ruby through Appscript-rb made a huge difference. I didn't want to learn a whole new language, but since I often only need one or two lines of Appscript in a script, this was a perfect solution. It took some practice to learn to craft obscure lines like

app("Google Chrome").windows[1].get.tabs[dt.windows[1].get.active_tab_index.get].get.URL.get

but once you get it, it's incredibly powerful.

Finding Sylvester Keils' bibtex-ruby and citeproc-ruby, which allowed me to process bibtex files and generate citations. Sylvester has been incredibly helpful whenever I ran into snags or problems.

I've put quite a lot of time into experimenting with these scripts etc, much of it because I was learning so much along the way, and also because I was gradually learning what was useful to my own learning - I didn't start out with a blueprint. Someone might very well come along and make much better scripts, or integrated programs that are more functional than my setup.

However, I am also spending a ton of time importing articles and metadata, reading and taking notes from articles, synthesizing into topical articles, etc. How “future proof” is all this? Will it all be useless in a few years, when a much better tool comes along?

I don't think so. BibTeX is not perfect, but it's an open standard that is recognized by almost every program - it would not be difficult to import all citations and PDFs to another citation management system, should I wish to do so in the future. It should also not be that hard to move the notes and highlights from the wiki to another system, if I should ever want to do so - I might miss some of the functionality, but the information is there, in simple text files that are easy to process and export.

Some of you might read this far, and say - this looks really neat, I'd like to install it on my computer and start using it for my research. Unfortunately, given that this isn't one single program, but rather a combination of a bunch of programs, and scripts that integrate with those programs, it's impossible for me to create one simple installer which will let you jump right in and use this setup.

So why I am spending all this time documenting this? In my mind, there are two kinds of “target groups”. The first are tinkerers, who are comfortable with Ruby, the terminal, etc. All of my scripts are available, and I've made some effort to document them, as well as (perhaps more valuably) documenting some of the “neat ways of doing things” that took me a long time to find. I hope this will be useful to you.

Perhaps you want to try to replicate the entire setup - it would be quite a bit of work, but it's certainly possible. Perhaps you just want a certain component (say side-by-side editing in DokuWiki, or automatic import of Kindle annotations)? This should also be quite doable (and if people contact me and tell me that a certain component would be very useful, I might try to separate that out, document it, and make it easier for people to access just that functionality).

But to people who are not able to or willing to spend lot's of time looking over scripts and commands, I hope that this could still be interesting as a demo of what is possible. During a university class, I was part of a group that designed a “mockup” of an academic workflow that we called “junior researchr” (this is where I took the name from). What we showed in that video was not a functional workflow, but something we “simulated”. In this case, the system I am showing you is something that I am using every day for hours, and exactly how it works has been shaped intimately by how I think.

I think we all deserve much more powerful tools for thinking and managing knowledge, and this is my “stab” at it. I am also fascinated by the relationship between how we grapple with ideas individually, and how we can do so in groups (I wrote about this here), and the one thing my system is missing is a social network. I don't want to open my wiki up to editing, but I'd love lot's of others to write about articles on blogs and wikis, and to be able to very easily discover those posts, and share our ideas.

Feel free to get in touch with me by e-mail for feedback or questions. I am happy to assist with using or modifying any of these scripts, or discussing the ideas behind them.

• Screenshots of the functionality
• Anatomy of a publication page
• What are citekeys, and why are they so important?

• Keyboard shortcuts
• Some neat "design patterns" that I used in my coding
• Description of all files in the github repository
• Descriptions of some of the plugins used
• Configuring Apache

• researchr:start
• Discussion

• Read
• Show pagesource
• Old revisions

Search