Hypothify: first outline

I’ve been pondering a hypothesis-based academic social media site for a few weeks now; and talked with a couple of people about it. Ideas are only just beginning to coalesce but now seems the right time to try to outline what I see hypothify doing, and how it might work. I’m conscious that it needs to start very simple, and remain as simple as possible. It’s easy to come up with a massive feature list, but identifying the most important stuff and planning how it might work is key. [edit after writing – there’s still A WAY to go on this!!]

What?

A place to propose,  discuss and aggregate evidence for/against hypotheses. Consensus building through democratic voting up/down of evidence/ discussion items. What I thought Quora was going to be when I first heard about it, but less question/answer and more evidence-based.

In traditional(ish) publishing terms it would represent a platform for ‘living’ review articles on each hypothesis. However, would integrate social media aspects (voting and sharing) and wider evidence than just academic papers.

Not peer-reviewed, or peer evaluated, but peer assembled.

Why?

Hypotheses are fundamental to academic work. They represent the ideas and concepts which propagate through the academic sphere and out into the wider world as our understanding of the natural world/ the universe/ the human condition etc. They are often dissociated from the piece-by-piece evidence in the traditional academic record. Currently academics are supposed  to read everything and make up their own mind on a particular matter. For each individual this is only possible for a limited number of concepts/hypotheses because of the massive time cost of i) finding all the literature and ii) reading it all and ii) keeping up to date. In reality we all take ‘received wisdom’ on many matters on trust from other academics, or tend to disbelieve everything we’re told and argue it out ourselves from first principles! Hypothify would solve the pain of i) and negates ii) and iii) by providing community-maintained consensus instead of ‘received wisdom’ on each given hypothesis.

How?

The platform would allow the proposal of hypotheses by any user. Evidence items (papers [via Mendeley API if poss], unpublished data [figshare, slideshare, blogs or notebook entries], snippets of discussion / reasoned argument [from discussion of this hypothesis or elsewhere via e.g Disqus]) can be presented by any member of the community  as being ‘for’ or ‘against’ the hypothesis. Key to the usefulness of evidence items will be a tweet-length summary of what that evidence item contributes to the assessment of the hypothesis. One will have to be added by the person introducing the evidence, other ‘competing’ summaries may be added. Where necessary, discussion of the evidence can be conducted and this itself can be cited as evidence itself. It is conceivable that a single piece of evidence may be argued to support either side of a hypothesis. Maybe it’s necessary to recognise that evidence can be ‘agnostic’?

Key to the success of the platform will be the voting up/down of content (a la stack exchange). Hypotheses themselves should not be voteable on I think  -i.e. there will be no opportunity for individuals  to vote subjectively/dogmatically for/against a hypothesis, only vote up or down evidence supporting or contradicting the hypothesis. Plus vote up or down particular summaries of evidence items, so the best summaries float to the top for each bit of evidence. So the ‘hypothesis view’ page will show the hypothesis at the top and evidence items for and against (highest voted first), with the best summary pertaining to that hypothesis for each one. Plus a link to the evidence item (i.e. NOT stored on-site). I think this is really neat because a user can find a hypothesis they’re interested in, find what the community thinks it the base evidence for and against, read those bits, and make and informed decision based on comprehensive community review of the field. It may or may not be useful to have a ‘swingometer’ for each hypothesis which  represents the net votes for evidence for and against the hypothesis, which give a ‘community assesment’ of the hypothesis.

Attracting users?

What’s in it for users? Firstly, being seen to propose and and contribute to hypothesis assessment will bring kudos to users. A ‘reputation’ system (also a la Stack Exchange) could be implemented to measure reputation / value of contributions… Even badges etc would probably work for academics, but I think there’s a more instantly attractive ‘doughnut’ (as my good friend Charlie calls them) – promotion of your research output. If you add a good summary of your paper which informs debate on a particular hypothesis it will (if it’s good) float towards the top for that hypothesis. You will be able to engage with other interested parties and discuss your research. Google will love it.

Engage ‘the enemy’. Let’s say I propose a hypothesis, which just happens to be something I’ve proposed in papers in the literature in the past. Great. I put up the hypothesis, provide evidence items. As I’m the only contributor, the hypothesis is only ‘proposed’. To move it to the ‘community debated’ stage I need to get other people involved. So I share it on Twitter, but also invite people I know will be interested, to join the debate. Furthermore, other established hypothify users will be automatically invited to join based on their interests  and other hypotheses they’re active on and the tags which have been associated with the hypothesis in question.

As evidence items are added, the system will attempt to ‘scrobble’ originator details (emails, figshare user details, Mendeley author details) and contact the originators to inform them that their work is being used to inform debate on a particular hypothesis. They will be invited to join the debate. I’m guessing if their work is being ‘correctly’ cited they will be flattered enough to go and have a look, and if it’s being ‘incorrectly’ cited (in their opinion) they will be incensed enough to wade in and put those upstarts right. Thus the experts will hopefully filter in.

Furthermore, as evidence and discussion accumulates and more people vote evidence, evidence summaries and the hypothesis summary  up and down, the ‘top contributors’ will be identified. Those top contibutors, plus the hypothesis proposer (i.e. the proposer on hypothify, not necessarily the originator of the hypothesis in the outside world [who should be cited and acknowledged]) will be identified at the head of the hypothesis view as the ‘authors’ of the synthesis. Thus each hypothesis becomes a peer assembled citeable document (hopefully with a doi assigned). A publication! And as we all know in academia, publications are all. And what’s really nice is that it doesn’t matter which ‘side’ you’re on. If you’re presenting valuable evidence and discussion in either direction, you’ll be listed. So all those old vested interests evaporate away like the eyewash they are.

Synthify?

Not all problems present well as hypotheses. For instance, in my field – marine biogeochemistry – much science is exploratory, and/or based on assessing magnitudes of things : “What is the globally representative concentration of ammonium in the surface ocean“; “What is the annual global carbon uptake by primary producers in the ocean“. Of course, these can be presented as hypotheses “The globally representative concentration of ammonium in the surface ocean is 150nM“, but this is rather pointless. However, the accumulation of evidence leading to an assessment is much the same process as outlined above for hypotheses, only without the FOR and AGAINST argument. And these syntheses could then feed in to wider hypotheses as evidence. Synthify.com has been taken unfortunately, but I think it’s reasonable to conduct such data synthesis work under the hypothify banner. For the field I work in at least, I think the ‘synthify’ function will be as useful as the ‘hypothify’ one.

Anything else?

Moderation will be important and will rely strongly on the community. Controversial topics could get very sticky very quickly. Need to think about policing. Free speech is important, but balanced debate more so. Anthropogenic global warming deniers and intelligent designers are going to be a challenge to the stability and value of the system.

Integration with the rest of the web is obviously very important. All items will obviously be fully shareable, but a proper API would ideally allow full functional push and pull to and from other sites – mendeley, peer evaluation, wikipedia, quora, disqus etc etc.

If all this sounds irresistably interesting, please hit the pre-sign-up  at http://hypothify.kickofflabs.com

Advertisements

17 thoughts on “Hypothify: first outline

  1. foo

    Oh, loving it, I want something like that! Pretty please?

    The thing about the reputation system that you mentioned is, I think, essential, and not just to value and reward contributions. What comes to mind is some kind of “Web of Trust” that is itself evidence-based, if you will. If I’m to judge the contribution of a user, I do not simply want to know how often they’ve contributed, but also what their network of peers is, what those peers think about them, the stuff they’ve published (if any), the topic areas they’re most active in, the evidence they’ve upvoted or downvoted and on what grounds, possibly links to their Mendeley and CiteULike profiles, stuff like that. The reason is that I’m much better able to find my way around the screaming chaos of evidence for issues that are not necessarily (yet) my areas of expertise if I can get an idea of the backgrounds of people who upvote or downvote this or that piece of evidence.
    In this vein, you’d essentially have two types of interconnected nodes that get evidence of credibility attached to them: 1) hypotheses and 2) people.

    A powerful reputation system would also make it much harder for one hypothesis or another to be taken over by large groups of very dedicated people who share a certain bias – yes, I’d see that some piece of evidence was upvoted or downvoted a gazillion times, but I’d also see that most of the voters were either “blank slates” as far as the reputation system is concerned, or that they’re individuals or members of peer networks that are in the habit of aggressively upvoting a certain type of evidence along a certain bias.

    As a bonus, this would also generate an awesome data set that could provide insight into different schools of thought etc. and that would make historians, sociologists and philosophers of science very happy.

    Reply
  2. martwine Post author

    Wow, Foo – glad you like it and thanks so much for your insight! I’m really grateful.

    Completely agree that trust is important and a reputation system can represent this. It’s a really good (and positive) point that a good reputation system would make ‘hypothesis hijack’ more obvious and thus more difficult. Nonetheless, good evidence could still be voted down considerably by a hijack, so it would be missed by people looking to read the e.g. 5 most influential pieces of evidence as selected by the community. This would be really bad. Three possible solutions: i) that moderators could make evidence items ‘sticky’ so they stay at the top in spite of ‘attack’ by ‘zealots’ (subjective); ii) that evidence score is weighted massively based on voter reputation – so the vote of one person highly reputable in that topic area and with a high hypothify rep. score might be worth e.g. 500 votes from zero-reppers. This is possible, but rather serves to reinforce the ‘status quo’ of received wisdom. Better might be to have two top 5 lists – one that gives equal weighting to everyone – a ‘public’ score, and one where only the votes of high-reppers, major evidence contributors, evidence originators and hypothesis proposer are taken into account – an ‘expert’ score. The expert score would only be displayed when significantly different to the public score. It’s worth saying that for most of the geeky stuff academics are into, potential sabboteurs are unlikely to be sufficiently interested in to care. I don’t anticipate hundreds of people offering up-vote and down-vote opinions on most hypotheses – only the community immediately concerned with that particular area of study or proximate ones.

    This raises an interesting Q – how do we determine which topic areas are similar to other topic areas. do we let people ‘free tag’ their hypotheses? We can hardly provide a super granular a priori classification of all the research in the world, but making a big tree of it all as it preciptates out would be super-awesome. @Afternoon suggested a good tagline for Hypothify might be “Mapping scientific knowledge one hypothesis at a time” – seems to fit with this.

    We need to get the design and the maths of the reputation system right, then hopefully it shouldn’t be too(!) difficult to actually implement. I suspect a fully refined reputation system will have to be in ‘phase 2’ development, after we’ve got enough interest to hopefully raise some funding. We’d love to engage with you on the design process if you’re interested – drop us a line at hypothify_at_gmail-dot-com with your contact details and we’ll be in touch. 😀

    Reply
  3. Ben Godfrey (@afternoon)

    Hypothify sounds really exciting. Citeable resources for hypotheses would be a valuable class of research objects.

    On evidence: A piece of evidence is a URL. It’s connection to a hypothesis is some metadata, proposer, summary, etc. If a piece evidence can have multiple implications, then there’s a many-to-many relationship. One paper could present 10 useful pieces of evidence. In hypertext/RDF terms, each piece of evidence could be a fragment (http://example.com/evidence#fragment), allowing researchers to link directly to sub-objects.

    On voting: I agree that voting on a hypothesis directly is subjective and therefore open to abuse. Voting on evidence changes the semantics of the interaction subtly. It hopefully would encourages voters to think rationally.

    On similarity: there are number of clustering techniques that would be able to identify groups of related hypotheses.

    On reputation: StackOverflow and Quora are great models of deriving reputation from behaviour. There’s a lot to be said for following their lead.

    There’s a deep mine of value here. I think even just exposing connections between hypotheses and evidence would be very valuable.

    Reply
  4. mat

    Pretty much what Ben said. Reputation and how that plays into ranking stuff is going to be the important thing. On marine chemistry for example, your vote should be worth way more than mine, but getting the balance right will require some finessing.

    Also, never underestimate the value to people of earning a little gold star next to their username. Promotion of one’s material is a good thing, and boosting rep scores, but everybody loves doughnuts.

    Reply
  5. pavel

    Martin, I’m excited by the possibilities this might offer for qualitative research. As a humanities scholar and professor, I am interested in leveraging the collaborative potential in networks to assess hypotheses (and perhaps less defined research questions) in fields where there is not necessarily a shared praxis. I would be very interested in talking more about the project.
    Cheers,
    Pavel

    Reply
  6. martwine Post author

    Pavel – it’s really exciting that people can see research potential in the ‘meta’ data about collabroation and interaction amongst academics – it hadn’t occurred to me that there would be this second ‘level’ of value. We need to ensure this information is openly accessible in the way scholars need it. would be very pleased to have your input! Tweet me @martwine or @hypothify

    Reply
  7. Pingback: The Open Science Debate: Part 1.5 – open access roundup – Stanford Neuroblog

  8. Robert Gledhill

    Martin, here’s a few thoughts, possibly very silly; possibly not even vaguely related to what you have in mind, but here goes:

    * Maybe there’d be milage for linking ‘ordinary’ hypotheses together by special ‘implies’ hypotheses. For example, given two regular ones, A and B, you could have A -X-> B where X is a hypothesis like ‘if A is true then B must also be true’, or ‘if A is true then B cannot be true’. If you did it as a hypothesis, then people could use the system to help bash out whether it is actually right. You could get the machine to stitch in additional infered links, e.g. A -implies-> B -implies-> C therefore A -implies-> C.

    * What about having accounts for people to log into which will allow them to store their own *personal* views on which hypotheses they believe are true. Probably let people keep this to themselves or make this public as they wish. If you also had implication hypotheses in there, then you could alert people to where their views are logically inconsistent, i.e. you believe A and B are true, and yet you also said you believe X is true, where X is ‘A implies B cannot be true’.

    * If plenty of people had their own views for a particular subject area it would be straightforward to cluster people’s beliefs and get an overview of what ‘tribes’ there are of academic opinion on an issue. This would be great mining material for people interested in how science works from a social perspective.

    * [this might be just totally barking mad -> ] on ‘synthify’ type stuff, maybe you could (not at the beginning, obviously) suss out some sort of ‘expertise weighted’ bayesian search type thing for right values where it is appropriate (like what the USN supposedly did for finding their sunk Scorpion sub way-back-when).

    * It would be super-handy for writing lit reviews of subjects to be able to go through the site and button-select a big grab-bag of hypotheses, arguments, data refs and so forth and have the machine spit out a single, neatly assembled hyperlinked document.

    * In the past few years a few people have had a go at writing algorithms that process tags (from websites) to automatically derive an ‘ontology’ of sorts automatically. No idea how well they work, but it might come in handy.

    Sorry if that’s a bit of a long post, but I read what you wrote and got all excited and just couldn’t stop typing. It looks like this idea might just be a way to make researchers *a lot* more effective at getting stuff done. I’m totally with the idea of bringing the _logic_ inside investigations and research programs out where it can be clearly seen, rather than entombed across a span of papers.

    Where do I sign up for alpha test?

    Skol,

    Robert

    Reply
  9. Pingback: Slightly plagiarised! AIBU? | Confessions of a marine chemist

  10. Jessy Kate Schingler (@jessykate)

    a nice aspect of your system design is that because the evidence with the most support goes to the top, it also supports skimming since readers get the most important information first. i also love the fact that users cannot vote on a hypothesis directly.

    i mentioned this in the comments over on my blog, but it would be nice if there was simple markup of each hypothesis body, such that the basic hypothesis object was encapsulated and discoverable by automated parsers, scripts etc. (in the vCard and iCal tradition). imagine if these individual hypotheses could be automatically combined and the evidence for and against each input hypothesis processed by an algorithm to give probabilistic outcomes for certain combined hypotheses – a sort of generative hypothesis engine driven by real world, human-annotated atomic hypotheses. kinda neat.

    Reply
    1. martwine Post author

      Rob, Jesse, thanks for you comments – all the input is very helpful. I’ve been busy with other things mostly for the past week, but also trying to lay the data models down as the basis for the web application. Hopefully we’re looking at a closed alpha test by early summer, given current progress!

      So, in response to your comments:

      Rob – formally linking hypotheses is definitely in the plan. As you say it will highlight logically inconsistent views, but also hypotheses tend to be rather hierarchical – each ‘assuming’ in a hypothesis is a hypothesis in itself so a web of understanding will hopefully precipitate out of the system when there are enough users.

      Personal ‘notebooks’ is cool – will add that to the features wish-list, the grab-bag too.

      Jesse – skimming is totally what this is all about – we can’t all be experts in everything and unless we’re doing research into a very specific and limited and constrained area, then we all very quickly reach expertise deficit one way or another. Let the people who are experts, or who are developing their expertise tell us what the top things to read are for those peripheral topics! Even if we want to become and expert in X, it’s still good to start by reading the most relevant stuff…

      You both mention two very important ideas. Firstly, the ontology of the information encoded. @Afternoon has suggested that we should store (or at least, express) the data using RDF (http://en.wikipedia.org/wiki/Resource_Description_Framework). This flexible, standard format would expose the ontology of the information accumulated on hypothify and effectively provide a ‘read-only API’ without us having to do any extra work – really good for data mining. Jesse – this might be a good solution to your research microformats problem too? I don’t know a great deal about it, and I’m not an information scientist, so I’m going to need to harness some help on this aspect! My plan is to get a ‘normal’ database based web app up and running and we’ll look at RDFing (or OWLing) it in ‘phase 2’!

      The probabilistic assesment of the hypotheses and syntheses conducted by the community is super-sexy and had occurred to me. It may be a ‘secondary’ product of the project, or possible something someone else would like to do with the data. Let’s pencil that in for phase 3 🙂

      *Back to django*

      Reply
      1. Robert Gledhill

        Quick thought: if you haven’t considered them, it might be worth taking a look into ‘triplestore’ type databases. Depending on the smallprint of what you are trying to do, these will be either 1) an elegant solution to a really messy database/graph theory/efficiency problem or 2) a mind-mangling cursed tarpit from the 5th ring of computer hell.

        en.wikipedia.org/wiki/Triplestore

  11. jessykate

    super exciting! i’m a software person and reasonably familiar with RDF. since that original post on microformats for science, RDFa (a subset of RDF) and microdata standards have come a long way. i’m working on a post that explores some of the markup options but definitely RDF and/or microdata are the directions i’ve started leaning towards. i’d be happy (interested, even!) to help with incorporating RDF-type markup into hypothify, or even help write some of that code, if that would be useful (plus i think it could be an important test case for the whole idea of science markup). feel free to rope me in or share code on github… github.com/jessykate

    Reply
    1. martwine Post author

      Jessy, wow, sounds like your just the girl for the job. I’m gonna leap on that offer with a thousand thanks! Do you envisage it being a matter of marking up data properly in the templates or is it a case of re-thinking everything from the database up? I know how to get a mysql driven prototype up and running quickly using Django so I’ll keep on with that – and github it as soon as I get a chance so you can have a poke around and make some suggestions. Thanks ever so much 🙂

      Reply
      1. jessykate

        yeah, microdata and RDF(a) are meant for markup (in the templates). of course it doesn’t hurt to think through a good back end storage schema too 😉 but as long as you can send the data to the templates then we can mark it up with a bit of semantic goodness.

        fun, i’m excited to have a test case. i’ll shoot you a link to the microdata post when it’s up, and look forward to seeing you over on github :D.

        btw, there is an open science hack day on the 31st, being coordinated by OKFN in london. i have some info up about it here: http://superhappydevhouse.org/w/page/51749453/OpenScienceHackDayMarch312012. i’ve been planning to hold court over in california and see if i can find some others interested in hacking on open science stuff. i’m planning to work on the microdata markup that day. you’re probably closer to london time, but maybe we can coordinate in IRC that day:D (and if not no worries!).

  12. Jan Paul Posma

    Hey Martin, Just came across this post, and it’s very, very close to what we have been building at Factlink over the last couple of years. Would love to talk to you (and anyone else reading this)! I hope this is not too much of a “shameless plug”, but I strongly believe collaborative efforts are hugely beneficial. 🙂
    – JP

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s