crowdresearch.stanford.educrowdresearch

Main Page From Jump to: navigation , search Welcome to the Crowd Research collective wiki, we're a group of one thousand designers, engineers, crowd workers, and crowd requesters from around the world who are building Daemo - a self-governed crowdsourcing marketplace. NEWS/UPDATES Our meta paper on Crowd Research got accepted at ACM UIST 2017. Learn more about the Crowd Research Initiative . Our paper on Guilds got accepted at ACM CSCW 2017. Our paper on Boomerang got accepted at ACM UIST 2016. Crowd Research students have gone on to undergraduate and graduate programs like MIT Media Lab, Cornell University, UC Berkeley, Carnegie Mellon, more... Daemo crowdsourcing marketplace has been successfully used to create a dataset of 100,000+ question­ answer pairs on 500+ Wikipedia articles — a paper based on this work recently won the best paper award at EMNLP 2016 . Winter 2016 applications are closed now, check back again later. Welcome, Winter 2016 batch. Inviting applications for the winter session, aspiring researchers and hackers apply now . To help you get started and understand what we've done so far, we've created this -> " Introducing Crowd Research Initiative and Recap - Winter 16 " (recap and overview page), where you can learn more about this project. Our first version (Daemo) is live on daemo.stanford.edu/ . We're finalists (top 20 of 1,000+) at the Knight Challenge 2015, check out our proposal and video for the brief overview of Daemo. Our ACM UIST'15 got accepted, read it for more details. Some research projects are too big and too important to tackle alone. Sometimes, we need to team up. At Stanford’s Computer Science department, we've observed that people who are aiming to get research experience or launch their research career will often fall into an expertise valley. Undergraduates are assigned extremely tightly scoped activities within research projects, getting little room for creativity. Then these folks get into PhD programs, have literally the entire space of human knowledge to explore, and don’t have enough scaffolding to make quick progress. We're going to create a crowdsourced research team to tackle both these challenges together. We’ll gather as many talented folks as we can get, and work to build out that intervening bridge between tightly-scoped work and open-ended exploration. It will have far more flexibility than a typical research experience, but with a focused goal where we can bring each other back on track each week and nobody gets lost. Contents 1 About the project 1.1 About Daemo - a self-governed crowdsourcing marketplace 1.2 About Crowd Research initiative 2 Affiliation policy 3 Participant contribution 4 Spring 17 meetings and slides 5 General weekly plan 6 Skill requirements 7 Resources 8 About this wiki 9 Contact crowd research admins About the project About Daemo - a self-governed crowdsourcing marketplace Whether you need help gathering data, labeling machine learning training examples, running experiments, or transcribing audio, today we use crowdsourcing platforms such as Amazon Mechanical Turk . However, these platforms are notoriously bad at ensuring high quality results, producing respect and fair wages for workers, and making it easy to author effective tasks. It’s not hard to imagine that we could do better. This research will be a complete design, implementation, launch, and evaluation of a new crowdsourcing platform. What would it take to create an effective marketplace? One where workers have more power in the employment relationship, or could take additional responsibility for the result quality? How might we design such a market? Could we launch it and become the new standard? This research in human-computer interaction will involve a combination of design thinking, web development, and experimental design. This is far more ambitious than your typical project. It’s an entire marketplace design question. Thus, we’re banding together to solve it. About Crowd Research initiative Crowd Research is an experimental initiative by professor and students at Stanford to recognize the potential of upcoming student researchers and exploring the possibility of massive research collaboration between the two. It is a first of its kind research-at-scale initiative in the world. We're a group of about 500 students, professionals and researchers from all around the world, working together by engineering, building, designing and making Daemo a reality. We call ourselves, " Stanford Crowd Research Collective ". The project is organized by Prof. Michael Bernstein at Stanford HCI and his students. Affiliation policy When you write emails or update LinkedIn profiles/resume or cover letters or statements of purpose, ​*please be careful about how you state your affiliation and your contributions*​. Other faculty or people can find it very confusing when folks say they "work with Michael Bernstein at Stanford”, because it implies you’re a student here. It is actually hurting the people who do that, because when the faculty or other people realize you’re not actually here, they don’t trust the other things you say you’ve done. It also hurts Prof. Michael Bernstein's Stanford students, because people no longer assume they’re actually at Stanford. I want to make sure you can leverage your hard work here, and carry all the benefits but avoid making those faculty or other people confused and cynical. I suggest the best route is to make sure not to imply that you’re a Stanford student, and instead that you’re part of our crowd research effort. I suggest saying something like “I’m a member of the Stanford Crowd Research Collective, a worldwide group of researchers led by Michael Bernstein in Stanford CS. I’ve worked directly with Michael and other crowd researchers on [your contributions]”. If you're sending an important email, you can either cc me on the email, or directly suggest that they email me for verification on your contributions. That way I can speak up to reinforce it. Participant contribution Well, first, there’s creating a crowdsourcing market that becomes the new standard. This could lead to a far better future for crowdsourcing and crowd work, and millions of people could eventually use it. It’s research, of course, so there’s always risk it might not work out — but if we knew it would work, it wouldn’t be research! Second, we’ll be planning papers to top-tier conferences based on our work. If you are considering an MS or especially a PhD program, being a heavily contributing author on a paper can greatly improve your chances. How much you contribute to the project will determine author order. Last, I really do hope to build relationships with a diverse range of researchers. Spring 17 meetings and slides See Archives for past meetings - from Spring 2015 to Winter 2017 . Spring 2017 Meeting 1 : Youtube link of the meeting today: watch Youtube link of the meeting today: watch Youtube link of the meeting today: watch General weekly plan We’ll meet weekly over videochat and lay out our goals for the next week. At the end of the week, you’ll submit what you’ve been working on. Your peers and a Ph.D. student here at Stanford will peer critique the work, and we’ll talk about the best stuff each week in our meeting. The sky’s the limit. I’m sure we’ll adjust this as we go. Because, this entire crowdsourced research idea is a bit of a research project in itself, too. Monday morning 9 am PST : Prof meeting with participants and milestone set for the next week (over Google Hangout on Air) Monday after meeting - Sunday 8 am PST : participants work on their milestones (~6 days) - Not undertaken at the moment. Post Sunday 8 am PST - Sunday midnight PST : peer-evaluation by the participants (12+ hours). This is occasional, not undertaken at the moment. Monday morning 9 am PST : Prof meets based on the input from RAs and top submissions. Participants receive their next milestone and a feedback survey after every meeting. Skil...