Monday, April 8, 2013

Protyping Peer Assessment for Webmaker

For quite some time now we have been talking about the idea of integrating some form of peer assessment and feedback into Webmaker, as a way to level up an individual user's craft and community.  Today I want to share some ideas that Chloe Varelidi and I have been tinkering with and starting to actually prototype with Atul Varma.  I am going to do a little walkthrough of a potential user experience. Keep in mind that some of this is probably wrong, but I want to put it out there so that we can work on polishing those bits.
Imagine that you came to Webmaker and you were looking for a way to learn or gain specific skills. You click on the "Skills" tab (I'm sure there are tons of other names that could be more appropriate for this navigation item). On the skills page you see all of the possible badges that you can apply for - and these badges in the future will not just be individual small badges, but represent larger learning pathways. 


If a user clicks on a badge - they will learn about the badge through a playful interactive displayer and have the ability to immediately apply to earn that badge. 


When the user clicks "Apply" they are given the apply interface. This reminds the user what the criteria is for the badge, gives them the ability to upload evidence and write reflective thoughts on their process.  There are several things that we could do here, one idea was to have the user create a "how to" showing how they made their project, which eventually could be integrated into some kind of maker gallery.  What's great about this approach to "badge pledge" is that it is a useful  interaction in its own regard, and not existing solely for assessment/recognition purposes - that way there's both extrinsic and intrinsic incentives involved.


The user then submits their badge application. There are several directions that we could go from here. One option is to have all of the entries go into a queue that is accessible for mentor reviews from a secure interface that they will be able to access by logging into the queue through a mentor username and password to the site.
Another option might be to completely make this public and leverage a service like twitter. I imagine that the interaction here (after a user signs in and then clicks submit) could be similar the Army of Awesome (see below) where a the applications compile in a tweet queue and then peers pick up the individual applications to review. If we went in this direction, some more sophisticated flow will need to be developed so that there is a way to in essence vet the quality of peer review.


The actual application review for a mentor might be something like this:



Here, the reviewer can see the badge applicant's evidence and reflection and then provide feedback as well as recommend the user for a badge. One of the ideas that we had for this particular touchpoint is to create a feedback widget that encouraged constructive feedback. Taking a cue from the Lifelong Kindergarten Lab Chloe actually helped us to work on our internal tools for feedback at Mozilla meetings. Since we are often typing on etherpads during calls, we organize our feedback into categories. We added the blue one to the model to encourage collaborative making.
  • Green (what's awesome)
  • Yellow (what I am unsure about)
  • Red (what's not working)
  • Blue (an idea that I have)
Here is a screenshot of one that we recently did:


Since this has been a successful prototype for us internally, we thought that we could take this concept to a feedback widget. We were thinking of something like disqus that could be modified by different users to suit their own needs and own kinds of evaluations (imagine evaluating using just emoji's ?!!!)  The idea is that a mentor is able to click on the different colors and then is given prompts to help them to construct their feedback. In this moment they are able to recommend the badge as well. There are individual rubrics that we could create for the individual assessments as well, which is not shown.  Here is the sketch for this general idea: 


After this, let's imagine that the user was recommended for a badge. They are then sent a link (via twitter, via their Webmaker account -- depending on what direction we take above) that would look something like this:



What is worth noting here is that we include options for the user to send thanks or to rebound the feedback. This sets us up for beginning a dialog as well as using this touchpoint to evaluate the evaluator. This is a crucial step to the user flow, since it helps mentors improve their feedback, which is a skill in and of itself. There is some work that still needs to be figured out here in terms of what the interaction is if a user is not given the badge. We want that to be a positive experience. 

Chloe made this great stop motion video to quickly go over the overall interaction that I described:



Wondering what the next steps are here? Well, I am happy to tell you:

1. Prototype: We are going to use the Chicago Summer of Learning work to prototype a version of some of this interaction and then iterate on that to develop some of this flow out for Webmaker.

2. Identify mentors: we need to identify how mentors are going to play into this work flow for the Summer campaign. I know that we are going to have an active, dedicated community, however the decisions that we make around their involvement will impact some of the choices that we make moving forward with the assessment interaction design.

3. Work with the Webmaker, Mentor and Badges teams to make sure that we are all collaborating on this work.

13 comments:

David Gibson said...

I love Chloe's stop action demo. I like the spin-to-explore and the enlarge-to-interact aspects.

Elyse Eidman-Aadahl said...

Hi Jess, What I like about this—far beyond Webmaker—is the way that the badge itself conveys information about the (essentially) curriculum embedded in the badge, giving the learner more information that could help him/her decide to pursue it or not and helping the learner see the path and relationship among badges.

arasbm said...

Hi Jess,
I love the interaction ideas that take badges and add new affordances to them. It seems very intuitive and fun way to explore them. I think there are lots of potential to integrate aspects of the assessment stories into the badge backpack. Looking forward to see how this project unfolds.

Anonymous said...

Thank you, I have recently been searching for information approximately this subject for
a while and yours is the greatest I have came
upon so far. But, what about the bottom line? Are you
sure concerning the supply?

Check out my web site graduate certificate programs online

Anonymous said...

Incredible! This blog looks just like my old one! It's on a entirely different subject but it has pretty much the same page layout and design. Superb choice of colors!

Also visit my weblog - Gang Bang Videos Videos

Anonymous said...

I'm not sure where you're getting your info, but great
topic. I needs to spend some time learning more or understanding more.
Thanks for magnificent information I was looking for this information for my mission.


My web-site ... http://sexvideovoyeur.com

Anonymous said...

This paragraph is genuinely a good one it helps new net
people, who are wishing in favor of blogging.

my blog post - Tennis 2013

Anonymous said...

I visit daily a few sites and information sites to read content, however this web site presents quality based writing.


Here is my blog; maquillage pas cher

Anonymous said...

Excellent web site you have got here.. It's hard to find high-quality writing like yours these days. I really appreciate people like you! Take care!!

Also visit my web-site :: Devis Travaux

Anonymous said...

hi!,I like your writing very much! percentage we communicate extra about your article
on AOL? I require an expert in this space to solve my problem.
May be that is you! Taking a look ahead to peer you.



Also visit my site :: www.jnmassage.info

Anonymous said...

For most up-to-date news you have to visit world-wide-web and on
internet I found this site as a best website for
newest updates.

Also visit my blog post :: voyance

Anonymous said...

Awesοme website you have heгe but І wаs wanting to κnow if yοu
kneω of аny disсuѕѕion bοards that coνer
thе same topics diѕcussed heгe? I'd really like to be a part of online community where I can get suggestions from other knowledgeable people that share the same interest. If you have any suggestions, please let me know. Thanks!

my webpage: comparatif ssd

Anonymous said...

research children dating http://loveepicentre.com/advice/ after 40 dating las vegas
malaysian dating website [url=http://loveepicentre.com/]russian dating in culture[/url] mackenzie rosman whos dating who
principals dating staff [url=http://loveepicentre.com/]rules to dating filipina women[/url] catholic christian dating los angeles [url=http://loveepicentre.com/user/7SariTa7/]7SariTa7[/url] online dating popular