Protyping Peer Assessment for Webmaker

For quite some time now we have been talking about the idea of integrating some form of peer assessment and feedback into Webmaker, as a way to level up an individual user's craft and community.  Today I want to share some ideas that Chloe Varelidi and I have been tinkering with and starting to actually prototype with Atul Varma.  I am going to do a little walkthrough of a potential user experience. Keep in mind that some of this is probably wrong, but I want to put it out there so that we can work on polishing those bits.

Imagine that you came to Webmaker and you were looking for a way to learn or gain specific skills. You click on the "Skills" tab (I'm sure there are tons of other names that could be more appropriate for this navigation item). On the skills page you see all of the possible badges that you can apply for - and these badges in the future will not just be individual small badges, but represent larger learning pathways. 

If a user clicks on a badge - they will learn about the badge through a playful interactive displayer and have the ability to immediately apply to earn that badge. 

When the user clicks "Apply" they are given the apply interface. This reminds the user what the criteria is for the badge, gives them the ability to upload evidence and write reflective thoughts on their process.  There are several things that we could do here, one idea was to have the user create a "how to" showing how they made their project, which eventually could be integrated into some kind of maker gallery.  What's great about this approach to "badge pledge" is that it is a useful  interaction in its own regard, and not existing solely for assessment/recognition purposes - that way there's both extrinsic and intrinsic incentives involved.

The user then submits their badge application. There are several directions that we could go from here. One option is to have all of the entries go into a queue that is accessible for mentor reviews from a secure interface that they will be able to access by logging into the queue through a mentor username and password to the site.

Another option might be to completely make this public and leverage a service like twitter. I imagine that the interaction here (after a user signs in and then clicks submit) could be similar the Army of Awesome (see below) where a the applications compile in a tweet queue and then peers pick up the individual applications to review. If we went in this direction, some more sophisticated flow will need to be developed so that there is a way to in essence vet the quality of peer review.

 

The actual application review for a mentor might be something like this:

Here, the reviewer can see the badge applicant's evidence and reflection and then provide feedback as well as recommend the user for a badge. One of the ideas that we had for this particular touchpoint is to create a feedback widget that encouraged constructive feedback. Taking a cue from the Lifelong Kindergarten Lab Chloe actually helped us to work on our internal tools for feedback at Mozilla meetings. Since we are often typing on etherpads during calls, we organize our feedback into categories. We added the blue one to the model to encourage collaborative making.

• Green (what's awesome)

• Yellow (what I am unsure about)

• Red (what's not working)

• Blue (an idea that I have)

Here is a screenshot of one that we recently did:

Since this has been a successful prototype for us internally, we thought that we could take this concept to a feedback widget. We were thinking of something like disqus that could be modified by different users to suit their own needs and own kinds of evaluations (imagine evaluating using just emoji's ?!!!)  The idea is that a mentor is able to click on the different colors and then is given prompts to help them to construct their feedback. In this moment they are able to recommend the badge as well. There are individual rubrics that we could create for the individual assessments as well, which is not shown.  Here is the sketch for this general idea: 

 

After this, let's imagine that the user was recommended for a badge. They are then sent a link (via twitter, via their Webmaker account -- depending on what direction we take above) that would look something like this:

What is worth noting here is that we include options for the user to send thanks or to rebound the feedback. This sets us up for beginning a dialog as well as using this touchpoint to evaluate the evaluator. This is a crucial step to the user flow, since it helps mentors improve their feedback, which is a skill in and of itself. There is some work that still needs to be figured out here in terms of what the interaction is if a user is not given the badge. We want that to be a positive experience. 

Chloe made this great stop motion video to quickly go over the overall interaction that I described:

Wondering what the next steps are here? Well, I am happy to tell you:

1. Prototype: We are going to use the Chicago Summer of Learning work to prototype a version of some of this interaction and then iterate on that to develop some of this flow out for Webmaker.

2. Identify mentors: we need to identify how mentors are going to play into this work flow for the Summer campaign. I know that we are going to have an active, dedicated community, however the decisions that we make around their involvement will impact some of the choices that we make moving forward with the assessment interaction design.

3. Work with the Webmaker, Mentor and Badges teams to make sure that we are all collaborating on this work.

Badges, Assessment and Webmaker+

I am going to talk to you about some of my thinking around badges, assessment and the often talked about, but never before explained concept of Webmaker+.  

When we talk about the tools and resources that we are building at Mozilla we often contextualize them as learning experiences. So, for example, you might go to Webmaker.org - find a project- make and publish that project and along the way gain some basic HTML skills. However, in our current model, you could very well walk away after publishing your project and have no realization that you just learned something. And- because of that, you might not be able to level up your tinkering to craftsmenship.  Check out the vidcast to hear more:

(and based on the unexpected fanclub I've gained from my singing on the last vidcast, I have some special singing action here for you today)

In the vidcast, I showed 4 wireframe iterations for incorporating badges and embedded assessment into Thimble.

Iteration #1:

I started to imagine the badges appearing as you accomplish various things, taking a cue from the Cheevos system. It would look something like this ^^ Cons: the badge is super intrusive and could be distruptive to the learners experience. Pros: the actual "ingredients" displayed in the badge

Iteration #2:

 

 

Imagine the orange tab subtly popping up almost as if it is coming out of the toolbar. You then have the ability to transition to the next slide where you can add the badge to your backpack and share via social media. But is this too subtle?

Iteration #3

 

badge icon in toolbar wiggle when you receive badges (nothing pops up till you click it)

 

on click, you can see your badges- so if you have earned 4 badges- it will appear as a slide show 

Iteration #4: Webmaker +

As you see here, we could add a nav bar that gives a user the ability to opt in to the community, once they join, they are essentially signing in to be part of the game. It then becomes almost like an exclusive layer that sits on top of the software- which by the way could be thimble and popcorn but also community designed tools. The badge here could actually reflect data that is unique to the user. So for example, if they indicated that they were trying to achieve the goal of learning HTML and earning the uber HTML basics badge, then the badge could reflect that progress as well.

Extra Iteration:

With Webmaker+, the badge backpack gets transformed into a user dashboard, where they could see their progress, but also control it- so they can set goals, join challenges, and add badges. 

This is an initial brainstorm. To get a better understand of what we are thinking about specifically at Mozilla in relationship to the badges that we issue, head over to Chloe Varelidi's blog ( Chloe, did the map I featured in the video). We have been talking about this webmaker+ layer for quite a while and have been letting the ideas percolate. 

In terms of feedback, I would love to hear from your thoughts, but specifically:

1.  If we were to implement the pink iteration (webmaker +) would the "dashboard" (last image above)  be something that lives uniquely on the webmaker site and leverages the Mozilla webmaker community, or would it be a more open ended tool that anyone, anywhere could tap into that gets featured and promoted through the Mozilla Webmaker site?

2.If you had the ability to curate your online learning and making - what kinds of stats would want to track?

Badges: Level Up your Webmaking Skills

Badges are huge topic in the digital media and learning world and something that I have been working with in some form ever since I have entered into the realm of educational technology. To be honest, I have struggled with my feeling towards badges, that is to say, assessment for skills that you aren't always given credit for.

The Mozilla Open Badges Project describes badges on their wiki:

Badges give you public recognition for your skills, achievements and learning beyond the classroom -- unlocking job and educational opportunities.

The reason that I struggle is that I am concerned that badges allow you to formalize something that is blissfully informal. One example of my fear around this was when I was in Barcelona at the Learning, Freedom and the Web festival. My colleague Jack Martin and I participated in this local learning incubator where we told a story with twitter. It was a fantastic and fun day and we loved what we made just as much as we did making it. However, after the activity was over, a learning assessment team came over during our presentation of our story and gave us badges for our story. It somehow cheapened the experience that Jack and I had and sort of reminded me that, yeah this was about learning and grading-not the fun experience.

Ok, now that I have that off my shoulders- I find the idea of using badges to assess web design skills intriguing. Many web designers and developers learned webmaking skills in informal settings- hanging out with friends, navigating the web on their own, tinkering with blog sites. However, theses skills equate to real life work competencies that often need to be proven in some form when you are applying for work (for example). What if there WAS a way to say, bring along some form of street cred and accreditation into your interviews along with your portfolio?

So with all of this enthusiasm, and mild trepidation- I started to look at the learning objectives for Hackasaurus and, with the help of Matt Thompson and Atul Varma, started to answer the question: if Hackasaurus had a system for badges what would the badges look like?

Essentially I came up with 5 core badges and then 3 "level up" badges. The level up badges are really geared towards refinement and mastery, whereas the core badges are a compilation of hard and soft webmaking skills. These badges can be mapped to a structured and scaffolded curriculum, or be achieved in a more organic (non-linear) way. What will be an interesting mechanic for the project is the dissemination of the badges. Right now, we are thinking that a learner can acquire a badge physically in a design jam or online through some sort of self guided experience

The badges I came up with are described here. It's just a barely baked idea, and Atul, Matt and I are really just starting to tinker with how badges could support the learning experience. We won't be actually implementing this online for a little bit, however, I would love to hear your thoughts on how this maps to the learning outcomes and if you have any other ideas. The Tuesday Hackasaurus community call will focus on badges. Join us if you have the time.