Protyping Peer Assessment for Webmaker

For quite some time now we have been talking about the idea of integrating some form of peer assessment and feedback into Webmaker, as a way to level up an individual user's craft and community.  Today I want to share some ideas that Chloe Varelidi and I have been tinkering with and starting to actually prototype with Atul Varma.  I am going to do a little walkthrough of a potential user experience. Keep in mind that some of this is probably wrong, but I want to put it out there so that we can work on polishing those bits.

Imagine that you came to Webmaker and you were looking for a way to learn or gain specific skills. You click on the "Skills" tab (I'm sure there are tons of other names that could be more appropriate for this navigation item). On the skills page you see all of the possible badges that you can apply for - and these badges in the future will not just be individual small badges, but represent larger learning pathways. 

If a user clicks on a badge - they will learn about the badge through a playful interactive displayer and have the ability to immediately apply to earn that badge. 

When the user clicks "Apply" they are given the apply interface. This reminds the user what the criteria is for the badge, gives them the ability to upload evidence and write reflective thoughts on their process.  There are several things that we could do here, one idea was to have the user create a "how to" showing how they made their project, which eventually could be integrated into some kind of maker gallery.  What's great about this approach to "badge pledge" is that it is a useful  interaction in its own regard, and not existing solely for assessment/recognition purposes - that way there's both extrinsic and intrinsic incentives involved.

The user then submits their badge application. There are several directions that we could go from here. One option is to have all of the entries go into a queue that is accessible for mentor reviews from a secure interface that they will be able to access by logging into the queue through a mentor username and password to the site.

Another option might be to completely make this public and leverage a service like twitter. I imagine that the interaction here (after a user signs in and then clicks submit) could be similar the Army of Awesome (see below) where a the applications compile in a tweet queue and then peers pick up the individual applications to review. If we went in this direction, some more sophisticated flow will need to be developed so that there is a way to in essence vet the quality of peer review.

 

The actual application review for a mentor might be something like this:

Here, the reviewer can see the badge applicant's evidence and reflection and then provide feedback as well as recommend the user for a badge. One of the ideas that we had for this particular touchpoint is to create a feedback widget that encouraged constructive feedback. Taking a cue from the Lifelong Kindergarten Lab Chloe actually helped us to work on our internal tools for feedback at Mozilla meetings. Since we are often typing on etherpads during calls, we organize our feedback into categories. We added the blue one to the model to encourage collaborative making.

• Green (what's awesome)

• Yellow (what I am unsure about)

• Red (what's not working)

• Blue (an idea that I have)

Here is a screenshot of one that we recently did:

Since this has been a successful prototype for us internally, we thought that we could take this concept to a feedback widget. We were thinking of something like disqus that could be modified by different users to suit their own needs and own kinds of evaluations (imagine evaluating using just emoji's ?!!!)  The idea is that a mentor is able to click on the different colors and then is given prompts to help them to construct their feedback. In this moment they are able to recommend the badge as well. There are individual rubrics that we could create for the individual assessments as well, which is not shown.  Here is the sketch for this general idea: 

 

After this, let's imagine that the user was recommended for a badge. They are then sent a link (via twitter, via their Webmaker account -- depending on what direction we take above) that would look something like this:

What is worth noting here is that we include options for the user to send thanks or to rebound the feedback. This sets us up for beginning a dialog as well as using this touchpoint to evaluate the evaluator. This is a crucial step to the user flow, since it helps mentors improve their feedback, which is a skill in and of itself. There is some work that still needs to be figured out here in terms of what the interaction is if a user is not given the badge. We want that to be a positive experience. 

Chloe made this great stop motion video to quickly go over the overall interaction that I described:

Wondering what the next steps are here? Well, I am happy to tell you:

1. Prototype: We are going to use the Chicago Summer of Learning work to prototype a version of some of this interaction and then iterate on that to develop some of this flow out for Webmaker.

2. Identify mentors: we need to identify how mentors are going to play into this work flow for the Summer campaign. I know that we are going to have an active, dedicated community, however the decisions that we make around their involvement will impact some of the choices that we make moving forward with the assessment interaction design.

3. Work with the Webmaker, Mentor and Badges teams to make sure that we are all collaborating on this work.

The Value of Showcasing Craft

Lately I've thinking been about how we can value the craft of webmaking within the context of Webmaker.org. The topic is fresh on my mind because we are starting to seriously think about how to develop out a gallery presence for the platform that is true to the spirit of our community. This got me to researching a bit about how other web properties highlight this stage in the design process.

Threadless does a nice job of surfacing the process of individual t-shirt designers on their site. Very simply, they have a curated section of blogposts featuring designers going into detail about how they made the product.

First they profile the user, and then they show great process shots like this one below by Joshua Kemble:

Additionally, they have a critique section of their site, where users can rate in-progress shirts and provide feedback via commenting. Again, super simple and effective. There are so many reasons to do this kind of highlighting of process, but for Threadless, it serves three functions well: 1) as an educational resource that isn't in your face "here is how it is made," but in fact offers many ways to make similar things and 2) as a way to elevate their product- they highlight masters who have done cool work, and show you how to do that too and 3) as a way to build community.

Sketchpad.cc highlights process but in a very different way. In their gallery, instead of just showing the end product, they include the code which can be seen as a video replay. I find it compelling because it allows you to see the different decisions that the creator made when designing the animation. In turn the entire process is treated as a journey. I've seen other sites do similar things, and in a less organic way, a lot of cooking websites do this with their video documentation of their step-by-step tutorials, but I like that it's not really a tutorial- that the verb - or the action is actually an appreciated part of the design- possibly even elevated higher than the end product.

So we have two forms of process documentation so far, and my last site that I am going to talk about is so documentation heavy- that it might blow your mind: Github. Like other version or bug trackers, here you find a community of people who are collaboratively crafting documentation for their project. I like this as an example because you can see the history and different phases of the project.  So on an incremental level you can see how the project was built up, who forked what code, who hacked what - and from time to time, particularly with their "Issues" section of the app- you can see conversations about how decisions were made.

It seems like there is some awesome formula for success here- like 1 part documentation plus 2 parts social plus 1/8 part inspiration, 2 tsps of process and 4 cups design thinking = craft. 
Right now we are still in the research phase of the gallery for Webmaker- but what's clear to me is that it needs to accomplish a few things: it needs to be social, inspiring, educational, live across the web, and very importantly - have some level of showcasing the craft of making things on the web.

Building the Next Generation of Web- Craftsmen

I was reading this article this morning in the NY Times by Louis Uchitelle about the American workforce and how we are losing our value of craftsmanship, and it got me thinking about the word "craft." 

Photo by Michael Falco of the NYTimes

We are at a stage where we are building up the contents of our "toolbox" at Mozilla, figuring out ways to sharpen those tools. We have Popcorn, X-Ray Goggles and Thimble- as well as learning projects that can accompany those tools within the context of the tool or in the form of a D.I.Y. recipe.  I am wondering, however, in the long run- how are we going to turn web-hobbyists into web-craftsmen (women, girls and boys)?  We are designing badge systems and trajectories, for our learning content at Mozilla,  which will reveal to our users pathways for further exploration, but in project based learning, I wonder what that extra secret ingredient is that will empower someone to go above and beyond and refine their work to the point that it is a craft.

Implicit in the word craft is some understanding of design, whether that be design of the system, the tool or the concept. In fact, the American Craft Museum (my former employer) has now changed their name to the Museum of Arts & Design. To some degree, the name change was because the term "craft" has lost its relevance in our society- people think of old women crochetting doilies when they hear craft. However, more and more I hear people reclaiming the word and using it as a brand advantage- something that was made by hand. The Renegade Craft Fair,  encourage local artisans to connect with eachother and show (and sell) their wares. Two key takeaways that I have from being a participant in their events is 1) a emphasis on hand- made design and 2) a celebration of community. I believe, these might be the two ingredients required for craftsmenship.

Do not call Atul Varma a codemonkey!

Design is a concept that extends far beyond "coding" when we talk about it in the context of webmaking. When someone creates a webpage, for example, a lot more goes into it than just implementing it- there is the concepting phase- where you brainstorm and make decisions on what and why of your page, the asset creation stage and finally the implementation stage.  It's the big picture.  The developers who I work with at Mozilla understand this. In fact, Atul, my thought partner at Mozilla, and co-developer of Hackasaurus hates it when people refer to him as a "code monkey" (someone who only inputs code)- and while I would only ever say that expression in jest,  the reason he gets so offended is that he is a craftsman. He researches, he tinkers, he looks for a community and ultimately, shares his work within that community. In short, he designs, develops and contributes to a community.

Learn about how much character (get it?) is inside your browser!

In the article, the author mentions how people know how to use their computers but they don't know how to build them- which got me starting to think about what the appropriate analogy is for the work that we do- because yes I could force teens to take apart their computers - but I think that doesn't resonate in the same way that it did for us in the 80s/90s. I think that the canvas for today is not the computer, but the browser- and I feel that deconstructing that browser will help learners understand what is possible- and why it is important to refine your craft to the point that your work is not just digestible data, but beautiful content that is accessible. Atul and I started to dabble with this idea using the Navigator badge concept- but I would love to push that further, and not think about the assessment, but the lessons here.

Hive NYC has developed a community of practice amongst professionals and learners

The community piece is a bit tricky because there are so many ways for us to engage communities. One way, is to build communities of practice with professionals and youth at like minded organizations- which is what we are doing with Hive NYC at Mozilla.  Another way would be to meet youth where they already are, which is kind of the beauty of modular concepts like Open Badges- because it wouldn't matter what circle a learner is based in, because they would be able to show their skill through an understood social currency that is backed up by a portfolio of work. Then again, there are other ways, like imagine mixing the making piece within the context of a community of online practice- like hacking facebook to be the best facebook within facebook- with your facebook peers (oh yeah, that sounded like Inception!)

I often say that the web is a handcrafted community and medium- one that was created by individual people who work to make it reflect of our own local interests. If we are to elevate the level of innovation and representation within the web, we need to develop a community of web- craftsmen. That means providing tangible learning tools and projects, trajectories that implicitly incorporate design thinking and integrating with communities of practice.