thing 4: Assessment Overview

Reading

Oakleaf, M. (2014). A roadmap for assessing student learning using the new framework for information literacy for higher education. The Journal of Academic Librarianship, 5(40), 510-514.  http://meganoakleaf.info/framework.pdf [pre-print]

Activity

Please answer the following questions below in a comment:

  • Which of the steps on Oakleaf’s “Roadmap” has your institution completed? Describe a project or partnership that has helped you complete this step.
  • Which steps from Oakleaf’s “Roadmap” will take more work to complete? Identify one next step that would help you make progress toward incorporating the Framework into your assessment work.

27 comments

  1. I think the steps presented in this article give folks who need guidance a well thought out path to follow. Additionally, for people that may have felt less “stuck” by the Framework, it provides a structure they can use to articulate the work they may have done.
    However, I am personally still not convinced that the Framework requires a complete overhaul of what many librarians might already have in place. For example, I reviewed Learning Outcomes that we already had written, and they were in-line with the Framework. If I am already overhauling something, I will start from scratch with the Framework in mind, but I find that my end product isn’t radically different from what I would imagine otherwise. So, while I find the content of this article useful, I think it is because this is good teaching practices, and Oakleaf addresses this with the article’s closing line “The inspiration may have changed, but the road—as circuitous as it is—is well traveled and there are plenty of guides to follow.”
    I would guess that we are on Step 8, if I had to commit to one, and I would guess that most librarians will struggle with Step 2. I think many library schools lack in preparing librarians for instruction, so they are missing ground work like how to write a learning outcome.

    Like

  2. I think our library is still at the “get inspired” stage — step 1. As I was reading these posts, we got another email from our university president announcing another round of budget cuts to education in Kentucky. Our institution has been preoccupied with political turmoil but we’re trying to focus on the job we wish to do. There is so much turnover among our key players right now — trying to focus on what our organization will look like and do.

    Like

  3. As I read the article, it occurred to me that I’ve reordered/blended some of Oakleaf’s Steps. My inspiration (Step 1) comes from my primary charge – to completely revamp our library instructional module to meet the requirements of the Framework.

    Oakleaf’s Step 3 is actually Step 2 in my process. I’m actively engaging with the teaching faculty (or trying to) via a survey and a short meeting with questions. On the survey, faculty are asked to identify the
    Frames (threshold concepts) that align most closely with their coursework and departmental curriculums. Once this work is done and the results have been codified, I will return to Oakleaf’s Step 2 as my Step 3 and convert the most commonly-chosen Frames to learning outcomes and move forward with the remaining steps in Oakleaf’s outline.

    With regard to Step 4, I have started a rough sketch of a student assessment tool based on ideas in articles I read earlier this semester. My supervisor, the library director, asked me to create a roadmap for the instructional revamp, which I have done, so part of Step 5 is completed and/or in progress.

    In other words, I’m jumping back and forth among the steps that Oakleaf has outlined. My timeframe for this project is limited, so as I finish one step or come to a logical stopping point, it makes sense to move to the next logical point in the process; and that point may not match up with Oakleaf’s outline. This isn’t a good thing or a bad thing; it simply is what it is – my work process.

    Like

  4. As a unit my library is probably at Step 1. We librarians are inspired to develop our instruction program and have begun to have instruction meetings. As individuals we already write outcomes for our sessions.

    Step 3 would move us forward. Getting stakeholders together is a challenge for large, decentralized institutions such as mine, yet how can we write programmatic outcomes if we don’t know our stakeholders’ needs? Further, many higher ed institutions (mine included) are trying to define themselves and their needs in the current environment.

    That said we have a seat at many university-wide tables (committees, etc.). We can continue to take those seats. As the university determines its needs we’d show that we are ready to help.

    In the meantime we can continue our work with our instructors. They are the stakeholders with whom we can coordinate our efforts here and now. We can work with our faculty both on a session-by-session basis and as departments/programs.

    Then we can share these individual efforts at our instruction meetings. We’ll see common patterns, which might help us develop some common outcomes.

    Overall I find it interesting that these steps are called “steps,” which suggests a linear progression. As noted above, Step 3 can certainly influence step 2 and vice versa. Also, wouldn’t “knowing it when you see it” (Step 8) also impact the writing of outcomes? Perhaps the real key is closing the loop every step of the way.

    Like

  5. This past fall semester we were integrated into a new freshman course, Inquiry Seminar, where each student learned to read, write, and speak about a topic they were interested in pursuing. Librarians were part of the ongoing process in each course with an introduction to what the library had to offer, how to find the research needed, and how to evaluate the research. The library sessions were a “flipped classroom” model where the students needed to watch a series of videos and answer questions before they could proceed. According to the Roadmap, we have used all of the steps to some degree but this is just within one course, although it is one that lays the groundwork for further research needs of the students

    Because we are working with a wide variety of professors in various disciplines, we are not exactly in control of the class. This gives us limited time to carry out what we have agreed upon as key concepts. We are looking beyond this Inquiry Seminar course, knowing all students will come to us in their next research course having the same background, or exposure to it. At the same time we are teaching sessions with students that are now juniors and seniors who have not had this same beginning so it is a challenge to run two systems at the same time, keeping the framework in mind while leaning away from the standards. We are in the beginning stages and are somewhat optimistic of the results but can still modify any part of the process as needed.

    Like

    • Karen,
      Do you folks use an LMS, where the flipped videos could be made available for a wide variety of courses? Do transfer students have to take Inquiry Seminar as well?

      Like

  6. At our institution, we have done steps 1 and 2, and are in the process of going through step 3. It’s been an interesting journey because we’ve been in the process of putting together an information literacy program for the whole college and seminary (the library serves both on our campus). We tried putting together a working committee with teaching faculty to come up with how this would look, including turning the frames into measurable outcomes. However it wasn’t long until no one was able to come to the meetings because of busy schedules. So low and behold, the library went at it by themselves. I took the bare bones of what we had come up with and have been crafting the learning outcomes over the past year. That along with our information literacy plan has been put before the faculty and we are waiting on feedback. In the meantime, I have already been using the outcomes to shape the lessons that I teach, so at least we are getting some practice in that.

    The step that will take more work is Draw the Map. We have a small staff of librarians here and some of us are already doubled up on the schools that we are liaison to. We are going to have to break this up into chunks and take one step at a time. One of the first goals to tackle in the information literacy plan is incorporation into the General Education program. I think that is a reasonable goal to start with, and something I’ve already been doing a little bit of, because I teach every semester for classes in the program.

    Like

  7. Our library has instituted pre and post tests for information literacy instruction as an assessment tool. Participation by librarians is voluntary and is not tied to the Framework at this time. We have however, come up with a Library Assessment document that looks at all services, including information literacy outcomes.

    Because our library has yet to take on the Frames, the first step is to identify a Frame on which to focus. The English department is our biggest customer therefore, I would start our disciplinary faculty partnership with them, as we have in the past with the development of the information literacy outcomes rubric that is currently being used the department..

    I think step #9 “Roll it Up & Report It Out” is the most difficult step because it is the end result of having gathered data from a variety of sources.

    Step #2 “Bite the Bullet” by identifying learning outcomes and choosing a formula to articulate said outcomes, would be the next step after choosing a Frame as a foundation for all the work that follows.

    Like

    • Jackie,
      You make a good point about choosing a frame as a focus. On the one hand that can be a challenge because the frames sort of interconnect. On the other hand the interconnections are helpful because you can still relate many concepts to the chosen frame.

      I like the idea of identifying a biggest customer. My institution could do that to move forward.
      –All my best,
      Maureen

      Like

  8. I think we’ve done pretty well with steps 1-3. Most of my teaching isn’t based specifically on the framework but I have incorporated the framework into expressed learning outcomes of the professional programs. The one area that isn’t as specific is with the general undergrad degree. I see potential to have more autonomy in developing those learning outcomes if I can get more time with them.

    I think “Draw the map” will be one of the most difficult areas to address. Right now, we are planning a curriculum mapping that I hope will open doors for library services and instruction but I don’t envision that it will be specifically library related. The focus is on clinical patient care and many clinicians don’t see how the library knowledge can help in patient care. My goal is to be as present as I can be while the mapping is done. I’m hoping instructors will see gaps in their curriculum regarding evidence informed practice and I can help them develop activities to keep those skills alive.

    Like

  9. I think I can safely say we have completed two of these steps, however working in a smaller community college with only two librarians delivering instruction we haven’t really had to struggle with getting on the same page. We do have to comply with a state system set in place where we have to meet certain learning outcomes commiserate with 16 other community college libraries’ instructional outcomes. Those are agreed on at a higher level and represent the minimum outcomes of the instruction we deliver. Those outcomes are Standards based and cannot be easily changed to reflect the Framework until agreed on by certain administrative powers and a consensus among the library directors.

    Using the required system wide learning outcomes for library instruction I have created a rubric which links those to certain frames and adds learning outcomes I would like to implement at a local level. As the article mentioned, this was not a difficult feat, I have a pretty clear vision of what the information literate student should look like upon completing the highest credentials offered by our college, as that would ideally provide them with the greatest exposure to library services and instruction opportunities. My director has been very supportive not only in mapping our system mandated learning outcomes to the Framework but also in experimenting with our instruction and assessment styles to determine best practice for instating this mapping. Support from the director, and a certain level of freedom to experiment, seems imperative to successfully implementing any part of the Framework or any change to the instructional paradigm of an institution.

    Being a new employee, I began working here at the start of the year, I have had to depend on her existing relationships with faculty to get some professors on board with a change in how their usual library instruction is delivered (like using a flipped classroom and active learning activity instead of relying solely on a PowerPoint delivered lecture). Encouraging faculty to participate in implementing the Framework based instructional activities and learning outcomes will most likely be the most time consuming step on the Roadmap (step 3) just because faculty do not regularly take advantage of library instruction. One instructional session a year is not enough to really connect with these instructors to map the framework to the learning outcomes they would prefer and getting feedback on our general information literacy rubric has been nearly impossible. They seem to defer to me as an instructor though I have no hand in designing their curriculum and do not see their students enough to develop any kind of relationship or a full understanding of where they are literacy-wise. Building relationships with faculty, and then earning their trust to really collaborate and streamline instruction to meet their specific students’ and curriculum’s needs will be the biggest but most rewarding step towards embracing the Framework.

    Like

  10. Which of the steps on Oakleaf’s “Roadmap” has your institution completed? Describe a project or partnership that has helped you complete this step.
    I feel that my library has gotten in the zone of acceptance(but has also gotten a tad scared with 1 librarian) about the six frames. The previous standards were nice little boxes that we could check yes or no to, but these frameworks require more institutional thought and self-examination. It helped when a coworker attended an ALA sponsored series of workshops on assessment based on what we hoped would be the start of a new way of teaching. While it did not work as we hoped, the result was the librarian who attended came back with a new sense of purpose that the instruction is critical; it’s not an accessory but it is the coat we wear. Step 2- we have begun to list for our own benefit what the objectives we intend to cover. I strongly agree we knew although we don’t always use the ‘right’ words what we want students to know or be able to do from instruction. Step 3 is one we have not really broached. We intellectually know that the faculty, and their accreditation standards must view these frameworks and their professional standards as important otherwise we are truly ONLY talking to ourselves. Part of me thinks we sometimes still are scared that faculty will completely shut us out if we begin or act like our part is important enough to make some noise. Finally, we have some strong ideas of how to have activities that suggest the outcome is being achieved but again that will require we make some noise, and perhaps suggest that the every library concept imaginable 1 shot model will not be taught. We frequently are not sure what we will do if faculty does not ‘like’ that so we continue with what in retrospect might be teeny weeny steps.
    Which steps from Oakleaf’s “Roadmap” will take more work to complete? Identify one next step that would help you make progress toward incorporating the Framework into your assessment work.
    Steps 5 – draw the map; 6 – get real; 7- hunt and gather; 8- know it when you see it and steps 9 and 10. If I were to push something forward it would be the draw the map one. Maybe we look at the top 3 disciplines that have graduates and find out the threshold concepts for 1 are,etc… then there might be a real sustained basis of instruction and connection to what these students are moving towards.

    Like

  11. I think we’re on Step 2 at my institution, and I have a feeling that steps 3 and 5 are going to be most difficult. We do have as one of our library-wide goals for the upcoming fiscal year to create an assessment plan for the library. It’s one of the parts of our strategic plan that we’re trying to complete this year: “Develop an Assessment Plan to ensure that evaluation of library resources and services is intentional and programmatic.” I think the stepwise approach of this article will be very helpful in that, as we are in the process of getting our information literacy outcomes (closely tied to the framework) adopted by the college and we use those outcomes when planning all library instruction.

    Like

  12. The best way to describe what is occurring at my institution is a simultaneous baby steps into Step 1 and dipping our toes into Step 2. As a library staff we are very interested in the Framework, but as I have mentioned in prior posts the curriculum here leaves very little wiggle room for either the students or the professors. We have friends in the English Writing program with whom we need to cultivate our already established relationships, and are working on getting our foot in the door with both our equivalent of FYE and School of Continuing Education. In our individual instruction sessions we attempt to do Step 4 and as a department we are exploring how to best evaluate the results.

    I am hoping that we can take different steps in not necessarily the exact order as Oakleaf laid out in her article. Much of what we can do is to seize opportunities when they arise. What I need to do is prepare to be ready for any of those moments, as they can occur anywhere on campus at anytime.

    As Cristy Moran posted earlier, just getting on this road is part of the journey. I like how she describes that getting a campus all onto “one page” can take longer than expected. Especially with all the different disciplines and departments working in different ways.

    Like

  13. It is difficult to say, institution wide, what step we are on. In some places, we are still in step 1. But there are areas where we are further along. Our first year experience seminar now has a library unit that was constructed by a group of librarians. The objectives and the assessment pieces are built in. It was a collaborative effort with FYE instructors. They even have options for deploying the instruction based on the needs of the class. So, in that area, it looks like we are in step 6. I think the biggest challenge is that most of us are still doing one-shot instruction. We will have to be very creative in finding ways to collaborate with instructors to make library instruction more integrated, and in finding ways to follow through with authentic assessment pieces, even in a one-shot session.

    Like

  14. I believe we are at Step 1 and also pre-Step 1. One colleague, who is also on this forum, is most likely at Step 2 or beyond at this point, as she is working to adapt our information literacy sessions to the Framework. However, I believe others have not yet addressed the change from Standards to Framework. I am relatively new to my institution and am not sure if the step that will take the most effort to complete will be getting everyone to Step 1 or if it will be Step 3 Agree or Agree to Disagree. For me, developing learning outcomes based on thresholds for my next information literacy session is the next step in incorporating the Framework in my assessment work.

    Like

  15. I feel it is important to incorporate and educate faculty on the Framework. So far, we haven’t really even broached this topic of the Framework or assessing it in our library or how to use it. We are free to use/adopt as needed, but as adjuncts, we aren’t free to work with departments as a whole on how to implement this in a meaningful way for the different disciplines. That said, I do have a goal to start working on Step 2 when I create my learning objectives, or at least working to tie them to the Framework going forward. I would then like to think about how to assess in the right way, although I am not sure that one person assessing is as valuable as having a whole team agree on some basic pre- or post- assessments. As we may be getting a new Assoc. Dean at some point, that may be a place to approach these ideas.

    Like

  16. We are on Step one with a few of our freshman English Comp classes. We adapted a couple of the Frameworks to a 60 minute class although we have no plans to try to incorporate all the framework at the freshman level. With our upper level classes and graduate classes a couple of the other Frameworks are being discussed. Most of the other steps are waiting to see how we survive this first one. Feedback from our administration though wants some sort of quantifiable assessment so I guess that will be our next step.

    Like

  17. I wouldn’t necessarily say that the institution has completed any of the steps on Oakleaf’s Roadmap, since information literacy currently belongs to the English department. However, since library faculty are working on a document to assess the Library, and Information Literacy instruction, we appear to be on Step 2 of the Roadmap.

    The step which will take more work to complete is Step 3 – Agree or Agree to DIsagree. Since academic freedom is constantly mentioned, many want to continue teaching as they have been, and change is sometimes met with resistance. As Rachel mentioned in her post, “it takes a long time to make a change.”

    Like

  18. I really like Oakleaf’s roadmap and I think individual instruction librarians can use this for their own departmental teaching goals. I’m very new at my library, but I think we’re probably between 1 and 2 when it comes to our Freshmen English courses (the only instruction program multiple librarians work with).

    For me, I believe Step 4 will take the most time since each course you teach is different. I could see coming up with a variety of activities related to the learning outcomes for my departments so I have a nice variety to choose from to make my semesters easier.

    Like

  19. I think the hardest part for me will be writing my leaning objectives, though I see the necessity of them with the Frames. We’ve sort of been on automatic pilot with the one-shot, assignment-driven library sessions. Once we better know what we’re looking for in student performance, we can then better know how to access it.

    Like

    • Great point, Andrea. Writing learning outcomes is legitimately challenging! Thing 11 (https://23frameworkthings.wordpress.com/2017/03/21/thing-11/) provides some resources to help with the process of writing them. Personally, I find it really helpful to think of our learning outcomes as a living document that we can always update and change. That way, we don’t get as hung up on getting every word just right, or endlessly editing them.

      I think having learning outcomes that address the broader scope of information literacy (and the big concepts represented by the frames) can help us move beyond those one-off, assignment-focused sessions you mentioned. In many cases, faculty members just aren’t aware of what’s possible! Sharing learning outcomes that reflect the Framework might help you make them aware that there’s more to info lit than learning the databases most useful for a specific project.

      Like

  20. As an institution, I think we’re working toward a cohesive statement about IL that will take the Framework into account but, as yet, we’re at Oakleaf’s first step. I think different librarians are at different steps in their own practices. Many of us have discussed the move from the Standards and the opportunity the Framework presents us to “redefine” IL as an institution; but we’re working on lots of elements (including assessment, a mission statement, an organizational model change, etc.) that I see being informed by the introduction of the Framework.

    I think *getting* on Oakleaf’s road is probably the most difficult, most time-consuming aspect of the whole thing – rather than any isolated step. It is difficult as a large, multi-campus institution to get on any one page, especially when so many other activities and initiatives take priority. But I do think that setting up a committee or taskforce for IL (which we’ve done) is the first step to reevaluating the IL program, thus “getting on Oakleaf’s road.”

    Still…Framework or not, assessment and the collection of learning artifacts from which we can assess in an instructive, ethical, and significant manner will always be difficult – so Step 7 “Hunt and Gather” – is likely to continue to be a challenge. I look forward to continuing to work with that IL committee I mentioned to come up with ways to plan for those roadblocks so that we don’t end up just stalling out. I think working together and brainstorming the best methods to collect, the best contacts in departments/ faculty that can urge other faculty to participate, etc. will be integral in a successful Framework-informed IL program.

    Like

    • Cristy, I totally agree that in many cases, just getting on the road is the most challenging part! Your multi-institution situation sounds challenging and complex. It’s great that you’ve created an IL committee to tackle these roadblocks together. Doing this work right takes time, and thoughtful consideration of your institutional context and the challenges you’re likely to encounter in advance should help the whole process go more smoothly!

      Like

  21. I think we are still at Oakleaf’s Step 1 at our university library. We have done some work with SCONUL to map out our own library’s credit bearing courses and standards for library instruction and outcomes, but I think this is as far as we have gotten. It takes a long time to make change.

    Like

Comments

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s