TEAM Evaluation – 15% Achievement Measure Options

The time has once again rolled around to complete the Achievement Measure Worksheet which an educator uses to specify the measure and corresponding scale that will comprise 15% of the overall evaluation score under TEAM.  Having said that, now let me apologize for the length of this post :).

One significant change for 2014-15 is who has the final say in measure selection.  As is customary, LEAs may produce a list of recommended Achievement Measure Options that are inline with District and School goals while also being sound.  In previous years, this document was used to guide evaluators as they were the ones who had the final say in what was an appropriate selection for an educator.  During the most recent legislative session however, Public Chapter 885 was passed which now states that the person being evaluated has the final say in selecting an appropriate achievement measure.  The significance here is that there are a number of additional options to choose from because you are no longer limited to measures and scales on any ‘recommended’ list.

This post is my attempt at better explaining the Achievement Measure options and corresponding scales so that educators can be as informed as possible.  Please note that I am not and will not make any specific recommendations and am likewise unable to provide personalized advice.  It is the educator’s responsibility to acquire the necessary information to make a selection in their best interest that is also as closely aligned to their job description as possible.  Please also note that I firmly stand behind the recommendations made by MNPS though I also know that not everyone has the capacity or experience to explain the scales appropriately.  As you have probably already ascertained, this post is specific to educators within MNPS though many points can be generalized and applied to other school systems.

Using the Achievement Measure Worksheet and accompanying points, I will highlight each option below while also providing my own narrative of understanding.

  • SCHOOL-WIDE TVAAS
    • I am listing this measure first because I like what it indicates – there, I said it.  This does not mean that I think it is an appropriate or fair selection for educators.
    • Rather than comparing cohorts of students (e.g. TCAP), TVAAS looks at individual students in a cohort and measures their academic growth across time.  In other words, you are comparing Johnny in 2014 to Johnny in 2015 rather than comparing Johnny to Susie.
    • There are a couple of things to consider before choosing TVAAS:
      • If you work in a PreK-4 elementary school, this data is entirely determined by the 4th grade students (by comparing their 3rd grade TCAP NCE to their 4th grade NCE) and the instruction provided by the 4th grade teachers.  So ask yourself, do you have the necessary confidence that the 4th grade team of teachers can provide effective instruction in a manner that will bring about academic gains in excess of the growth standard?
      • To be included in the calculation, students must have prior data from which growth can be determined.  In other words, they would have had to have taken the TCAP assessment (ACH or ELSA) last year.  If they moved in from out of state they have no prior data.  Also, if they took the MAAS version of TCAP last year, they are considered to have no prior data.
    • There are four (4) primary TVAAS calculations that you may select from:
      • Overall – for 2014-15 this includes RLA, Math, & Science
      • Numeracy – Math only
      • Literacy – RLA only
      • Literacy and Numeracy – a combination of Math and RLA TVAAS
        • Before you select one of the four options, become informed.  Access your school’s TVAAS Evaluation Composite Report.
        • This report will provide you with trend information that can hopefully inform your decision making.
    • NOTE:  If you are considered a non-tested teacher (PE, PreK, K-3, Counselor, Instructional Coach, etc.), your TEAM Evaluation growth measure (25%) will likely already be tied to School-Wide TVAAS.  Ask yourself, do you want to put all your eggs (total of 40%) in one basket (measure)?
  • STATE ASSESSMENT (i.e. TCAP/EOC)
    • There are a few usable scales that can be applied when looking at TCAP achievement.  Additionally, one can specify the content area (Math, RLA, or Science), grade level (or school-wide), or even class level (for teachers of grades 3+ and who teach one of the tested content areas).
      • NOTE:  Social Studies is not an option as that test will be piloted this year and there is no comparable data and norms have not yet been identified.
    • Comparison of the % of students scoring proficient or advanced
      • This scale is identified as the TCAP and EOC achievement scale and requires a minimum increase of seven percentage points over the previous year in order to get a score of 5 (e.g. 2014 %P/A was 53.5% so the 2015 %P/A would need to be at least 60.5%)
      • This measure and scale selection compares cohorts of students.  In other words, if one were to select 3rd grade math, last year’s 3rd grade math proficiency level would be compared against this year’s 3rd grade math proficiency.
      • It is important to note that this year there will be no MAAS assessment so any students with an IEP will be required to take the regular achievement test (ACH or ELSA).
      • This scale may not be ideal for high-achieving schools where an increase of 7.0% or more may be difficult or even impossible for schools with greater than 93% of students scoring proficient or advanced.  Alternatively, this scale is certainly practical for schools with lower achievement levels.
      • An increase of 7.0% or more is not outside the realm of possibilities.
      • In order to choose class level data, one would have needed to have taught a comparable class (grade and content) the previous year.
    • TCAP or EOC Mean Achievement
      • This scale is applied by converting the 4 performance categories to a numeric scale and then averaging the scores.  Following is the formula considering a sample of test takers:
        • [(#Advanced x 4)+(#Proficient x 3)+(#Basic x 2)+(#BelowBasic x 1)] ÷ TotalTestTakers
      • Mean Achievement scales do NOT compare cohorts of students so the outcome is based solely on the current year’s test takers.
        • This then could be ideal for teachers new to a grade, school, or content area.
      • This scale is ideal for high achieving schools who may not be able to show an increase of 7.0% as required by the previous scale.  Conversely, this scale would not be ideal for schools with lower achievement rates.
      • The same variations apply in that one can select grade level, school-wide, or class level data that is content specific (Math, RLA, or Science).
    • TCAP or EOC Mean Achievement Level Increase (MALI)
      • I would be remiss if I did not mention this scale.  You will not find it on any published documents because it is school specific and would require individual calculations.  In other words, it is highly personalized and not at all generic.  The process of developing the scale though is standardized.
      • While this approach also does not compare cohorts, the previous year’s Mean Achievement is used to set the baseline at a 3 for the current year.
        • EXAMPLE:  Let’s assume a 2014 Mean Achievement of 2.25.
          • 5 = Between 2.47 and 4.00 (2.47 represents moving 22% of test takers up at least  one proficiency level)
          • 4 = Between 2.36 and 2.46 (2.36 represents moving 11% of test takers up at least one proficiency level)
          • 3 = Between 2.15 and 2.35 (Note that 2.25 is the midline)
          • 2 = Between 2.04 and 2.14 (2.14 represents moving 11% of test takers down at least one proficiency level)
          • 1 = Between 1.00 and 2.03 (2.03 represents moving 22% of test takers down at least one proficiency level)
      • This is a practical option for lower achieving schools.  While a mean achievement of 2.25 (from the example) is still equivalent to a 3, using MALI makes it much easier to get a 5 (consider the range to get a 5 on the MALI example here compared to the Mean Achievement scale range of 3.4 to 4).
      • While this scale does allow support for increasing achievement over the prior year, it is not focused strictly on the students who score proficient or advanced.  Consider 100 test takers where each proficiency level included 25 test takers.  It is possible to get a 4 by decreasing the number of Below Basic test takers by 11 (14 remain) and increasing the number of Basic test takers by an equal amount (now 36) with no change in the number of proficient or advanced test takers.
        • This is still movement that deserves recognition.
      • Unfortunately, I cannot support individual schools with developing a MALI scale.
  • “OFF THE SHELF” ASSESSMENTS
    • Discovery Education / ThinkLink
      • Because Discovery ED assigns proficiency levels like TCAP, those same scales could be used.
      • This option is only available to schools who have chosen to purchase the Discovery ED Benchmark assessments.
      • As an administrator, if I were to receive a completed 15% Achievement Measure Worksheet identifying Discovery ED as the selected measure, I would file a Disagreement with the State of TN without hesitation for the following reasons:
        • It is not a secure assessment.  The Benchmark Assessments are published and accessible two weeks prior to the administration window.
        • My experience as a Data Coach has shown that DEA was NOT a valid predictor of TCAP proficiency during the 2013-14 school year (for many schools).
        • DEA tests are not inherently comparable given that different SPIs are assessed on each benchmark.
    • Fountas and Pinnell 
      • Fountas and Pinnell are the current Text Level Assessments for MNPS Elementary Schools
      • As an administrator, if I were to receive a completed 15% Achievement Measure Worksheet identifying Fountas & Pinnell as the selected measure, I would file a Disagreement with the State of TN without hesitation for the following reasons:
        • The TLA is meant to be an instructional tool and not an evaluation measure.  Once the data is tainted by tying it to a teacher’s evaluation, the viability of the data as an instructional tool is lost.
        • The TLA are administered by the teacher being evaluated and so then are not a secure assessment.  The only option would be to have an alternate party administer the TLAs for a class which further diminishes the viability of the measure as an instructional tool given that a wealth of information is provided to the educator through the process of hearing the learner read.
        • The TLAs are subjective in nature as so much is left to the interpretation of the educator administering the TLA.  The only option then would be to have a single person or team administer all TLAs in an effort to create the most standardized administration.
    • AIMSweb
      • I can appreciate AIMSweb as an option, however, there are several points to consider:
        • There is no ‘composite’ score assigned so a specific subtest would have to be identified.
        • There are no proficiency levels but only national norms assigned for each student.
          • In prior years, the State of TN published some sample scales that could be used as guidance for completing the Achievement Measure Worksheet.  The scale published for measures where percentiles were assigned is as follows:
            • 5 = 80.00 – 99.99
            • 4 = 60.00 – 79.99
            • 3 = 40.00 – 59.99
            • 2 = 20.00 – 39.99
            • 1 = 0 – 19.99
          • To apply this scale, one is supposed to average the percentiles of all test-takers.
            • This scale is flawed statistically in that percentiles are not to be averaged given the normal distribution.  Percentiles follow a normal curve such that the majority of the population (roughly 68%) falls in the middle.  Considering this, it would be extremely difficult to get a 1 or a 5.
            • I will say it again, the percentiles we receive from AIMSweb are national percentiles.
        • This is the first time MNPS has utilized AIMSweb so the data will be new for our population of students.  In other words, who knows how it will play out?
        • I would support a growth model with AIMSweb such that beginning of year data is compared to end of year data but …
          • Only subtests administered at both points in time could be used
          • Considering percentiles, what constitutes average or expected growth?  One could assume that if a student holds his own in comparison to the norm then that would equate to a 3 but what equates to a 1, 2, 4, or 5?

So there you have it.  Hopefully this information is informative and supports educators in making a sound decision but I fear that I have only created more questions than answers.  Again, this was not meant to guide the decision making of anyone but rather to support informed decision making.

For me, it is important that Educators are empowered.  This first happens through information sought and knowledge acquired.  My hope is that at least one person will be better positioned to ask the necessary questions about this portion of the TEAM evaluation.

I apologize again for the length, and for putting the post off until now.

Leave a Reply

Your email address will not be published.