NRC Assessment of Research Doctoral Programs

May 3, 2011

  • There were 2 separate NRC posts -- 4/28 corrected art history (call this v3.1); 4/29 maintained art history as in v3.1 and set all Harvard pct tenured to N/D, at Harvard's request.  Call this v3.2 
  • The Mac version of the big Excel (v2 v3.1 v3.2, some or all) has the wrong column headings starting at column AO through column BA.  NRC has been informed. 

May 2, 2011:  New postings and insights: 

  • The website http://www.colorado.edu/pba/nrc/ now links (near the top) to Data Shell of NRC Column/Variable Names: Used in reading NRC's download of rankings and other stats, ResDocTableWin.xls, into SAS. Updated 5/2/2011.  This is an Excel similar to that released behind a password to AAUDE schools in September 2010.  It works for v2 (4-21-11) and v3 (4-29-11).   There won't be any SAS code to read the Excel -- the SAS variable names are in the shell and that plus proc import are all you need. 
  • Differences between v2 and v3 - are ONLY in programs in the art history field, ONLY in the completion rate measure.  The rates in v2 were incorrect.  Those in v3 for art history programs are the same as the ones in v1, except for the number of decimal places.  For example, Boston U's art history grad rate in v1 is (picked up by SAS as) 0.166667 but from v3 comes in as 0.16667.
  • There are other variables and programs with changes between v1 and v2 (not between 2 and 3) involving 1 fewer decimal place in v2 than in v1.  This seems also to have changed rankings somewhat -- e.g., changes in diversity rankings in fields where none of the programs had any data change.  Remember, a tiny change in an input value could be enough to make a change in a ranking.  
  • NRC stated all along that it would change (between v1 and v2) only errors that it made, not errors made by institutions on submission.  We do not know the genesis of some of the individual changes made between V1 and v2 because NRC has not released explanations.  See LMcC to NRC, catalog of v1 to v2 changes and request, 4/23, posted 5/1, for a listing of these changes.
  • Jonathan Cole critique of NRC process at http://chronicle.com/article/Too-Big-to-Fail/127212/?sid=at&utm_source=at&utm_medium=en

May 1, 2011: NRC released a revised spreadsheet (now known as v3) on April 29 (on the publications page), saying that it revised art history completion rates and time to degree between v2 and v3, to fix errors in v2.    What else we know: 

  • The art history completion rates were wrong in V2 and have been changed in v3.  We have not yet compared them v3 to V1.
  • The art history time to degree did not change from v1 and v2 or we suspect from v2 to v3 and are as OK as ever in v3
  • We asked for explanation and verification of a number of other changes v1 to v2 (v2 was released 4/21/11).  Kuh said such an explanation would be posted 4/28 or 4/29.  As of 5/1 noon, it has not.    LMcC to NRC, catalog of v1 to v2 changes and request, 4/23, posted 5/1.
  • Colorado will post SAS code for reading v2 and v3 (which have columns in them that are not in v1) to the Colorado website, and will put notice here. 
  • We'll also post our comparison of v2 to v1, and requests to NRC.  
  • And comparison of v2 to v3 
  • Odds that v3 is the last:  72%.   Odds that it answers all the issues with, say, academic placement - small N's - missing - changes in denominators -- zero.  Odds that a v4 would do so: small.  
  • We (AAUDE NRC-er's Lou McClelland and Julie Carpenter-Hubin) will attempt to use this site to update.  We may send messages to AAUDE or AAUDE_NRC listservs announcing new postings here.   

 

  • April 23, 2011:  NRC issued a 4-21-2011 release of all data and rankings with a statement of 4 systematic changes to measures plus an Excel on changes in rankings only.  The "big Excel" of all data is available from the NRC publications page.   NRC has acknowledged errors in the 4-21-11 release, at least in art history programs, and is planning still another release.  AAUDE is working with NRC to get documentation of all changes before that next release.    --LMcC. 
  •  
  • Official NRC website
  • This letter was sent from AAUDE to Charlotte Kuh on 1/3/2011.  She sent an unofficial response a few weeks later to the signatories saying our suggestions were "mostly excellent".
  • Summary of issues with measures released by NRC (Lou McClelland, March 2011)
  • Information on reporting errors to the NRC is now in the Frequently Asked Questions section of the NRC website.
  • AAU representation in NRC Research-Doctorate Programs and Reported Student Enrollment

NRC Convocation

The following are the papers presented by AAUDE members at the NRC convocation held on March 4, 2011.

Working with the data

Example Data Displays

All example displays use fake data or do not have any data available.

Analyses

Interactive Websites

Articles

News (reverse sorted by date added)

Methods

History and misc.

Institutional NRC pages

 

* Steps to compare histograms of rankings available from PhDs.org

  1. Click on Find Graduate programs
  2. Click on the Broad Field, then select your program, say Economics
  3. Scroll down to ‘Choose your own Priorities’ and click on 5 for ‘NRC quality measure’
  4. Scroll down farther and click on ‘Rank programs’
  5. All the programs will be listed in order of R Quality.
  6. Check up to four boxes for particular institutions ( it says 5 but I couldn’t get it to display them all)
  7. Click ‘Compare’
  8. Wait—a new page will display the histograms for both R and S rankings for selected institutions.