Do the failings of the oral defense vacate the doctorate degree?

The oral defense at the end of doctoral education lacks legitimacy.  A reputable tests would be both reliable and valid.  assessmenthitthemark

Getting three examiners in a room, to put  a candidate through some kind of ringer does not a valid and reliable assessment make.  Who or what are they examining?  The candidate or candidate’s work or some combination thereof?  On what basis would each determine a fail?  Is it the same benchmark for all three?  Does the university give them guidelines and check for their common understanding?  Does the university check that all defenses look for standards set by the program?

What if examiners get it wrong? The examiners for Jason Richwine’s Harvard PhD got it wrong.  What process is in place to check the oral defense committee’s work?  No checks on Richwine’s committee and supervisor occurred even after a journalist did his own check and broke the story .   Journalists often unmask wrong doing in a doctorate, but why don’t universities put checks in place?  Laxity in assessment in doctoral programs puts them in the way of ridicule and undermines their legitimacy.

In Latin, 'nullius in verba.'

Did Harvard or other universities ask the questions Richwine’s committee;s epic fail asks?  Richwine got a standard doctoral education.  He applied to, was accepted and enrolled in a doctoral program at the Kennedy School of Government, wrote a dissertation, sat a defense and presto, Harvard awarded him its esteemed PhD.  Richwine’s story shows even the best universities fail to fix  problems in doctoral programs.

Harvard experienced protests and national notoriety but merely placated protesters and let the unrest die out.   If Richwine didn’t believe in the inferiority of Hispanic immigrants, he could sue Harvard to place blame on his program, his committee, and his university.  As a student at Harvard he had every right to expect the best doctoral education and highest standards of assessment.

What argument would his lawyers make?  Harvard gave Richwine a PhD in 2009 for research based on an assumption that ‘illegal Hispanic immigrants to America had lower IQs than non-Hispanic whites’.  After a reporter blogged about Richwine’s dissertation in 2013, many academics read it.  The complainant then learned that the research questions, methodology and literature review suffered grievous and invalidating errors.  The complainant asks Harvard to refund all expenses incurred in his five years of study and award recompense for a failure to assess his doctoral work to withstand the scrutiny of  academics outside of his committee and supervisor.   He asks Harvard to make a public apology placing blame on the PhD program he took to properly assess his doctoral work.

Without a rigorous process to assess whatever it is oral defense committees assess, shouldn’t all doctoral students demand transparency about checks of their work (beside typos), checks on the work of the examining committees, and for the committee’s scores on validity markers in the oral defense?

The next time a journalist unmasks a questionable dissertation to shame an esteemed official with a PhD, it’d be wonderful to also blame the granting institution.  Without checks for cheating and soundness within doctoral programs, deficient dissertations will continue to come to light.  Research on the quality of dissertations is needed.

Right now programs place too much onus on the integrity of the candidate.  At the very least shouldn’t all dissertations go through a thorough plagiarism check, even a digital check like Turnitin, before any oral defense takes place?   Plagiarism shows up  more frequently than a robust, thorough, quality control process should let through.  Plagiarism aside, the research questions, design, methods, and conclusions may fail the thin-red-line-thread-of-coherence test taught in the Cohen, Manion and Morris text for social science research.

The rarely failed oral defense gives false confidence.  How could Richwine be trusted with setting up and conducting research save for an right-wing foundation?  Why did he need a Harvard PhD to work for a right-wing foundation, save to confer a sheen of legitimacy to him?

The sample of one research project also lends doubt about the soundness of a doctoral program structure.   Research done after the degree shows whether the committee recognized the, as yet, undefined confidence level conferred in a doctoral degree.  Yes, of course, the record shows holders of the degree and those early career researchers going through its dubious assessment process produce influential works even with the problems of doctoral programs.

Greater confidence in the doctoral program could come from multiple micro-assessments by multiple examiners and multiple research investigations.  Assessment data tied into block chains linked to discrete aspects of  ‘doctoralness‘  overcome validity and reliability assessment problems within doctoral programs.  Let’s say students submitted their questions, methodology and literature review to random, bona fide external reviewers.  Remember a  journalist and the web provided reviewers for every aspect of Richwine’s notorious screed.   Associating reviews to a block chain would give more confidence in the doctoral program.

The bogus doctoral degrees now destroying Russian doctoral education come in here too.   Russians seeking a PhD or two to pump up their prestige took advantage of the laxity in standards in doctoral education. They turned Russian universities into doctorate diploma mills which now sadly calls all Russian doctorates into disrepute.

Outside of Russia, doctoral education suffers from the same lack of rigor in assessment that allowed for mass corruption to take hold in Russia.   Unless doctoral programs declare exactly what that achievement mark is and devise a robust means to test for it, the same fate could befall doctorates outside of Russia.  The hocus pocus and hokey pokey that passes for assessment in a doctoral program disrespects and makes vulnerable the work of all doctoral students.

As a solution, an entrepreneur or international not-for-profit could set up a bitcoin like block chain system to outsource assessment for doctoral programs and students.  This could be called the ‘journalist test’.

A block chain would secure pass/fail assessments of characteristic aspects of doctoral study.

Originality test: How does the research make an original contribution to the literature?

Research instruments: Does the research show fidelity to principles inherent in the methodologies?

Questions:  How does the research plan execute on the questions asked?  Is the quality of the questions aligned to the literature and problems in research?

Literature Review: To what degree is the research based on a comprehensive interrogation of the  literature?  In the case of doctorates in the sciences, a literature review skill may require programming a lit review search and critiquing the results.

Plagiarism checks: The work should undergo a check for plagiarism.

Multiple draft changes:  This sort of assessment checks that the writing came from the candidate.  Taped evidence discussing changes to the manuscript through multiple iterations lends authenticity as to the author.

Assessment would secure aspects of doctoral know-how against habits of mind derived of a doctoral education.  The above list is not, of course, exhaustive.   If doctoral programs decided what the oral exam tested or recognized, the check list would follow from it.

Evaluations of discrete aspects of doctoral education against specific criteria and exemplars could be sliced into a bitcoin kind of security.  Unless granting institutions and disciplinary bodies make-up a robust assessment system, capped off by an oral defense if necessary, to assess the lessons of a doctoral program, glaring omissions may still occur.

So back to the fitness of the oral defense.  Presently, the oral defense makes candidates defend something they may have thought better of, from which they  learned what not to do the next time round.  Post PhD, a newly minted PhD will likely not repeat the solitary toil in graduate school.  The vast majority of academics work with others.  Besides which how much experience do students get in a doctoral program with defending anything before an audience save in the oral?  Re-framing the defense as a candidate’s public critique of a first contribution to scholarship before curious and supportive fellow scholars makes more sense.  Re-framing the defense as a lively discussion between collaborators on research watched by an examining committee more closely mirrors research practice.  Ironically, concern over assessing individual understanding has kept doctoral programs from offering collaborative research pathways to students.

The problem of a sample of one undermines doctoral assessment.  Demanding less than book-size output from a novice researcher in favor of more samplings of research acumen, would change the scholarly output but not the purpose of doctoral study.  Doctoral programs structured around three discrete research collaborations would get a richer sampling of the skills of scholarship.  It’s easy enough to improve on the standard doctoral program.

For all that toil, all that time taken, all the treachery students take on, don’t doctorate students deserve better assessments and more thoughtful doctoral programming?





Happy 350th Scholarly Journal: What Can Doctoral Programs Learn From Your History?

On the 350th anniversary of the scholarly journal, changes to scholarly communication call for incorporation of new vehicles and methods of communication in doctoral education.   On March 6, 1655, Henry Oldenburg published The Philosophical Transactions of the Royal Society of London, Giving some Account of the present Undertakings, Studies, and Labour of the Ingenious in Many Considerable Parts of the World; the first scholarly journal. In 1660,The Royal Society of London for Improving Natural Knowledge, had brought together natural philosophers by royal charter. 

Oldenburg’s publication evolved to its present day counterpart, which now gives account of the studies of the Ingenious and leaves out their present undertakings and labour.  Improving natural knowledge meant study of an indivisible realm which encompassed botany, mathematics, zoology, physiology, chemistry, physics, geology, geography and so on.  Imagine improvement of knowledge without scientific disciplinary divisions and journals.  That was scholarship in 1660.

Oldenburg assembled the literature of the royal society into a publication both to speed up sharing of knowledge, which had until now occurred through letter writing and meetings, and to make some money

Robert Boyle, the scientist whom school children now learn about, was a member of the society and he wanted Oldenburg to archive, develop provenance, registration and dating of works published in the journal.   Accordingly Oldenburg marked the articles published in Philosophical Transactions, however he didn’t make much money.   Even so, acceptance for publication conferred recognition and became sought after with the added features of archiving and tracking of provenance.

The motto of the society “Nullius in Verba” meant don’t trust claims (but check them) and entrenched the scholarly habit of skepticism and independent thought, associated with the charter for the first university in Bologna in 1088.  In 1752, when the publishing of Philosophical Transactions was getting off point, the society righted its self and declared that the journal would from now on be published ‘for the sole use and benefit of the (royal) society.’  The society therefore asserted integrity for the scholarly brand that continued to confer recognition to natural philosophers of the works it published.  The society established a pedigree of the highest order for its membership, a pedigree associated with a lengthy list of scientific achievements.

At times during its history, Philosophical Transactions was a drain on the finances of the society with the cost to put scientific drawings to print.  Given that the technology of printing then facilitated type and drawings or diagrams required much more labour and therefore cost, the publication struggled.  Taylor & Francis was a specialist printer for the publication at one point, and evolved along with other technological and academic developments in the knowledge communication infrastructure to occupy its present perch as a major player in the now lucrative business of scholarly publications.

As scholarly transactions increased, the journal evolved the standards and specifications for communication of scientific knowledge.  By the late 19th century, it introduced standardization of format, better peer review of articles, and it split into two journals, one for biological sciences and one for physical sciences.

In Latin, 'nullius in verba.'
In Latin, ‘nullius in verba.’

In the history of scholarship, certain threads continue over the centuries, while existing technology supports the root purpose of scholarship, ‘nullius in verba.’  Today, the multiple author academic blog (MAABs) and new genres of scholarly knowledge production give new forms, venues and meaning to publication.  MAABs disrupt and undercut publication pathways to specialty journals which now takes too long, is too slow, and too verbose when a good graphic or video will do.

Scholarly communication continues to evolve and influence the nature of scholarship.  With the inter, cross, and trans discipline pollination just beginning in MAABS, will ‘disciplines’ continue to become broader and more hybrid?  Will students and scholars join together to study phenomena instead of disciplines?  How will the scholars of the future communicate?  What kind of communication skills should doctoral programs impart?  How much should doctoral programs encourage forms of scholarly communication outside of the journal article and print?

Doctoral programs should welcome and invite disciplinary promiscuity and a broadening of experience with new forms of  scholarly communications by:

  •  A requirement that doctoral students write in a multidisciplinary academic blog.
  • Grouping all students who will be using similar research methods together for research methods courses regardless of discipline.
  • Encouraging collaborative and networked research around a common phenomena like water use, aggression, change or garbage.
  • Experimenting with signature pedagogies from other disciplines.

Scholars working today incorporate new media for scholarly communication that tear at disciplinary barriers.  Imagine improvement of knowledge without disciplinary distinctions or journals.  That could be scholarship in the future.  Doctoral education needs to equip its grads to work in the scholarship of the future or loose credibility.  On this 350th anniversary, scholarship has come full circle back to another starting point.  Doctoral education needs to place its graduates at that starting point and needs to question everything.

See the history of copyright battles are writ small in the following chart of access to the publications of Philosophical Transactions.

a) 1665-1943* Free in perpetuity
b) 1944*-2003* under access control
c) 2004*-2 years ago Free
d) Last 2 years under access control
Each January the years with an * will move forward by one

%d bloggers like this: