Assessing the Credibility of Published Material

Bob Brown
School of Computing and Software Engineering
Southern Polytechnic State University

The World Wide Web has put a wealth of scientific material at the finger tips of students and researchers. Unfortunately, not all the material that is available is of high quality. Some of it is not credible at all. It is always important to assess the credibility of material you use or refer to. It is especially important to do so for material published only on the World Wide Web.

When you write academic papers, your references must be not just credible but authoritative. It is up to you to assess the material you will use to support your arguments and to choose accordingly.

Here are some things to consider when assessing the credibility and authority of what you read.

Assess the Content

At the bottom, what is important about published material is the content. Consider whether the article presents evidence to support the information or conclusions it gives. There are two general types of evidence that can be presented: first-hand research and reference to the works of others.

If the article presents research by the authors, consider consider the design of their research. Does it cover the subject thoroughly, or is it superficial? Is the research reported in sufficient detail to be understandable and persuasive? If there is an experiment, is there enough information for others to attempt to repeat it? Have others repeated it with the same result? Does the evidence presented hold up to your own close examination?

If the authors have cited the works of others, who are they? Are they authoritative in the field? Are there citations to several other scientists, or only to the work of two or three? Are the works cited in this paper also cited elsewhere? (In the field of computer science, you can find out something about how often a work is cited using CiteSeer, which is available on the World Wide Web at http://citeseer.ist.psu.edu/cs An even better source is Science Citation Index, but this is not widely available.)

Consider how current the work you are reading is. Also consider whether currency is important. Foundational work in an area may be years old and still be quite important. However, for areas of current research, recent work is likely to be more valuable than older work.

Assess the Author's Credentials

What credentials for the author are presented with the article? Is the author a researcher in the field of study? Is he or she well-known and well-respected?

What other publications have cited this author's work? (Go back to CiteSeer or Science Citation Index for this information.)

Sometimes an author may be a practitioner in a field rather than an academic researcher. That does not necessarily diminish the value of the work, but it should make you take a harder look at the other areas to evaluate.

If the author is a journalist, the article is likely to carry very little weight as an academic reference.

Assess the Publication

Where material is published says quite a lot about the credibility of the material. The most credibility comes from being published in a refereed scientific publication. A publication is "refereed" if, before a paper is published, the author's peers evaluate it and make a recommendation to publish it. You find out whether a publication is refereed by looking at the Web site of the publication to find out if they have, or solicit, "reviewers" or "referees." Sometimes a publication will have an editorial board of qualified scientists. This is almost as good as having submissions refereed provided that the editorial board really looks at the material to be published. That can be hard to find out.

Publication in a refereed journal does not automatically mean that a particular article has been refereed. Some refereed journals, among them Science and IEEE Computer also include news articles by journalists. You will have to sort these out. Check the author and the references as described below.

Here is some general advice about categories of pubications. There are exceptions, and you will need to apply judgement.

A "scientific" publication is one that is published to advance a particular branch of science. It may have advertising, but its main purpose is disseminating research. The publisher is generally a professional society or a university. In rare cases, corporations publish scientific journals. The IBM Systems Journal is one such journal. However, be aware that not everything published by a professional society is a scientific publication. You must apply some judgement.

A "trade press" publication exists to sell advertising and make money for the publisher. The publication focuses on a particular domain of knowledge, or "trade." The publisher will be a corporation. Material published in the trade press has much less credibility that material published in a scientific publication, especially if it is written by a journalist.

A "popular press" publication exists to make money from advertising and appeals to a wide audience, even if it is specific to a particular domain. PC Magazine is a popular press publication. Most material from the popular press has very little credibility for research purposes. Again, however, there are exceptions. Scientific American is a popular press publication with invited articles by authoritative experts in a wide range of fields.

Material self-published on the World Wide Web depends solely on its content and the credibility of the author for its credibility. Material which passes the content and author tests above may be used as a reference for research purposes. Material that fails either test may not.

Do not confuse self-published material with electronic copies of material published in print journals. Credibility extended to the print journal automatically applies to the electronic copy, provided it is identical. Sometimes you will find electronic copies on an author's Web site, and that is good. Many universities and professional societies provide electronic copies of journals, and they are just as credible as the corresponding print copies.

There are beginning to be "electronic journals" which exist only as Web documents, and never in printed form. To assess the credibility of such a journal, consider who publishes it and whether it is refereed by reputable scientists. Be more careful than usual in assessing the credibility of the author and the content.

Assess the Relevance of the Material

Sometimes you will come across material by an able and well-respected author and published in a top journal It may cry out to be included in your paper. Unless it is relevant to the purpose of your paper, it will not lend credibility. Leave it out.

However, do not confuse "irrelevant" with "presents a contrary view." If you know about credible material that contradicts the thesis of your paper, you are bound by the rules of academic conduct to include it.

Keeping a "Scorecard"

Here's a scorecard you can use with each source to begin the evaluation process of deciding whether or not it is suitable to include in your bibliography.

The scores assigned are arbitrary. For that reason, I have not tried to give a cutoff point that seprates "good" references from less good references.

You should not calculate a score for each reference you are considering, then sort the scores and take the top ones. By far the most important thing is to have citations that back up the structure and content of the argument you're making in your paper. The score is a way to find where your arguments have weak or nonexistent backing, which should lead you to do more research.

As you start doing academic research and writing, your relative lack of experience may necessitate that you take a somewhat mechanistic approach to evaluation of sources. Do not lose sight of the idea that you are trying to learn how to exercise sound academic judgment.

Item	Points
Content
Presents original research	5
But not convincingly	-3
The reference list in the article contains more than three citations	3
The reference list in the article contains more than ten citations	3
The work is current	2
This article is cited in at least one other article	2
This article is cited in three or more other articles.	3
Author's Credentials
The author is a well-known academic researcher	5
or the author is associated with a well-known research institution	3
or the author is associated with a college, university, or similar institution	2
The author is a journalist	-5
Publication
The publication is a top scientific journal	5
The article is in a lesser journal or a conference proceedings	3
The article is in the trade press	-1
The article is in a popular publication	-3
The article is a self-published Internet article	-5
Relevance
The article is relevant to this research	2
And presents an opposing viewpoint	2

As you well know, it is important for writers just starting out to practice absolutely correct usage. But this doesn't mean that as they advance they shouldn't judiciously employ figures of speech, slang, or similar devices to make their points in a more rhetorically effective way. In much the same way, as you advance in doing academic writing, you should not be too bound to some arbitrary scorecard. What you should do is to learn to justify exceptions to the rules to yourself in a satisfactory manner before you use citations that don't meet the letter of the law.

A Word About Books

So far, everything has been about journals, magazine articles and conference proceedings. You will also use books in your research. You assess books in much the same way that you assess shorter publications. With the exception of self-published "vanity" books, all books are published with the idea that the publisher will make money. Even so, it is easy to see that there are popular books and scientific books. Clearly, a scientific book will carry more authority as a reference than a "For Dummies" book. Evaluate the author's authority and expertise. Check CiteSeer or Science Citation Index to see who else has cited this book. Evaluate the content.

This document is available on the World Wide Web: http://bbrown.spsu.edu/papers/credibility.html

This document may be copied, reproduced, redistributed or republished without payment of royalty provided it is copied unchanged, including preserving the author's name, the copyright notice, and this notice. If you are using Assessing the Credibility of Published Material in a formal program, the author would appreciate a note.

Last updated: 2014-10-02 7:17
Originally published: 2003