Bishops Teaching Children: Chapter 5

A Practical Method
By Which Roman Catholic Bishops
Can Personally Direct
The Religious Education
Of the Children of Their Dioceses

John Kelleher
Cambridge, Massachusetts USA
Written in the Year of the Great Jubilee
A.D. 2000

It is no part of the Church's proclamation that intellectual pursuits - of any kind - are a privileged way to Christ. After all, the Devil himself knows the identity of the Holy One of God. [Lk 4:33-34] The Devil himself is quite able to cite passages in Scripture. [Mt 4:1-11] One could easily say that the Devil has a very advanced knowledge of 'religion.'

Yet, in keeping with the proper humility of religious education, Questions written by bishops are exclusively concerned with intellectual knowledge of the faith. Children get no extra points if they love their mothers or are kind to strangers, or even if they receive the sacraments of Penance and Eucharist frequently.

Except by the grace of God, religious education can not directly bring about any of those goods - it can not make the Church. To be successful, religious education needs to be resolutely as humble as dirt. It must not pretend, even for a minute, that it has competence either to judge or to directly develop any spiritual and moral qualities, but only intellectual ones.

When a student learns, he learns as himself. His moral and sacramental nature is engaged, just as surely as his intellectual nature is. Indirectly, therefore, his moral and sacramental nature may be developed by schooling, but only in the same way that his moral and sacramental nature may be developed by football practice or helping with the dishes.

In each case, there is an opportunity for the person to develop his moral nature and to find God, but no special opportunity for moral or spiritual development exists in a class whose subject happens to be 'religion' rather than 'football.' (To say it again, this is not to excuse even a single unkind or inhumane practice in schooling, but rather to stress that all social institutions share equally, but also, only mediatively, in the task of moral development).

Religious education needs to take these irritating and simple truths seriously, at every step. On the one hand, if it rejects its exclusive focus on intellectual knowledge of the faith, it pretends to be what it is not. The Eucharist makes the Church, not religious education, and moral development is largely caught, not taught, mostly shaped mediatively in and through daily social interactions of all kinds, and very little (though a little) by direct instruction.

On the other hand, if religious education pretends that intellectual knowledge of the faith is a privileged or more certain path to salvation, then it confronts the minor problem that the Devil himself has a great amount of 'religious knowledge.'

Religious education's job is a humble one, but it is a humble job that it can do. Religious education can develop in students an intellectual understanding of the faith.

Furthermore, bishops' Questions can probe the nature and extent of that intellectual knowledge, especially if the Questions are put to students within a standardized multiple-choice format. The remainder of this chapter is an extended, and occasionally, technical, outline of how bishops can write Questions in that format.

The term 'standardized' has a definite meaning within psychometrics, the field that studies testing. It refers to a test in which the score is not dependent on who is doing the grading.

Obviously, once the real meaning of 'standardized' is known, it is easy to see why we would want bishops' Questions to be 'standardized'. If both Bobby and Betty give identical answers to bishops' Questions, we certainly would want them both to receive the same score, regardless of who happened to be grading their separate answer sheets.

Most people understand that a major cost of testing is grading the tests after they have been answered, and therefore can appreciate that multiple-choice tests, which can be machine-scored, are considerably cheaper to grade. Further, most people can quickly be convinced that it would be impossible even to conduct large-scale tests if human beings had to grade each separate answer sheet. Imagine paying an army of people to grade 100,000 tests!

What most people do not realize, however, is that, for probing intellectual knowledge, good, standardized multiple-choice tests are considerably more reliable and more fair than most testing alternatives, and are substantially as reliable and as fair as the most expensive testing methods.

Although superficially it seems to go against common sense that one particular 'cheaper' method of testing is actually about as reliable and fair as the most expensive version, we should first consider that many cheap tests are barely worth the paper they're printed on. That is, one kind of cheap test (multiple-choice) - not all cheap tests - can be virtually as good at testing intellectual knowledge as any test we know how to devise, no matter how much we spend.

Multiple-choice tests actually represent something every American treasures: an actual bargain - high value for much less money. Since the idea that a cheaper test could be as good or better than an expensive one does superficially violate common sense, and since it strenuously violates the persistent claims of large segments of the American educational establishment, multiple-choice tests have undergone rigorous scientific scrutiny over many years. The results, for those not religiously committed to the claims and 'research' of educational progressivists, are unequivocal. Although low-quality multiple-choice tests certainly do exist, high-quality multiple-choice tests can be devised, and they are an actual bargain - high value, even the highest value, for much less money.

For reasons that are still not completely clear, the multiple-choice format can function as elegantly as a haiku or a sonnet. We do not imagine that a haiku is unable to touch something deep in us because it is 'limited' to seventeen syllables, or complain that all sonnets are invariably 'inauthentic' and 'lifeless' because they must follow a fixed meter and rhyme, and be exactly fourteen lines long.

Yet that is exactly the kind of thing many people say about all multiple-choice tests.

It definitely is possible to write a ludicrously inane haiku, or a sonnet so stilted and lifeless that the word 'amateurish' hardly suggests the travesty. It is just as possible to write truly bad multiple-choice questions. Yet in all three cases it is not true that the form itself makes meaning and depth impossible.

In fact, we can relish the opportunities provided by the form of the multiple-choice question, almost in the way that a poet might relish the opportunities provided by the form of the haiku or the sonnet.

That, of course, is wildly exaggerating the intellectual and emotional thrill of writing a good multiple-choice question, but there is no doubt that it is not that easy to write a good one. Writing good multiple-choice questions requires a pronounced grasp of the subject, knowledge of how to phrase questions so as to avoid confusions and misunderstandings, a modicum of ingenuity, the ability to write clearly and cleanly, and a real teacher's sense of the genuine but plausible errors that might be made in the particular context set by each question.

Having established that 'standardizing' bishops' Questions is something we definitely want to do, and having at least suggested that the form of multiple-choice tests has nothing to do with whether they are relevant, probing, and deep, it seems right to at least mention that there is no substantive basis for the other charges made by educational progressivists against the kind of standardized multiple-choice format that would be used for bishops' Questions.

As is customary within educational progressivism, an entire cottage industry exists within it to continually repeat these anti-testing claims and to do and cite 'research' that substantiates them, but in fact all substantive progressivist objections to standardized multiple-choice tests have been answered by repeated and numerous scientific investigations.

In his important book The Schools We Need And Why We Don't Have Them, E.D. Hirsch, Jr. devotes an entire chapter ("Test Evasion") 1* to standardized multiple-choice tests.

"Don't tap higher-order or real-world skills" needs to be changed to "Don't necessarily tap higher-order or real-world skills." However, "Can't tap higher-order or real-world skills" is flatly false.

"Are unfairly biased against certain groups, and indeed have contributed to the inequities they reflect" - ditto. The differences in scholastic achievement among groups are real.

What the differences in test scores actually show, therefore, is that the application of current American educational theory and methods, bad for all, has its worst effect on children who lack advantages such as the 'right' race or class, or in other words, that educational 'progressivism' is profoundly anti-progressive in its effects.

No wonder educational progressivists don't like standardized tests. Those tests consistently show that schooling children according to the theories and methods of educational progressivism does not narrow, or even widens, the academic gap between races and between rich and poor.

So, the form bishops' Questions will take will be standardized multiple-choice, but that says nothing about how probing, relevant, deep, and elegant those Questions can be. What could be concluded about sonnets from the knowledge that both I, and Shakespeare, have written one?

The wonderful difference between multiple-choice questions and sonnets is that, unlike writing sonnets, someone like me can learn the rudiments of how to write a good multiple-choice question, and then I can let trial-and-error hone my effort until it becomes an objectively good question.

The rudiments of how to write bishops' Questions will be discussed a little later on this chapter, but for now it might be good to see the broader picture in six main points.

2. The test should be in four parts, corresponding to the four parts of the Catechism of the Catholic Church.

3. On the test, all students, including the very youngest, must answer a series of progressively harder questions until they either 'test to failure' or establish that they have a defined adult minimum mastery of the matter of each of the four parts of the Catechism.

4. The test should be extensive enough that the amount of knowledge each child possesses of each part of the Catechism can be determined with good precision.

5. Every single Question must be written either by the local ordinary himself, or by another bishop of the Catholic Church, and it must be known by all that only bishops wrote the Questions. It must be absolutely clear to all that each and every Question resulted from the direct exercise of the episcopal office.

6. At least at present the Catechism of the Catholic Church must serve as the direct basis of all Questions, and no impulse to frame Questions to teach 'beyond' the Catechism should be indulged. The reasons for this are given below.

The Bishops Teaching Children method is highly intrusive ('transparent'). It gives both the local ordinary and the entire local church a great deal of power, a power that can be abused as well as used, as outside the sacraments "the minister leaves human traces that are not always signs of fidelity to the Gospel and consequently can harm the apostolic fruitfulness of the Church." [CCC 1550]

Further, the transparency intrinsic to the Bishops Teaching Children method enables many others, including both friends and enemies of the local church or the whole Catholic Church, to readily determine the extent of students' knowledge of the faith. This can be a great source of benefit to the local church and even a means of evangelization, but it should be noted that with this knowledge those others also will possess additional power, which, like all human power, can be used both for good and for ill. Such risks can be assumed with a glad heart only if it is clear that their assumption is part of the expression of the sacramental character of the local church and the sacramental and apostolic character of the local ordinary as chief teacher.

Furthermore, the writing of Questions, and the Bishops Teaching Children method itself, are not to be taken as exhausting either the local ordinary's apostolic office as chief teacher or the sacramental character of the entire local church to evangelize and to teach. Other, far less intrusive forms and structures for teaching can readily co-exist alongside the Bishops Teaching Children method, if the bishop and the local church desire it. These forms would seem to be on their face more appropriate for teachings that, while considered pastorally or locally appropriate and perhaps no less valid, are not the proclamation of the whole Catholic Church. [also, see the discussion on 'intrusive' in the next chapter.]

For all these reasons it seems right that, when using the Bishops Teaching Children method, the ordinary should be able to prove unequivocally that his apostolic authority as chief teacher is being exercised solely to hand on intellectual knowledge of the universal faith of the Catholic Church to those in his diocese. Therefore, at least at present, ordinary prudence appears to dictate basing Questions solely on the Catechism itself.

As suggested just above, at least for now, 'intellectual knowledge of the faith' ought to be a term identical in meaning to 'intellectual knowledge of the Catechism.' Thus, 'all' should learn 'as much as possible' about what is taught and proclaimed in the Catechism.

Paradoxically, students can not learn 'as much as possible' until a minimum standard is set. Until a lower limit for success is clearly defined, 'as much as possible' risks devolving into 'whatever the present system of religious education happens to accomplish.'

The technical term for examinations that pre-define a minimum standard is criterion-referenced. Another alternative is normative examinations, otherwise known as 'grading on the curve.' There are sometimes reasons to use normative examinations, but the fact remains that on normative examinations, as long as everyone else has even less knowledge than you, you can always remain 'above average,' no matter how little you know.

Bishops' Questions must be criterion-referenced. That is, the writing of Questions must focus on ascertaining whether 'all' have demonstrably reached a reasonable minimum adult knowledge of the Catechism in all four of its parts.

Criterion-referenced Questions do exactly what is needed for the Bishops Teaching Children method to work correctly: they focus the entire system of religious education in a diocese on what needs to be known, rather than on what needs to be taught, or on how it is taught.

To say it plainly, the Bishops Teaching Children method is, of its very nature, completely incurious about either curriculums or teaching methods. All it cares about is the ultimate goal of religious education, adult-level minimum mastery of all four parts of the Catechism.

Both curriculums and teaching methods are merely means to a goal - adult-level knowledge. The current system of religious education at very least diminishes the importance of this irrefutable fact. The Bishops Teaching Children method, with criterion-referenced Questions, makes that fact crystal clear to everyone, year after year.

Moreover, the Bishops Teaching Children method, with criterion-referenced Questions, has the proper sacramental foundation. No local ordinary need be an expert in curriculums or methods to use it. He need not decide what is the 'appropriate' age to introduce certain subject matter. He need not even be curious about such things. However, by sacramental ordination and apostolic authority he is competent to 'hand on' what should be known and to monitor whether it is known, and that is exactly what he does in his yearly Questions.

Thus, the primary goal of Questions for students of any age is not to establish that they have 'mastered the material appropriate to their grade' but far rather, to ascertain whether they have attained minimally-competent adult knowledge in all four parts of the Catechism. This, after all, is the whole purpose of their religious education, and thus bishops' Questions should monitor it directly.

In the yearly Questions, therefore, all students, including the very youngest, must answer a series of progressively harder questions until they either 'test to failure' or establish that they have a defined minimum mastery of the matter of each of the four parts of the Catechism.

Bishops' Questions should focus on adult minimal mastery not only because that is, after all, the goal and purpose of religious education, but also because it presents Competitors with the most incisive climate for thorough Competition. Why put artificial limits on the ambition and imagination of Competitors?

If a Competitor is able to demonstrate that, using his methods and curriculum, sixty per cent of children can attain minimal adult mastery of all four parts of the Catechism by the end of eighth grade, why shouldn't bishops' Questions make it easy for him to establish that? The obvious and easy way to see if any or all students, in the eighth grade, the twelfth grade, or the first grade, have demonstrated minimal adult mastery of the Catechism, is to ask students to try to demonstrate that.

The Bishops Teaching Children method has no theory about the best way to get students to that point. To the contrary, it is constitutionally incurious about methods and curriculums. All it cares about is where students end up. It has no interest in how or even when they get there. Establishing the methods and curriculums that get all students to adult minimal mastery of the Catechism as quickly as possible is the (literal) business of Competitors only.

For these and for many other reasons, bishops' yearly Questions should focus on assessing each students' 'test to failure' point in each of the four parts of the Catechism.

Thus, in marked contrast to the current system of religious education, the Bishops Teaching Children method sets no grade-by-grade curriculum for Competitors to follow. Instead, it is exclusively and incessantly curious about the sole goal of religious education, which is not methods and curriculums, but minimal adult mastery of the Catechism by 'all' the children of the local church.

Bishops qua bishops are not authorities in how to develop a carefully-sequenced curriculum that provably brings all the children of the local church to adult mastery of the Catechism as efficiently as possible. Indeed, the Bishops Teaching Children method argues that such 'expertise' is never established by anyone's 'credentials,' but only by rigorous Competition in the real world.

Thus, if bishops should set a grade-by-grade curriculum, they would range not only far outside their own sacramental competence, but also far outside anyone's actual practical competence. Even the best Competitors will forever only have theories about what might work most effectively. These still have to be tested against each other in fair Competition, and at any rate are only 'better' in a provisional sense. At any moment, a Competitor can prove himself to be even better than the current 'better,' and that is the way it should be.

At best, then, a bishop who presumes to set a grade-by-grade curriculum is deferring to his own educational theory, or to that of his advisors, as if he or his advisors were some kind of privileged Competitor with special knowledge others lack. None of this has anything to do with 'handing on' the faith of the whole Catholic Church.

On the other hand, a bishop is qualified to 'hand on' that faith, and as chief teacher is authorized to monitor 'all' the children's intellectual knowledge of it.

Also, the children of a diocese are going to be much better protected by the Bishops Teaching Children method, which focuses its rewards and punishments not on prima facie adherence to a curriculum but solely on results; that is, on how much Competitors have increased the number of children in the diocese who have achieved minimal adult competence in the Catechism. 2*

To repeat, the Bishops Teaching Children method is radically incurious about curriculums and methods. The practical reason for this lack of curiosity is that opinions about curriculums and methods pre-limit the imagination and ambition of Competitors. As a purely practical matter, that will hurt far more children than it will help.

However, the real or sacramental reason to preserve constitutional incuriosity about curriculums and methods is that even the tiniest particle of such curiosity goes well beyond both a bishop's apostolic authority and the sacramental character of the local church. A bishop qua bishop has precisely nothing to say about curriculums and methods. This belongs to the essential freedom of the faithful, and of Competitors.

Nonetheless, within an implementation of the Bishops Teaching Children method, there is still a role for prima facie evaluation. Before becoming part of ongoing Competition, curriculums and methods should be checked for prima facie consistency with the Catechism. Such evaluation would serve as the equivalent of the initial toxicology tests that the U.S. Food and Drug Administration requires for new drugs. When a parish is required by the local ordinary to take up an experimental curriculum and method a few times every twenty years in the interests of preserving the modicum of wild entrepreneurship that is essential to continued stiff Competition, the parish is at least entitled to the assurance that nothing explicitly contrary to the faith will be taught, and that all four parts of the Catechism will be taught.

The Catechism is a big book. Imagine a bishop, determined to use the elements of the Bishops Teaching Children method, alone in his study for the first time with a blank piece of paper and a copy of the Catechism. He knows his job as bishop is to write Questions. How should he proceed?

As stated, his focus should be on Questions that establish 'minimally competent' adult intellectual knowledge of the Catechism. (Further insight into how he might select Questions which, taken as a whole, set the standard of adult 'minimal competence,' is given shortly.)

Second, his Questions should focus on each of the Catechism's four parts: the profession of faith, the sacraments of faith, the life of faith, and prayer in the life of faith.

Thus, from the beginning it should be understood and emphasized that the four parts of the Catechism make up an organic unity, each part reflecting on and adding to the whole, and thus Questions should test student knowledge in all four parts, to the neglect of none of them.

Where to begin? A bishop can begin to organize his Question writing, if he wishes, with reference to the 'In Brief' summaries at the end of each thematic unit. The Catechism itself rather lamely suggests that these sections "may suggest to local catechists brief summary formulae that could be memorized." [CCC 22] We should perhaps all be grateful that neither the Catholic faith, nor the Bishops Teaching Children method, requires even a single bishop to be an educational 'expert.'

In fact, the 'In Brief' summaries can have a rather more important purpose - they can help to organize bishops' thoughts as they write Questions. Questions that do not at least cover the matter directly summarized in the 'In Brief' sections may not be adequate to a bishop's apostolic task.

Nor (more subtly) may Questions be adequate which, in their overall import, do not seem to regard the 'In Brief' sections as summing up "the essentials of that unit's teaching in condensed formulae." [CCC 22] Questions that address other teachings while down-playing those covered in these summaries may not adequately represent the Catechism to students.

In sum, bishops have already at hand at least a first approximation of the knowledge all minimally-competent Catholic adults should possess about the Catechism in the 'In Brief' sections of the Catechism itself.

A brief mention needs to be made of a crucial, and rather technical, feature of criterion-referenced exams. As is shown below, all questions on criterion-referenced exams need to be relatively discriminating. Even so, 'good' Questions will still vary in difficulty. There will be a range of difficulty among the Questions.

There are some things that the bishop would not expect every single 'minimally competent' person to know, but he would expect some of them to know. On the other hand, there are some things he might expect that virtually every 'minimally competent' person would know.

So, a bishop might decide that only 60 percent of persons he would consider to be 'minimally competent' would be able to answer a particular Question. On the other hand, he might decide that 95 percent of people he would consider to be 'minimally competent' would be able to answer another particular Question.

No panel of 'experts' can decide this for a bishop. The bishop alone is competent to decide this percentage, and he and he alone must decide it, for each and every Question.

The Bishops Teaching Children method is at pains to emphasize this again and again: bishops alone are sacramentally competent to decide what 'adult minimal competence' in the Catechism is. When a bishop uses the Bishops Teaching Children method, he defines what 'adult minimal competence' in the Catechism is not only by what Questions he writes, but also by what percentage of persons that he would consider to be 'minimally competent' would answer each Question correctly.

Both steps are thus a direct exercise of the local ordinary's apostolic authority and sacramental competence, and are not within the competence of any other human being except another bishop. Furthermore, the local ordinary, as the ordinary of the local church, typically exercises his apostolic authority and sacramental competence in this regard without encumbrance even from other bishops, though of course he may consult with them as he deems appropriate.

Once the bishop has written all the Questions, and decided on the percentage of 'minimally competent' persons who would answer each one correctly, the Scientist can take this information and, using technical means, determine the 'pass point' for each yearly set of Questions. The 'pass point' is thus not an arbitrary figure that applies to every single yearly set of Questions, but only to a particular yearly set.

For instance, suppose a bishop decided that 95 percent of 'minimally competent' persons would be able to answer every single one of his Questions. Then the 'pass point' for that yearly set would obviously be 95 percent. If another yearly set contained a mix of Questions, some of which the local ordinary expected only 60 percent of 'minimally competent' persons to be able to answer, then the 'pass point' for that yearly set would obviously be lower than 95 percent. Each 'pass point' would be unique to the particular yearly set of Questions.

As Competitors improve methods and curriculums over time, it may even be that bishops would then raise the standard of 'minimal' adult competence. Twenty years from the introduction of the Bishops Teaching Children method, a bishop may decide that a level of knowledge once barely attainable for high school seniors, but now easily attained by eighth graders, no longer represents what should be minimal adult competence, and write, or weight, new Questions accordingly. In any case, it remains remarkable how perfectly the Catechism is suited to the Questions element of the Bishops Teaching Children method. 3*

Finally, it should be recalled that the Bishops Teaching Children method makes the yearly set of Questions 'high stakes' for responsible adults, but not for children. If a child has tried hard all year, beyond the fact that every single person in the world will know his score, absolutely nothing bad will happen to him when he answers the bishop's yearly set of Questions. On the other hand, 'bad' things can indeed happen to hard-working, but ineffective, Competitors - they can lose their business. The local ordinary can set the standard for 'adult minimal competence' as high as he considers appropriate. It is up to Competitors to figure out how to get the children to that level.

When Using the Bishops Teaching Children Method, a Bishop Should Not Teach 'Beyond' the Catechism

Before delving further into Question-writing, it is well to stress another aspect of the constitutional humility of the Bishops Teaching Children method. While the Holy Father has declared the Catechism to be "a sure norm for teaching the faith," 4* and the Catechism "is conceived as an organic presentation of the Catholic faith in its entirety," [CCC 18, emphasis original] the Catechism almost certainly should not be taken as a mere set of 'logical' propositions, particularly if that is taken to mean, propositions whose interrelationships can 'logically' be derived immediately by any person whatever. To the contrary, interrelationships within the economy of salvation are instead worked out and developed in wonderful mystery within history in and through the Church's sacramental worship, and in and through the very bodies of the saints as they join in that eternal worship.

After all, the Catechism as a set of 'propositions' available to the manipulations of 'logic' makes a pre-existing, eternal, and universally available 'logic' the lord of everything, lord even of the Catechism, lord even of the living faith of the Church, a god which presumably we therefore ought to worship instead of the Most Holy Trinity. We need to take seriously St. Paul's plain statement that "the foolishness of God is wiser than men, and the weakness of God is stronger than men." [1 Cor 1:25]

The confidence of the Church, that she can understand her Lord, and understand more and more of Him, is unshakable, part of her very union with her Lord, the New Covenant itself. Yet clearly, much of the mystery of salvation remains obscure to men.

Bishops' Questions need to be at least as patient as the Church herself. Efforts to write 'advanced' Questions are laudable, and it is indeed possible to write 'advanced' or more sophisticated Questions, but bishops' Questions can not engage in speculative exercises, however 'logical,' since these are not what bishops are charged to 'hand on.'

As stated previously, the Bishops Teaching Children method is simply too intrusive and powerful to warrant such speculative Questions, however attractive they might seem. The Bishops Teaching Children method has its sacramental foundation in the sacramentally-ordered function of 'handing on.' It may be used for this, if a local ordinary and a local church wish.

The Bishops Teaching Children method could be used by the irresponsible to teach theology, or for a great many other purposes. It should not be used for any of these other purposes, even to teach theology. The intrusion into the freedom of the faithful is otherwise too great.

The faithful indeed may organize the schooling of children, in theology or in any other subject, with reference to mechanisms and structures similar to the Bishops Teaching Children method, but no one may disregard the essential difference between these and the Bishops Teaching Children method as implemented by a local ordinary and a local church, which is, the sacramental and apostolic foundation of that specific original version, a foundation that simply does not exist in any variant established for any other reason.

Which is to say, although the faithful have the freedom to build and use variants of the Bishops Teaching Children method to assist in the schooling of children in other matters, no bishop and no local church has any business imposing such a variant on anyone, however prudential or practical it may appear. Such an exercise would go well beyond the sacramental character of the local church, and well beyond the sacramental and apostolic authority of the local ordinary, and would therefore be a serious abuse of the essential dignity of the faithful. The Bishops Teaching Children method should be used by the local ordinary solely to teach to 'all' the faith the local church and the local ordinary profess in union with the whole Catholic Church.

So, bishops' Questions should deal with all four parts of the Catechism to the exclusion of none, they should, taken as a whole and within each part, characterize a standard of knowledge which represents an adult level of minimal competence, the 'In Brief' summaries are obvious starting and reference points to this end, but notwithstanding the above, bishops should take due care to avoid Questions that depend on intellectual knowledge that is not part of the profession of the whole Catholic Church.

Do Bishops Need Advanced Technical Training Before They Can Write Questions?

How should Questions be phrased? A recommended form for multiple-choice questions will be given shortly, but what of their content? Do bishops need to amass a great deal of technical knowledge, or go to school to learn how to write 'sophisticated' Questions, before they can write Questions?

The answer is no. In the first place, as long as bishops keep their Questions focused on two things: faithfully 'handing on' what the Catechism teaches in all four of its parts, and opposing heresies with all their might, again regarding all four of the Catechism's parts, then rest assured, those two jobs alone will require all the 'sophistication' that bishops can muster.

However, as noted above, bishops who have a firm grasp of the Catechism themselves, who can write clearly, and who have a teacher's knack for knowing just how the thinking of students might falter in a particular instance, will tend to write better Questions than other bishops.

It should also be mentioned that bishops with good knowledge either of known heresies, or of perennial misunderstandings of Church teaching, may also tend to write better Questions, because they may write better 'distractors.' As defined below, 'distractors' are the other possible choices on a multiple-choice question (besides the correct answer). They must be plausible, but definitely wrong, responses. By definition, known heresies and perennial misunderstandings were 'plausible' to some, at least at some points in history. Thus, writing distractors that articulate known heresies or perennial misunderstandings may serve both as a teaching tool and as a way to write better Questions.

Nonetheless, from a technical point of view the initial process of writing Questions is only the beginning of a long process of trial and error. In effect, over time, the students themselves will tell bishops what 'good' Questions look like.

A few subtle and perhaps surprising points now need to be made. First, examinations such as bishops' Questions do not actually measure a student's overall knowledge. What these examinations do instead is sample that knowledge.

Suppose we wanted to sample a child's knowledge of addition facts. We ask him, "What is 6 + 8?"

Now, if he immediately says, "14," do you think the chances go up or down that he also knows many other addition facts, besides the specific one you asked him?

Suppose you chose nine more addition facts at random, and he immediately gave you the correct answer for each one. Would you be willing to bet, from his ten correct answers in a row, that he knows all of his addition facts very well?

Yes, you would, and you would be well-justified in doing so. Because you did not actually ask him every single addition fact, there is a chance that he does not know a particular one, but this chance is relatively small.

This is how examinations like bishops' Questions work. They sample student knowledge, and thus can predict, within a certain level of accuracy, how the student might have answered other questions that were not specifically asked. They are not perfectly accurate measures of student knowledge, but even without asking anything like the number of questions needed to perfectly assess student knowledge, they can estimate the extent of student knowledge with relatively high precision.

Thus, bishops' Questions are designed to give everyone a relatively good idea of how much a student knows within a reasonable amount of time, while spending a reasonable amount of money. As noted above, it turns out to be surprisingly difficult to estimate a student's overall knowledge with better accuracy than that which good standardized multiple-choice tests provide. Good multiple-choice tests are not perfect, but they are surprisingly difficult to improve upon, for any amount of money.

In other words, there is no 'perfect' test. Nor do any known tests accurately assess some things, like creativity, that we would very much like to know about. Nonetheless, good standardized multiple-choice tests measure what tests can measure about as accurately and as fairly as any test we know how to devise. They can measure (for instance) how closely a student's intellectual knowledge of the Catechism approximates that of a defined minimally-competent adult. They do this by sampling, rather than really encompassing, the extent of a student's knowledge.

Thus, yearly Questions do not ask students to tell the local ordinary everything they know about the faith. The point is to ask a relatively small number of Questions that provide a reasonably accurate estimate of each student's overall knowledge. Ask too many Questions, and you will tire the students unduly. Ask too few, and the precision of your estimate of their knowledge degrades.

A 'Bad' Question is Simply One That Does Not Distinguish One Student From Another

That being so, every single Question has to mean something. They all have to count. Each and every Question has to help you estimate whether a student has mastery, or does not. Thus, a 'bad' Question is simply one that nearly all students answer in the same way.

It doesn't matter if everyone gets the Question wrong or right. The point is, if everyone is answering it the same way, the Question does not help you distinguish students who know the material from those who don't. The Question does not improve your ability to estimate whether a student has mastery. Therefore, it is a 'bad' Question.

It bears repeating constantly: religious education is as humble as dirt, and the Bishops Teaching Children method is humble to the point of ridiculousness. The Bishops Teaching Children method works, but it only works because it has very humble goals, and sticks to them. Within an implementation of the Bishops Teaching Children method, there is no lofty definition of a 'bad' Question. A 'bad' Question is simply and only a question that makes your job - to distinguish students who have mastery from those who do not - a bit harder. Thus, whether a Question is 'bad' can be answered only by trial and error.

This is how the process of trial and error works in Question writing. We might, of course, be delighted if it turned out that 97 per cent of Catholic first-graders got the following Question correct:

He probably hoped that the Question would be more discriminating. He may even have been certain that it would be. But as it turns out, 97 per cent of his first-graders answered it the same way. Of course it is wonderful that so many of the first-graders know that God thinks they are worthy of happiness with him in heaven. The Question remains a 'bad' Question, and he must write another one.

If Questions were a compendium of student knowledge, it would be perfectly all right to keep that Question. But since Questions are a sampling of student knowledge, that Question can not be kept, since it makes the job of Questions - to sample differences in knowledge with relative precision - harder to accomplish. That Question gave the local ordinary almost no information about how children with less mastery could be distinguished from children with more mastery. Therefore, it is a 'bad' Question.

So, the children themselves 'told' him he had written a 'bad' Question. This is exactly how bishops learn whether their brilliant and probing Questions were actually good or 'bad.' The children tell them.

As to the form within which Questions are put to students, the multiple-choice format has had many expressions, but this 'bad' sample Question does exemplify one of the clearest and cleanest modern variants of the form of multiple-choice questions. This is the form recommended for bishops' Questions, so this 'bad' Question that is a good example of the proper form for Questions will be reproduced here and discussed below.

In other words, a good stem gives enough information to at least suggest the general context for the correct answer. An example of a 'bad' stem which violates this principle would be

"All of the above" violates the rule that only one of the responses should be correct, as does "A and B, but not C."

In addition, responses like "all of the above" are not actually answers themselves, but simply refer to other, 'real' answers. This is another reason they should be omitted - they are not real answers.

In short, one of the responses in each question should be the correct answer, and the other two answers should be plausible but definitely incorrect answers (which are called "distractors"). 'Plausible' also implies that made-up words are not allowed in distractors, even if they do bamboozle an unsuspecting few.

Two additional rules that apply to both stems and responses increase the clarity of both.

and similar phrases, should all be rejected. It is well-known that processing a negative places additional cognitive demands on the reader. We don't want to know if the student can process negatives. We want to know if he knows the answer. Avoid the negative.

To repeat what was stated before this technical discussion began: the form recommended for bishops' Questions is stringent. The effect of these rules is to make everything as clear as possible for the test-taker, at a considerable cost in time and effort to the test-writer. Following the rules given above requires the question writer to create multiple-choice questions that are easy to read and to interpret. The only difficulty remaining for the test-taker then becomes answering the question correctly, and this is exactly the way a question writer should want it.

Each Question is typed separately on a single sheet of paper, with the correct Response appearing in boldface, or underlined, or otherwise clearly marked. Below the Question, the section or sections in the Catechism that the Question references should be indicated. Any other specifically relevant comments (for example: "Response a. is the Arian heresy.") should also be briefly noted. Finally, the name of the bishop who wrote the Question should be given.

Each written Question (obviously, the Question alone, without any indication of the correct answer and without the supporting documentation) is then given to a Scientifically-relevant small sample of children, whose responses to it tell the bishop whether it is a 'good' or 'bad' Question (as was outlined above). All the pages with 'good' Questions are then assembled into a master set, from which each yearly set of Questions is prepared.

In advance of any actual attempt to create the elements of the Bishops Teaching Children method, virtually all that can be said about bishops' Questions has now been said, save for two things.

The first can be handled briefly. How can we expect little children to answer Questions that are referenced to the intellectual knowledge displayed by a minimally-competent adult?

In brief, the answer is trial and error. Remember, the form each yearly set of Questions should take is a sequence of progressively more difficult Questions for each of the four parts of the Catechism, during which each student either 'tests to failure' or establishes minimal adult competence. There is no theoretical reason why the beginning levels of those progressively more difficult Questions could not be answered by little children.

There are some technical details that have to be handled, but the true answer really is trial and error. The children will tell a bishop when he's written a 'bad' Question of any kind, including those that are 'bad' because they are not understandable. A bishop just has to keep writing Questions until he writes enough that clearly distinguish the knowledge of one first-grader - and one twelfth-grader- from another. 5*

This brings up the second point. Although the total number of multiple-choice Questions on the yearly examination will not have to be huge (probably around 100 - 200), the total number of Questions that bishops will have to write will be much, much larger.

In the first place, trial and error must operate. The 'good' Questions must be sorted from the 'bad.' That probably means writing many 'bad' Questions for each 'good' one that eventually emerges.

Second, for security and sampling purposes, enough 'good' Questions to create many different yearly examinations (around 10 in all) will eventually have to be devised. This labor would be considerably compounded if, as a teaching tool, a local ordinary released the contents of each examination after Pentecost, and explained to all students why each distractor was not the correct answer. All of those questions could not be used again, and all-new ones would have to be written.

Thus, a very large number of Questions have to be written before even one bishop and his diocese could implement and use the Bishops Teaching Children method. However, that local ordinary does not have to write all those Questions himself.

Any bishop has the sacramental competence to write a Question. One bishop, desirous of using the Bishops Teaching Children method, could - probably must - enlist the support of many of his brother bishops around the world in order to develop enough Questions to implement and use it.

Indeed, beyond the practicalities, given the intrusive character of the Bishops Teaching Children method and the sacramental gravity it encourages, it is probably a very good idea for the local ordinary to have a very wide, perhaps even a universal, support from the episcopate.

Indeed, there is no reason that the Holy Father himself could not write a Question or two. (Naturally, all Questions given to the children would appear to be written anonymously.)

Imagine the excitement and awe in many Catholic families, and in many Catholic children, if they knew that bishops from all over the world had helped to write the Questions on the yearly Pentecost examination. Surely, there would every year be speculation about which, if any, of the Questions had been written by the Holy Father himself!

One could well ask if any current system of Catholic religious education anywhere, would so clearly represent the sacramental character and apostolic authority of the local ordinary in union with the universal episcopate, or the sacramental character of the local church in union with the whole Catholic Church, as this yearly Pentecostal giving of Questions to 'all' the children of the local church.

1. Hirsch ED Jr. The Schools We Need and Why We Don't Have Them. New York: Doubleday. 1996.

2. The Science part of the Bishops Teaching Children method must therefore include longitudinal evaluations of Competitors' methods and curriculums, because religious education to adult minimal competence will be a multi-year undertaking for the foreseeable future.

Many wayward methods and curriculums will work briefly, for a few students. Some will show promise that fades. Very few will be initially safe, promising over the short term, and demonstrably more effective at achieving the sole goal of religious education, adult minimal competence in the Catechism.

These, in the end, are the only curriculums and methods the Bishops Teaching Children method wishes to reward, and thus both short-term and longitudinal testing of Competitors' methods and curriculums is essential. In both cases, bishops' Questions, resolutely focused on adult minimal mastery of the Catechism, are the necessary foundation for evaluation.

3. Indeed, to those who understand the technical nature of question writing, it would be virtually impossible to exaggerate what a treasure the Catechism is. It is authoritative. It is comprehensive, presenting "the essential and fundamental contents of Catholic doctrine, in the light of the Second Vatican Council and the whole of the Church's Tradition." [CCC 11] It even includes unit-by-unit summaries!

It is normally difficult indeed to cobble together something even fractionally as good to serve as the basis for multiple-choice questions in some academic or professional discipline.

'Traditional' religious education never really imagined modern mass schooling, in the sense of the whole populace more or less attaining a relatively sophisticated intellectual knowledge of the faith, but instead did truly emphasize "brief summary formulae that could be memorized," [CCC 22] as the Catechism itself perhaps unwittingly reminds us.

Yet the American Catholic historian Mr. James Hitchcock was astute in suggesting that part of the meaning of the Second Vatican Council was the recognition by the Church that, particularly in developed countries in which mass schooling was a reality, "it would be obscurantist to try to preserve a pseudo-peasant culture of implicit meanings." [Hitchcock J. Recovery of the Sacred. 2nd ed. San Francisco: Ignatius Press. 1995. p. 132.] However, that kind of low-level intellectual knowledge is virtually all that 'traditional' religious education was designed to cultivate in the general population.

The saying goes, "Even a stopped clock is right twice a day." Educational progressivists are correct in contending that merely the acquisition of memorized formulas, however well-composed and authoritative those formulas are, is not the same as the attainment of sophisticated knowledge.

Into this technical and imaginative educational breach strode Catholic educational progressivists, promising the attainment of what traditional religious education, which was virtually only the memorization of formulae, had not even dared to imagine: true mass religious education - an entire populace attaining sophisticated knowledge of the faith.

However, at that very moment, true mass religious education became again impossible, since progressivist educational methods do not actually work. Indeed, the commonplaces of secular educational progressivists became the sophisticated discourse of American Catholic religious education 'experts,' to the point that it became virtually a theological given in some 'sophisticated' American Catholic quarters that the faith could not possibly be 'authentically' articulated in 'bookish' or 'intellectual' terms.

Thus, for various reasons, neither 'traditional' nor 'progressive' Catholic schooling could even imagine the possibility, let alone the necessity, of something like the Bishops Teaching Children method.

Then, on October 11, 1992, everything changed. Pope John Paul II promulgated the Catechism of the Catholic Church. By simply ignoring the (erroneous) contention of educational progressivists that 'authentic' matter can not be expressed in 'bookish' terms - and by being such an authoritative, comprehensive, and helpful articulation of the faith - the Catechism made the Bishops Teaching Children method imaginable and practical.

At last the technical and scientific wherewithal necessary for the Bishops Teaching Children method could be joined not only with authoritative content, but also and equally importantly, with a reaffirmed conviction that the faith of the Church, while hardly confined to verbal articulation, had a definite content that could be articulated, and therefore, learned. On October 11, 1992, the Bishops Teaching Children method, at long last, became imaginable.

That is, the Bishops Teaching Children method is now imaginable, as long as 'religious education' remains as humble as dirt, focused solely on intellectual knowledge of the faith, and sharply distinguished from both catechesis and moral development. The essential and foundational humility of the Bishops Teaching Children method always bears repeating.

4. John Paul II, Pope. Apostolic Constitution Fidei Depositum. October 11, 1992. Section 3.

5. The technical problems for test-to-failure (or more positively, testing to minimal competence) tests are considerably reduced when these tests are computer-assisted. The test is not only taken on a computer, but also, a computer program running in the background is immediately noting the answers given and trying to zero in on the level of difficulty at which the student can no longer provide answers. The program will thus provide a more difficult or less difficult next question, depending on the answers given to the previous several questions, until it can establish the level at which the student often answers incorrectly.

Each test-taker in effect then takes his own personalized test, being given just those Questions from the overall bank of bishops' Questions which zero in as quickly as possible on his personal test-to-fail level on all four parts of the Catechism. Therefore, (given that enough Questions have been written) it would be just as possible to test each first-grader as each twelfth-grader.

The difficulties are greater with paper-and-pencil versions, but are not insoluble. For instance, computer-assisted testing can be used on a sample of students to establish what would be a reasonable test for most first-graders. A paper-and-pencil version of this test can then be given to students, and so forth.