Your professor, Dr. Davies, is not a detail-oriented person and his work needs to be checked and re-checked to make sure he didn't screw something up. He's done it before.
Mysteries of the Mind has four quizzes. The Last quiz is the "final," but we try not to call it that.
Check Quiz Dates
What you need to do:
Make sure there are four quizzes on the syllabus and on Brightspace. Verify that the dates are the same for the syllabus and Brightspace. If there is a discrepancy, contact Dr. Davies.
Brightspace
Brightspace does not allow us to schedule the Friday and Monday quiz time options at once. So what you need to do is to go into Brightspace after the Friday quiz (on Saturday, preferably) and change the quiz date and time to the Monday. This needs to be done with every quiz, so make sure it's in your calendar for the semester.
Moodle/CULearn
Click “turn editing on” (top right)
Beside each quiz, Edit -> Edit Settings
Check on text box and make sure dates are correct there.
Open “timing”
Verify times are correct (should open for the Friday quiz and close after the Monday alternate time)
Open “restrict access”
verify times are correct (restricted between the Friday and Monday quiz times)
Click “save and return to course” if you made any changes.
Repeat for each quiz.
Check Quiz Questions
Before the first quiz, please go through the quiz questions and make sure that there are no quiz questions from bank categories that are not covered. Sometimes questions accidentally end up on the quiz that are not supposed to be there and it's a big headache for everyone.
Each quiz is cumulative, which means that it covers material from previous quizzes as well as the newer stuff.
Because this is an online class too, the online students do not get the lectures until a few days after the in-class students get them. This means that we can't test on Friday stuff that was presented Thursday. To be safe, I don't quiz them on anything from the last two lectures. Everything covered before that should be on the quiz.
What you need to do:
Check to make sure that my syllabus is correct regarding what material is covered on what quiz.
Once you make sure that the syllabus is correct, you need to make sure that the right questions are on the quiz. If there are questions on the quiz that should not be covered, it's a big mess for the class.
What you need to do:
Brightspace
Moodle/CULearn
Look at the questions on each quiz (on CULearn) and make sure there are no questions that should not be there.
Enter CuLearn.
Click on the Quiz in the main column
Click Edit Quiz on the left column
To add a question, click Add at the bottom right
select "a random question"
select category
add one random question at a time.
repeat until all questions are added.
To eliminate a question,
click the x next to a question
There should be 30 questions per quiz in CGSC1001.
When the quiz has all the questions you want it to have, and none of the questions you don't, repaginate with the button at the top of the questions list.
How to Allow a Student to Re-Take a Quiz
On the navbar, click the dropdown menu for Tools, and select Quizzes.
On the Manage Quizzes page, click the desired quiz.
Click the Restrictions tab.
Use the "find" feature to find the student's name on the page. If they are not there, add them by clicking "Add Users to Special Access"
If they are there, click the pencil after their name.
Change the number of attempts for that student, and don't forget to keep hitting "save" until it doesn't prompt you to anymore.
How to fix a quiz question
Sometimes a question will be mis-keyed (wrong answer marked as right) or the answers are confusing, etc. Do the following to fix a quiz question:
Brightspace
Go to course admin
Click on quizzes
Choose which quiz you wish to edit using the drop down arrow next to the quiz name
Click grade
Instead of grading by users, click on the "questions" tab
select "update all attempts"
choose the question you wish to edit
From this screen you can assign credit for various responses
Moodle/CULearn
Go to the cuLearn page for the course.
In the administration section on the left, click the arrow next to "Question Bank" in order to expand the list.
Click "Categories" to see the list of categories for the quiz (usually faster than searching through all the questions).
Click the number (in brackets) next to the category that the question is a part of.
Click the cog icon next to the question to edit it.
Make the required edits then "Save Changes."
Regrade the quizzes that included that question category (see next section).
How to remove questions and re-weight a quiz
If everything goes right, you or Dr. Davies will have removed all quiz questions that should not be there. But if some slip by, we have to fix things by making those questions not count, and regrade the quizzes. Students hate this. Talk to Dr. Davies before doing the following:
If you put questions on the quiz that should not be there
Brightspace
Course
Tools / Quizzes
Hit down arrow next to quiz name
Select Grade
Go to Questions tab under grey buttons
Click “update all attempts”
Scroll to the correct pool
Click first question
Go to “Give to all attempts”
And put 1 before points
Go back to questions
Then do it for all in the pool.
The marks will automatically update, and these changes will not affect future or past quizzes.
How to regrade a quiz
Brightspace
Moodle/CULearn
Go to the cuLearn page for the course.
Click on the quiz you wish to regrade.
In the administration section on the left, click the "Results" link.
Click "Regrade all" above the marks on the page.
Wait until the regrading process is complete on the page that opens.
Item Analysis
After the third quiz, you should check to make sure that none of the questions have gone bad. To do this, you do an Item Analysis.
Brightspace
Moodle/CULearn
On CULearn, navigate to the most recent class that has finished.
Adminstration > Quizzes > Final Exam
then
Administration > Quiz administration> Results > Statistics
This report gives a statistical (psychometric) analysis of the quiz, and the questions within it.
The top section of this report gives a summary of the whole quiz.
The next section gives an analysis showing all questions in a table format. There are links in this section to edit individual questions or drill down into a detailed analysis of a particular question. The last section of this report is a bar graph of the percent of correct answers (Facility index) and the Discriminative efficiency index.
The number I find most telling is the Discrimination index. Basically, did the students that did well on the quiz as a whole answer this question correctly and did the ones who did poorly get this question wrong.
Very low percentages (say below 10-15%) might mean the question was mis-keyed.
This is not good and you may want to check to see if the correct answer was set to grade properly or consider rewriting or dropping this question.
When looking at the "Discrimination index"
- This index looks the the lowest 27% and the highest 27% of the scores on the whole quiz and this question. Be careful with small class sizes, if the sample size is too small it is hard to pull out much statistical significance.
- in general, we tend to be able to spot issues when we see a low "Discrimination index"
coupled with a low "Facility index".
When looking at the "Facility index"
- we generally want it to fall between 40% and 80%.
- below 40% is difficult of a question and above 80% is too easy
If there are bad questions, see if they turned up online somewhere. With the bad questions, and questions that are redundant, arrange a meeting (in person or on the phone) to talk to Dr. Davies about them so we can replace or remove them.
General information on terms used in report
-
Q# - shows the question number (position), question type icon, and preview and edit icons
-
Question name - the name is also a link to the detailed analysis of this question (See Quiz Question Statistics below).
-
Attempts - how many students attempted this question.
-
Facility Index - the percentage of students that answered the question correctly.
-
Standard Deviation - how much variation there was in the scores for this question.
-
Random guess score - the score the student would get by guessing randomly
-
Intended/Effective weight - Intended weight is simply what you set up when editing the quiz. If question 1 is worth 3 marks out of a total of 10 for the quiz, the intended weight is 30%. The effective weight is an attempt to estimate, from the results, how much of the actual variation was due to this question. So, ideally the effective weights should be close to the intended weights.
-
Discrimination index - this is the correlation between the score for this question and the score for the whole quiz. That is, for a good question, you hope that the students who score highly on this question are the same students who score highly on the whole quiz. Higher numbers are better.
-
Discriminative efficiency - another measure that is similar to Discrimination index
Here is a general doc on developing MC questions
http://carleton.ca/edc/wp-content/uploads/Developing-Multiple-Choice-Tests-.pdf
The EDC hosts a one-day Multiple-Choice Retreat a few times a year. Let Dr. Davies know if you're interested in attending it; though you cannot use your TA hours to do this.
Comments (0)
You don't have permission to comment on this page.