Week 3 Blog

Week 3
Evaluating Programs & Human Performance
ETEC 561
Wilgus Burton
June 2013

All too often instruction is developed with little thought as to how evaluation of learning or the effectiveness of the instruction will take place. When evaluation is considered on the front end of the instructional design process, it is often limited to evaluating whether the instructional design is more effective than traditional methods.

For this week's reflection activities, I would like for you to concentrate on the following:

    Chapters in Section III discuss evaluation in instructional design and provide you with three evaluation models: CIPP, Five-Domain, and Kirkpatrick’s Training Evaluation model. Search for at least two other models used for evaluation and summarize these models. Describe how you would use them to evaluate your instruction.

Response:

I chose 2 evaluation methods based upon Robert Claser and Meg Ormiston.
Robert Glaser (January 18, 1921 – February 4, 2012) was an American educational psychologist. His evaluation concepts were based upon 2 factors, or two types of evaluations:
 
1. Criterion-Referenced Assessment   
2. Norm-Referenced Assessment

Criterion- Referenced Assessment is a simple right/wrong type of Assesssment.  It is based upon the domain of subject matter that the test is designed to assess.  In other words, a class of students takes a test on, let's say, formation of the Earth.  There are a certain steps or stages that are  hypothesised  that are needed for a proto-star  needs to go through to make a star and its planetary system.  If a class is asked to list the stages, in order, everyone's answers are either right or wrong.  These are based upon either on the textbook or teacher's given information. Just a simple yes or no answer.  This is an example of what Bloom's ideas of the lowest type of learning:  knowledge or remembering.

Norm-Referenced Assessment is a slight variation upon the Criterion-Referenced Assessment.  The N-R Assessment gives the same type of test, but now, the scores are based upon not being right or wrong, they are being assessed on how they compared to, let's say, the rest of the class.  Student A did better than student B, but not as well as student C. This test just determines if a student correctly answered more questions compared to other students in the class. This is also termed as "grading on the curve".

Meg Ormiston is a proponent of Authentic Assessent -  the measurement of intellectual accomplishments that are worthwhile, significant, and meaningful, as compared to multiple choice standardized tests.
This type of assessment tends to focus on  what's termed "contextualised tasks" where students demonstrate their competency in a particular authentic setting.  This assessment focuses on three types of "settings" for testing:

1. performance of the skills, or demonstrating use of a particular knowledge
2. simulations and role plays
3. studio portfolios, strategically selecting items.

Ormiston says "authentic learning mirrors the tasks and problem that are required in the reality of a location outside of the school setting", Ormiston, Meg (2011). [Creating a Digital-Rich Classroom: Teaching & Learning in a Web 2.0 World. Solution Tree Press. pp. 2–3.]

At our high school, we currently have two skills-type of courses: welding and body-shop.  During the course of a couple years of taking welding, for example, a student can progress enough that he/she may be willing to take a test that determines skills needed to become a welder as an occupation.  They may take their licensing test (usually in 11th or 12th grade), and if passed, the student has become a certified welder.
As a teacher that teaches elective-type classes, my evaluation processes tend to fall under the Criterion-Referenced Assessment.

    Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. Should return on investment and management of resources also be considered in evaluations of instructional programs? What other measures should/could be considered?

Response:

The main goal of an ID (Instructional Design) model or process is to construct a learning environment in order to provide the learners with the conditions that support the desired learning processes. An overly simplified way of putting this is (for some students) "you teach, I'll learn".  Actually, it is not as simple as that. What are you going to teach? Are there broad topics that need to be covered? Are the topics based across the curriculum of the school or business?  How are the assessments going to be accomplished? The ADDIE concept is one way to design and implement methods of learning and evaluation.

Two additional methods are generally taken into account:  Analysis and design.

An advancement to the general concept of Instructional Design is a much more broader reference known as Instructional System Design (ISD). there are 5 measures that are used:
1. analysis
2. design
3. development or production
4. Implementation or delivery
5. evaluations

A concept graphic that can show ISD is:

Here, both formative and summative  evaluations are performed to measure the success of the ISD process. Return on Investment is a useful concept that can determine if a business profits or fails.  In a business,  If during the process of testing the overall system or products, any determination that a part isn't working may make or brake the business.  Would customers be happy or will they go elsewhere for business?  Each step of these five parts if the ISD is crucial to survival of a business.

Education, on the other hand, is slightly different - but not by much.  Here, graduation may be considered the school district's return on investment.  In any school, the district/state makes a determination of what it costs to educate a student.  Based upon many factors, such as socioeconomic conditions, the educational district needs X amount of dollars to teach a student. And, a student's challenges may help determine his/her ability to learn the general curriculum (or be main-streamed). Or, is there another system for evaluating the student's performance based upon other challenges (physical or mental)?

Based upon a state-wide system (TEA) for education, there are guidelines set up for each school to follow for the advancement of students. These guidelines may have adjustments that can be made for the path that a student may want to follow (college readiness, service oriented, etc). A curriculum is designed as a standard for the success of each student. And if successful, by what ever means of design and assessment, the student will graduate.

If in a business, all all areas of that business are assessed, its success is measured by its return on investment.  It can either thrive or go out of business.


    Section IV focuses on human performance, performance support systems, knowledge management systems, and the concept of informal learning. Not all problems in learning and/or performance require an instructional one. Many times a non-instructional approach is a more appropriate solution. Identify a performance problem in your area of work and identify non-instructional solutions that may help solve the problem. Would better performance support systems, knowledge management systems, or opportunities for informal learning solve the problem?

Response:

Student's hate tests.  Well, some do.  I hear that alot.  Also, "when am I ever going to use this"? These are some of the problems I have with students that are placed in my classes when there's no other place to put them.  They have all their requirements or are finishing them in addition to  my classes. The students are Seniors.  They are ready to get out of high school.

Some of the students are going on to college, military, or into a trade. What ever the reason for being in my class - even those that really want to be there, I have a set of curriculum topics to cover.  I tell all my students that "our job is to not only give you information and evaluations that you need to master so that you can move on. We have to prepare you for a part in the world stage which means that we want to give you a well-rounded education, not for that to which you want to move into, but so that you can become an active part of society.  In doing that, you won't look like a clown to others around you if you don't know about topics in the news and certain things related to those topics".

There are other ways to educate students.  One example I mentioned above is that we have a Welding Certification Program at our high school.  That is hands-on and directly  related to a trade.  Throughout history, most people learn their trade through apprenticeships (like Ms Ormiston's Authentic Assessment concepts listed above). An apprenticeship takes a lot of time - many years at times. This is Authentic Learning.



Informal Learning is another way to educate students that are reluctant to, or don't care to, learn material for which they are placed in your care. One example to which this can be a positive step for many students is through field trips.  A place outside of the general learning environment that is related to your subject.

For Astronomy and Earth Sciences, if finances weren't an issue, field trips to a NASA facility or to an observatory, or to an all-night observing session or to an anthropological/archaeological dig, or even to a museum can get many reluctant students into learning their given subject. They may not know they are learning, but the majority of the time, they do learning something. Without the finances, my students are religated to taking on-campus trips such as night observing sessions in the parking lots, or local stream bed digging and measuring, weather and cloud observations.  These are fantastic ideas, but they cannot encompass everything on a particular subject.

 Both performance support systems, and knowledge management systems can play a role into some student's advancement.  Performance support systemsf can be something like a behavior modification program where students are placed into a temporary learning area not within the classroom.  These students' placements may be from too much talking, causing a disturbance in the classroom, or even worse where the student is placed or taken off campus.  A simple knowledge management system my be a private tutor or a educational facilitator that stays with the student to make sure they are learning, concentrating on the subject, behaving or falling asleep.

There are many ways for students to learn.  And, as a result, there may be many different means by which the learning presentation and processes may be evaluated.

Graphics from website:   http://www.nwlink.com/~donclark/index.html

Week 2: Theories & Models of Learning & Instruction

ETEC 561

Wilgus Burton

June 2013

Epistemology (the study of what and how we come to know) is discussed in multiple chapters in this section. Distinguish epistemology from instructional methods or theories. What are the differences between theories, methods, or models of learning and epistemologies or underlying beliefs about ways of knowing?

RESPONSE:

In the text, Reiser/Dempsey, page 36, “learning can be defined as a “persisting change in human performance or performance potential (Driscoll, 2005)”  and further defined as “how it is thought it is thought to occur (Driscoll, 2005)”.

It seems that I can to out on a limb and say that the difference between epistemology and instructional methods or theories is or can be analogous to a file cabinet and the files in that cabinet.  The epistemology aspect would be the file cabinet – an “all-encompassing” concept with the instructional methods or theories would be the files.

The thinking that instructional methods or theories as being etched in stone is not quite realistic.  Psychology has gone through many changes over the years; B.F. Skinner's behavior theory and on to Cognitive Information Processing Theory,  Schema, Cognitive Load, Situational Learning Theory, Gagne's Theories of Instruction to Constructivism. Those "theories" are methods (models) of the overall blanket of Epistemology of Learning.   As technology advances, and newer ways to collect data, these theories may change over time.

2. Chapters in this section discuss three contrasting epistemic stances: positivist, relativist, and contextualist (or hermeneutical). Positivists believe that the only truth or knowledge is objective truth. Relativists don’t believe that objective truth is possible and that all knowledge is subjective to perception or relative to a particular frame of reference. Contextualists believe that truth or knowledge is relative to context rather than individual, subjective understanding. While designers and educators with a positivist stance generally apply behaviorist principles to the design and development of instruction, those with either a contextualist or relativist epistemological framework employ constructivist theories and methods. Reflect on whether your stance is primarily positivist, relativist, or contextualist. Then, identify an instance when your perspective or stance as a learner conflicted with that of your instructor. Describe the conflict that you experienced and analyze whether opposing epistemic stances may have been at the heart of the conflict.

Response:

At this point in time, I think of myself as a Contextualist.  As a Teacher in a non-core (elective) science, I think that learning within the classroom is the content  that relates to the subject matter. The four parts of a Constructive theory is that:

"1) we should engage learners in activities related to the subject matter being taught
2) Provide for the collaboration and the opportunity to engage multiple perspectives
3) Support learners in setting their own goals to regulate their own learning and
4) Encourage learners to reflect on what and how they are learning" (Reiser and Dempsey, 2012).

The historical aspect of the Constructive theory is based on the findings/theories of John Dewey and Jean Piaget where inquiry and assimilation are the key factors. Currently there is a movement of combining the aspects of both  positivist and contextualist views.  As we move towards a more technological classroom environment, the learning environment (teachers and  students) are beginning to take on the epistemology of the Constructivism.

Looking back over my childhood, in elementary school, I realize that there was a major shift between positivism and constructivism.  Growing up, we had to learn, through rote memorization math tables, terms (spelling and definitions), history, and science - positivism.  Later, we had to start to put them to use.  My conflict there was one of being used to rote memorization towards constructivism.  We weren't taught how to use the memorized information.  We had to figure it out as we went along.  I still have a problem with that.

However, now, I have that built-in concept of assimilation of information, but, with a new appreciation of "learning the basics".

3. Differing epistemic stances lead to differing approaches to learning and instruction, and ultimately to problem-solving. Explain differences in problem-solving when approached from behaviorist and constructivist perspectives. How do the approaches differ in both the nature of the problem to be solved and in facilitating the problem solving process? Finally, what effect might these differences have on learner motivation? 

Response:

Problem-solving is a standard of math and science, figuratively and literally. It does go beyond those two general subjects, though.  With the viewpoint of a Science Teacher, I see both perspectives.
From a behaviorist point of view, where we learn from experiences.  We do homework, lab work, classwork, and tests and we get a grade. Whether or not  it is a from a Bloom's category of simple knowledge to a more complex area of evaluation..

Here, the better response to "learning" the content gets a positive response (good grades) and the lesser grades get a less than positive response - Sounds like Pavlov.

From a constructive perspective, it becomes a bit more complicated.
 
Students come from different types of backgrounds; good and bad (my subjectivity). Each student has their own style of learning and each has their environment that may affect there general knowledge and evaluation of the subject. Their previous skills in facilitating their synthesis and evaluation of differing types of problems that exist for them personally will and can be challenging.  Their environment and future job aspects can interfere with the type of situations that they are capable, or willing, to solve.

Whether or not they believe they will ever use "it", will go into the mix. For a high school senior, their future ambitions may determine their willingness to solve one type of problem as opposed to another type.  A student going into welding will determine how they solve a problem related to biology as opposed to the chemistry of welding gases. In my direct experience of teaching Astronomy, students with an interest in something other than Astronomy will, or not, solve the literal-types of problems as opposed to the more abstract problems.

Week 1:  Defining the Field

ETEC 561

Wilgus Burton

June 2013

 

 

1. How do the definitions in the first chapter compare to your own definition of instructional or educational technology? What experiences or other influences have shaped your definition? How has your definition changed from examining the definitions in the first chapter of this book?
Response:

This course is listed as ETEC 561 - Educational Technology.  For the past several years, I have been under the impression that Educational Technology involved designing curriculum content that can supplement concepts through the use of current and past technologies within the classroom.  Until recently, just prior to this course, I had begun noticing a slight change in that idea. 

As a Teacher within a local school district, we have Professional Development that utilizes the concept of PLCs (Professional Learning Communities).  Each Department (Science for example) has different courses whose curriculum do not necessarily overlap.  So, we have these "separate communities" like Biology and Astronomy and Physics, and Chemistry.  Within each community, there may be one or more than one teacher teaching a singular subject.  It is within these little communities where the overall subject curriculum is developed for each course such that all Teachers that teach each subject will be following the same curriculum. That makes sense. It is hoped that each Teacher teaching, say Physics, have access to the same technology that can be utilized for that course. 

But, not all Teachers are "created equal".  Some may be very comfortable using newer technology; while other's still hold their ground and stick with the text-and-chalkboard methods of teaching.  So, my first idea that Educational Technology was meant to be strictly used within the classroom. With the implementation of the PLCs, that viewpoint has changed. Dramatically!

The definition currently used by Reiser and Dempsey from the AECT (Association for Educational Communication and Technology) is "Educational Technology is the study and ethical practice of  facilitating learning and improving performance by creating, using, and managing appropriate technological processes and resources" (AECT, 2008; Trends and Issues in Instructional Design and Technology by Robert A Reiser and John V. Dempsey, 3rd Edition, 2012, page 4). Compared to previous versions as laid out in this first chapter,  this is a much better and clearer idea as to what Educational Technology should be.  According to the current ACCOUNTABILITY trend, this fits nicely.

In the district I teach in, accountability is a major factor into how we approach making new, and revising,  our curriculum, teaching, and discipline methods. It is understandable that this be taken seriously in today's judicial-happy society.  In this new definition, the first thing that struck me as needed is "the ethical practice ...".  Some school districts are conservative, some are moderate, and some are more liberal (no political statement intended here).  The social-economic, ethnic, and racial make-up of the districts work into the mix here as well.  The school districts have to be held accountable for the education of each student regardless of  the student's capabilities. The districts have to  also be constantly aware of each students potential, challenges, and goals for the educational systems. The general idea of  "the ethical practice of..." will be approached differently for each school district and the values of the local community of each school.

The remaining part of the definition fits exactly what my understanding of what Educational Technology should be.  But now, this is a two branched concept; one for the classroom use and the other for Teacher development.

The first "branch" is that of direct use within the classroom itself.  How can the Teacher make a concept more exciting instead of being the lecture-of-the-day type?  What else can be done to excite the students into learning the concepts without them being bored or falling asleep?   This can partially be addressed with some sort of non-chalkboard methods.  Using powerpoint presentations is one way (without them getting the feeling of "death by powerpoint".  A change is to get the students involved into making their own powerpoint presentations for a topic.  An example of this idea is one that I have used in my Earth & Space Sciences classes.  During the sections of Meteorology, I make up a set of cards (3X5 inch) with a different topic related to atmospheric phenomena (rainbows, sundogs, etc.) that can be seen by anyone.  The students draw 1 card out of a hat, box or something that they cannot see what they are choosing.  Each card has a different topic, so no two students will have the same topic.  Their task is to research their topic and create a short powerpoint or video presentation for the class to see.  The students have a set of concepts to include in their presentations as well as a rubric for them to follow to see just how well their presentation should be.  Unless "senioritis" has set in, this is a nice chance for the students to be the Teacher. 

Since my classes are in the Planetarium, my Astronomy classes do the same, but with different constellations to research.  Then, on the due date, each student will point out their constellation in the theater the way they would see it in the real night sky.  Both of these activities are very popular with the Seniors, and a great way for them to become familiar with uses of technology other than social  media technology.

Both of these two types of activities fit the rest of the definition very well.  Except now, the student becomes the one in charge of the use of technology use for a concept.

The other "branch"  for this definition to work is directly related to the Teachers and administrators themselves.  How can the Teachers be trained to use new technology (hardware and software) that can enhance their teaching processes?  How can additional professional personnel be taught to use the definition directly? "Using and managing appropriate technology" has to be addressed during Professional Development days and during PLC meetings.  Perhaps this branch should be called " Teachnology..." since it works with the Teacher training of new and changing technology as well as other professional personnel.  This is also where a type of change to the  name from Educational Technology / Instructional Design for Technology should be two separate types of the use of technology within the educational system.

The current (2008) definition of Educational Technology fits nicely into the educational system I know. Compared to earlier definitions, it fits well to our current educational and societal trends that we currently utilize for the benefit of our students AND faculty.



2. Next, think of a lesson or unit of instruction that you have developed. Or if you haven’t ever taught or developed instruction, think of one that you have received. How does that lesson adhere or fail to adhere to the six characteristics of instructional design? How would you redesign it to better adhere to the six characteristics.
Response:
 
As I understand the question, the 6 characteristics of Instructional Design, based on the 2008 definition, are: student centered, goal oriented, focuses on meaningful performance, outcomes can be measured in a reliable and valid way, empirical iterative and self-correcting, and typically a team effort.

As mentioned in the first question's response " A change is to get the students involved into making their own powerpoint presentations for a topic.  An example of this idea is one that I have used in my Earth & Space Sciences classes.  During the sections of Meteorology, I make up a set of cards (3X5 inch) with a different topic related to atmospheric phenomena (rainbows, sundogs, etc.) that can be seen by anyone.  The students draw 1 card out of a hat, box or something that they cannot see what they are choosing.  Each card has a different topic, so no two students will have the same topic.  Their task is to research their topic and create a short powerpoint or video presentation for the class to see.  The students have a set of concepts to include in their presentations as well as a rubric for them to follow to see just how well their presentation should be.  Unless "senioritis" has set in, this is a nice chance for the students to be the Teacher",  these two activities work the entire list of Instructional Design within the classroom., even the last (team effort).

The concept of  Students as Classroom Teachers is a form of peer teaching.  This can be summarized nicely from the website " http://www.soundout.org/teaching.html " . Peer teaching gives each student a chance to shine showing their capabilities and knowledge of a self-learned (by random choice) topic. This shifts the task from the Teacher teaching each individual concept and through self-learning, the students teach their topic. The goal is for the student to learn not just the information, but the actual presentation aids each student with skills that could be enhanced through their further education or workplace skills.  This creates a greater feeling of self-confidence while they do their individual research as well as during their presentations, the use and experience gives a meaningful action to their learning and performance.  This not only improves the student performance but  potentially helps improve the student's future classroom learning. The idea of each student creating their own presentation, presenting it, and during practicing it helps in aiding the student through the technological processes of building a presentation.  Through using the goal list and rubric, each student knows what type of material is needed to reach a measurable presentation (grades).  Granted, some students will need some assistance as they proceed with their presentation  skills and thus becomes a way for the student to self-correct their presentation for the class to see.  As far as a team effort, it can be utilized as a way for the students that are more technologically proficient  to aid those with less technological skills in completing their presentations. Again, this leaves the Teacher free to watch and guide the students toward a productive project.

The same concept can be extended towards the faculty and additional staff during in-service and PLC sessions. The use of classroom Teachers for in-service training is an additional boost to a Teacher's confidence as well as a learning potential whose information can be passed on and built upon by additional faculty and staff for their PLC groups.

3. In the 3rd chapter, Reiser distinguishes instructional media from instructional design, excluding teachers, chalkboards, and textbooks from the definition of instructional media. Why? Would you consider teachers, chalkboards, and textbooks instructional media? Is the purpose of instructional design to incorporate media into instruction?
Response:

Instructional media, in the most basic and literal sense, is the physical aspect of delivering information to students (school, businesses, etc). This includes, but not limited to, projectors, videos, audio tapes, radio, computers (including tablets, cellular devices), but excluding Teachers, chalkboards, and textbooks. Instructional Design is how technology is used for education, businesses, military, etc.

Perhaps the separation discussed in the textbook (ID&T by Reiser and Dempsey) is based upon the 1970 concepts laid out by the Commission on Instructional Technology. The exclusion of Teachers, chalkboards, and textbooks may be kept separate, in my opinion, because of historical reasons.

In the earlier times of Socrates, Aristotle, Pythagoras, Eratosthenes, Ptolomy, and Hammurabi (~ 1772 B.C.), education began as stories used to explain the unknown to children.  As time passed, and organized writing began (Egyptians, Hammurabi, etc), education had to be more than just stories passed down to the younger generations. As a large network of communication began from city to city and country to country, writing became a way for those in power to discuss business transactions, treaties, and manage local inventory of food, tools, animals, and taxes. For people to succeed in this manner, tools were developed to explain what is to be taught and understood.  This eventually went on to be a tool for developing and explain abstract concepts such as geometry, geography, mathematics, and philosophy (science).  A further advancement that became useful was the invention of tablets (rocks), paper, and eventually books.   Today, we use all of these teaching methods despite new advances in technology. 

Despite any new technology, we still need to fall back on the Teacher, chalkboards or whiteboards, and textbooks in the eventuality of some sort of technology-based failure.  It appears that the use of the Teacher/chalkboard/textbook, due to historical development and ease of use, will always be a necessity for education, businesses, research and even to teach Teachers. 

I would say that the use of Teacher, chalkboards, and textbooks (TCT) is a necessity for both Instructional Media and as a tool for Instructional Design. But, with the historical aspect of using tools to explain concrete and abstract concepts, it makes sense that the use of Instructional Media has to be a tool to be incorporated into Instructional Design.  With the changes of technology, the use of new Instructional Media, along with new methods of teaching (Bloom, Gagne, Skinner and others) has to be incorporated into the classroom through the development of curriculum, PLCs, In-service days, and personal use for education to advance to a higher level.








Ok.... Still trying to get this working.....

Ok, this is a test......

Just a quick sneak at one of my favorite astronomical objects from the Hubble Space Telescope.....