Competence Assessment in Education - Research, Models and Instruments

von: Detlev Leutner, Jens Fleischer, Juliane Grünkorn, Eckhard Klieme

Springer-Verlag, 2017

ISBN: 9783319500300

Sprache: Englisch

491 Seiten, Download: 11444 KB

Format: PDF, auch als Online-Lesen

geeignet für:

eBook anfordern

▸

Mehr zum Inhalt

Competence Assessment in Education - Research, Models and Instruments

	Contents	6
	Chapter 1: Competence Assessment in Education: An Introduction	10
	1.1 The German DFG-Priority Program “Competence Models for Assessing Individual Learning Outcomes and Evaluating Educational Processes”	10
	1.2 Research Areas of the DFG-Priority Program	11
	1.2.1 Cognitive Modeling and Assessment of Competencies	12
	1.2.2 Innovations in Psychometric Models and Computer-Based Assessment	13
	1.2.3 Reception and Usage of Assessment Results	13
	1.3 Conclusion	14
	References	14
	Part I: Modeling and Assessing Student Competencies	16
	Chapter 2: Science-P I: Modeling Conceptual Understanding in Primary School	17
	2.1 The Assessment of Science Competency in Primary School	18
	2.2 Modeling Conceptual Understanding in Primary School Science	18
	2.2.1 Model Specification and Item Construction	18
	2.2.2 Conceptual Understanding: Dimensions and Levels	19
	2.2.3 Validity	21
	2.3 The Development of Conceptual Understanding in Primary School Science	22
	2.4 Conceptual Understanding and Scientific Reasoning	23
	References	24
	Chapter 3: Science-P II: Modeling Scientific Reasoning in Primary School	26
	3.1 Science-P	26
	3.2 Development of Our Inventory	27
	3.3 Convergent Validity of Paper-and-Pencil Inventory and Interviews	29
	3.3.1 Method	30
	3.3.1.1 Participants	30
	3.3.1.2 Material	30
	3.3.1.3 Procedure	31
	3.3.2 Results	31
	3.3.2.1 Pre-analyses	31
	3.3.2.2 Convergent Validity	31
	3.3.3 Discussion	32
	3.4 Scientific Reasoning: Development from Grades 2 to 4	33
	3.5 Competence-Structure Model of Scientific Reasoning: Hierarchical Levels of Competence	34
	3.6 Outlook	35
	References	35
	Chapter 4: The Heidelberg Inventory of Geographic System Competency Model	37
	4.1 The Role of Geographic System Competency in Geography Education	38
	4.2 Study Overview	39
	4.3 CogLabs	41
	4.3.1 Description of the Measurement Instruments	41
	4.3.2 Selected Results	43
	4.4 First Quantitative Study (Q1)	45
	4.4.1 Description of the Measurement Instruments	45
	4.4.2 Dimensions of the Competency Model	45
	4.4.3 Levels of the Competency Model	47
	4.5 Second Quantitative Study (Q2)	49
	4.5.1 Description of the Measurement Instruments	49
	4.5.2 Dimensions of the Competency Model	49
	4.5.3 Levels of the Competency Model	53
	4.6 Discussion	53
	4.6.1 Dimensions of GSC	53
	4.6.2 GSC Levels	55
	4.7 Conclusions	56
	References	57
	Chapter 5: An Extended Model of Literary Literacy	60
	5.1 The Comprehension of Literary and Expository Texts	61
	5.2 Current Research on Literary Literacy and Further Directions	61
	5.2.1 The Internal Structure of Literary Literacy	62
	5.2.2 The Need for an Extended Model of Literary Literacy	63
	5.3 Research Objectives	65
	5.4 Method	65
	5.4.1 Sample	65
	5.4.2 Measures	65
	5.4.2.1 Semantic Literary Literacy	66
	5.4.2.2 Idiolectal Literary Literacy	67
	5.4.2.3 The Ability to Recognize Foregrounded Passages (Foregrounding)	67
	5.4.2.4 Specific Literary Knowledge	67
	5.4.2.5 The Ability to Recognize Emotions Intended by a Literary Text	68
	5.4.3 Statistical Analyses	68
	5.5 Results	70
	5.6 Discussion	72
	5.6.1 The Structure of Literary Literacy	74
	5.6.2 Some Considerations on Teaching Literary Literacy	74
	5.6.3 Limitations of the Study	76
	References	76
	Chapter 6: Self-Regulated Learning with Expository Texts as a Competence: Competence Structure and Competence Training	80
	6.1 Theoretical Background	80
	6.1.1 Integrated Model of Self-Regulated Learning	81
	6.2 Research Questions and Hypotheses	84
	6.3 Studies 1a and 1b: A “Component Perspective” on the Structure of Self-Regulated Learning Competence	85
	6.4 Study 2: A “Process Perspective” on the Structure of Self-Regulated Learning Competence	89
	6.5 Discussion	91
	References	93
	Part II: Modeling and Assessing Teacher Competencies	95
	Chapter 7: Investigating Pre-service Teachers’ Professional Vision Within University-Based Teacher Education	96
	7.1 Introduction	96
	7.2 Modeling the Structure of Professional Vision	97
	7.2.1 Noticing: Selective Attention to Important Classroom Events	98
	7.2.2 Reasoning: Interpretation of Important Classroom Events	99
	7.3 Testing the Structure of Professional Vision	100
	7.3.1 The Assessment Tool Observer	101
	7.3.2 Interrelation Between the Three Reasoning Dimensions	103
	7.4 Investigating Changes in Professional Vision Within University-Based Teacher Education	103
	7.4.1 The Role of Formal and Informal OTL	104
	7.4.2 The Design of Formal and Informal OTL	105
	7.5 Building the Bridge from Professional Vision to Teaching Action	106
	7.5.1 M-Teach Events as Assessment of Teaching Action	107
	7.5.2 Pre-service Teachers’ Teaching Skills in M-Teach Events	108
	7.6 Conclusion and Outlook	109
	References	109
	Chapter 8: Teacher Knowledge Experiment: Conditions of the Development of Pedagogical Content Knowledge	113
	8.1 Introduction	114
	8.1.1 The Construct of Pedagogical Content Knowledge	114
	8.1.2 Conditions for the Development of Pedagogical Content Knowledge: The Role of Prior Content Knowledge and Pedagogical Knowledge	115
	8.1.3 The Present Study	117
	8.2 Methods	118
	8.2.1 Participants	119
	8.2.2 Treatments	119
	8.2.3 Measures	122
	8.2.4 Baseline Equivalence and Treatment Implementation Checks	123
	8.3 Results	124
	8.3.1 Measurement of Pre-service Teachers’ Knowledge	124
	8.3.2 Testing the Assumptions on PCK Development	127
	8.4 Discussion	127
	References	129
	Chapter 9: Teachers’ School Tracking Decisions	132
	9.1 Introduction	133
	9.2 Dual Process Models of Decision Making	134
	9.3 The Adaptive Diagnostic Competency Model (ADCM)	135
	9.4 Testing the ADCM	136
	9.4.1 Student Case Vignettes (Pre-studies 1 and 2)	137
	9.4.2 The ADCM: Case Consistency and Expertise (Study 1)	139
	9.4.3 The ADCM: Case Consistency and Accountability (Study 2)	141
	9.5 Training Study	143
	9.6 Discussion	144
	References	146
	Chapter 10: Modeling, Measuring, and Training Teachers’ Counseling and Diagnostic Competencies	149
	10.1 Introduction	150
	10.2 Project Goals	151
	10.3 Modeling Teachers’ Counseling and Diagnostic Competencies	152
	10.3.1 Theoretical Background	152
	10.3.2 Method	154
	10.3.3 Results	154
	10.3.4 Relationship Between Teachers’ Counseling and Diagnostic Competence	155
	10.4 Measuring Teachers’ Counseling and Diagnostic Competence	156
	10.4.1 Scenario Tests	156
	10.4.2 Situational Judgment Test	157
	10.4.3 Knowledge Tests and Self-Assessment Questionnaires	157
	10.5 Training Teachers’ Counseling and Diagnostic Competence	158
	10.5.1 Training Program in Diagnostic Competence for In-Service Teachers	159
	10.5.2 Training Programs in Counseling and Diagnostic Competence for Prospective Teachers	160
	10.6 Conclusions and Outlook	162
	References	163
	Chapter 11: Development and Evaluation of a Competence Model for Teaching Integrative Processing of Texts and Pictures (BiTe)	167
	11.1 The “BiTe-Project”	168
	11.2 Theoretical Background	168
	11.2.1 Challenges of Picture-Text-Integration (PTI)	168
	11.2.2 Teachers’ Competencies for Teaching the Integrative Processing of Pictures and Texts	168
	11.2.3 Quality of Instruction	169
	11.3 Research Questions and Hypotheses	169
	11.4 Methods	170
	11.4.1 Sample and Study Design	170
	11.4.2 Measures	171
	11.5 Results	175
	11.5.1 Research Question (1): Knowledge About PTI	175
	11.5.2 Research Question (2): Teachers’ Attitudes, Motivation, and Self-Related Cognitions Towards PTI and Diagnostics in PTI	175
	11.5.3 Research Question (3a): Teachers’ Accuracy of Judgment	176
	11.5.4 Research Question (3b): Teachers’ Accuracy of Judgment, Knowledge, and Duration of Contact	177
	11.5.5 Research Question (4a): Relations Between Teachers’ Competencies and Instruction	177
	11.5.6 Research Question (4b): Relations Between Teachers’ Competencies and Students’ Competence and Engagement	178
	11.6 Discussion	178
	References	179
	Part III: Modeling and Assessing Vocational Competencies and Adult Learning	181
	Chapter 12: Multidimensional Competency Assessments and Structures in VET	182
	12.1 Introduction	183
	12.2 Theoretical Issues and Operationalization	183
	12.3 Research Questions	186
	12.4 Research Design	187
	12.5 Hypotheses and Results	188
	12.5.1 Research Question 1: Competency Structures Within the Construct of Not Directly Action-Centered Occupation-Specific Knowledge: Dimensionality and Its Development	188
	12.5.2 Research Question 1: Competency Structures Within the Construct of Action-Centered Occupation-Specific Knowledge: Dimensionality and Its Development	193
	12.5.3 Preliminary Analysis for Research Question 2: Construct of Occupation-Specific Problem Solving: Validity and Reliability	195
	12.5.4 Research Question 2: Competency Structures Between Different Constructs	197
	12.6 Summary and Final Discussion	198
	References	200
	Chapter 13: Professional Competencies of Building Trade Apprentices After Their First Year of Training	202
	13.1 Introduction	202
	13.2 Aims and Objectives	204
	13.3 Vocational Training in the Building Trades	204
	13.4 Professional Competence of Building Trade Apprentices	205
	13.5 Research Design and Data Collection	206
	13.6 Results	207
	13.6.1 Sociodemographic Factors, Cognitive Abilities and Apprentices’ Performance at the Beginning of the First Training Year	208
	13.6.2 Professional Competence at the End of the First Training Year	210
	13.7 Additional Findings and Prospects	215
	References	216
	Chapter 14: Assessing Tomorrow’s Potential: A Competence Measuring Approach in Vocational Education and Training	220
	14.1 Background	221
	14.1.1 Prospects and Demand for Adequate Competence Assessments in Vocational Education	221
	14.1.2 The Original Conceptualization of Final Examinations in the Area of Business and Commerce	222
	14.1.3 Assessment Model for Commercial Vocations	225
	14.2 Method	228
	14.2.1 Sample	228
	14.2.2 Examination of Validity	228
	14.2.3 Examination of Reliability	229
	14.3 Results	230
	14.3.1 Results for the Test’s Validity	230
	14.3.2 Results for the Test’s Reliability	231
	14.4 Discussion	232
	14.5 Conclusions	234
	Appendix	235
	Ceraforma Keramik AG	235
	Business Process 1	236
	References	240
	Part IV: Competency Development: Modeling of Change and Training of Competencies	243
	Chapter 15: The Development of Students’ Physics Competence in Middle School	244
	15.1 Introduction	245
	15.2 Theoretical Background	245
	15.2.1 Students’ Understanding of Energy	245
	15.2.2 Students’ Learning About Energy	246
	15.3 Research Questions	247
	15.4 Project Design	249
	15.5 Phase 1: The Cross-Sectional Study	250
	15.5.1 Method	250
	15.5.2 Results	251
	15.6 Phase 2: Longitudinal Study	253
	15.6.1 Method	253
	15.6.2 Results	254
	15.7 Summary and Outlook	256
	References	257
	Chapter 16: Modeling and Fostering Decision-Making Competencies Regarding Challenging Issues of Sustainable Development	260
	16.1 Introduction	261
	16.2 A Competence Model for Decision Making with Respect to Sustainable Development	261
	16.3 Measurement Instruments and Competence Modeling	263
	16.3.1 “Developing Solutions”: Development of the Measurement Instrument	263
	16.3.1.1 Sample	263
	16.3.1.2 Measures: Tasks and Items	263
	16.3.1.3 Instrument Functioning	264
	16.3.2 Modeling of “Developing Solutions”	266
	16.3.3 Discussion	268
	16.4 Experimental Validation: A Comparison of Socioscientific Decision Making with Analytical Problem Solving	268
	16.4.1 Objectives and Research Design	269
	16.4.2 Methods	269
	16.4.2.1 Participating Students and Teachers	270
	16.4.2.2 Trainings and Learning Material	270
	16.4.2.3 Measures	272
	16.4.3 Results of the Pilot Study	274
	16.4.4 Discussion	275
	16.5 Conclusions and Outlook	277
	References	278
	Chapter 17: Metacognitive Knowledge in Secondary School Students: Assessment, Structure, and Developmental Change	282
	17.1 Theoretical Background	283
	17.1.1 Methodological Issues Regarding the Assessment of Declarative Metacognitive Knowledge	285
	17.1.2 Design of the EWIKO Study	286
	17.1.3 Test Instruments	287
	17.2 Overview of Major Results	290
	17.2.1 Development of Metacognitive Knowledge: Sources of Interindividual Differences	290
	17.2.2 Domain-Specificity—A Transitional Period of Metacognitive Development?	292
	17.2.3 Interrelations Between Metacognitive Knowledge and Achievement	294
	17.3 Discussion	295
	References	296
	Chapter 18: Development of Dynamic Usage of Strategies for Integrating Text and Picture Information in Secondary Schools	300
	18.1 Texts Combined with Instructional Pictures	301
	18.2 Theoretical Background	301
	18.2.1 Taxonomies of Text-Picture-Integration	301
	18.2.2 Strategies for Integrative Processing of Text and Pictures	302
	18.3 Research Questions and Hypotheses	302
	18.4 Method	303
	18.5 Results	305
	18.5.1 Reading and Observation Times	305
	18.5.2 Transitions Between Texts, Pictures, and Items	306
	18.6 Discussion	306
	References	309
	Chapter 19: Training in Components of Problem-Solving Competence: An Experimental Study of Aspects of the Cognitive Potential Exploitation Hypothesis	311
	19.1 Introduction	312
	19.2 Theoretical Framework	312
	19.2.1 Problem Solving in PISA 2003: The Cognitive Potential Exploitation Hypothesis	313
	19.2.2 Components of Problem-Solving Competence	315
	19.3 Research Questions	316
	19.4 Study I	316
	19.4.1 Methods	316
	19.4.2 Results	317
	19.4.3 Discussion	318
	19.5 Study II	318
	19.5.1 Methods	318
	19.5.1.1 Data Analysis	320
	19.5.2 Results	320
	19.5.2.1 Planning	320
	19.5.2.2 Problem Solving	320
	19.5.2.3 Mathematics	321
	19.5.3 Discussion	322
	19.6 General Discussion	323
	19.6.1 Limitations and Future Research	323
	References	324
	Chapter 20: An Intensive Longitudinal Study of the Development of Student Achievement over Two Years (LUISE)	328
	20.1 Introduction	329
	20.2 Student Achievement in Languages	330
	20.3 Student Achievement in Mathematics	331
	20.4 Changes in Student Achievement in Mathematics and Native Language	331
	20.5 The Present Study	333
	20.5.1 Method	334
	20.5.1.1 Procedure and Participants	334
	20.5.1.2 Measures	335
	20.5.1.2.1 German Achievement Tests	335
	20.5.1.2.2 Mathematics Achievement Tests	337
	20.5.2 Results	337
	20.5.2.1 German Achievement	337
	20.5.2.2 Mathematics Achievement	338
	20.6 Summary and Discussion	344
	References	346
	Part V: Innovations in Psychometric Models and Computer-Based Assessment	350
	Chapter 21: Multidimensional Structures of Competencies: Focusing on Text Comprehension in English as a Foreign Language	351
	21.1 Introduction	352
	21.2 Test Development	352
	21.2.1 Item Characteristics	352
	21.2.2 Item Development	355
	21.2.3 Validation of Item Characteristics	356
	21.3 Test and Item Analysis	356
	21.3.1 Sample and Data Collection	356
	21.3.2 Unidimensional Test and Item Analysis	356
	21.3.3 Item Difficulties Across Domains	357
	21.3.4 Local Dependencies	357
	21.3.5 Multidimensional Analysis	358
	21.4 Discussion	359
	21.4.1 Research Perspectives	359
	21.4.2 Implications for Educational Contexts	360
	References	361
	Chapter 22: Multidimensional Adaptive Measurement of Competencies	363
	22.1 Problem	364
	22.2 Consideration of Multiple Constraints in MAT	366
	22.2.1 Multidimensional Maximum Priority Index	367
	22.2.2 Research Objective	367
	22.2.3 Method	368
	22.2.4 Procedure	368
	22.2.5 Results	369
	22.2.6 Discussion	369
	22.3 Using Prior Information for Item Selection and Ability Estimation	370
	22.3.1 Research Questions	370
	22.3.2 Method	370
	22.3.3 Results	371
	22.3.4 Discussion	373
	22.4 The Multidimensional Adaptive Testing Environment (MATE)	373
	22.4.1 Computerization of Items	373
	22.4.2 Assignment of Item Parameters	374
	22.4.3 Configuration of Tests and Test Batteries	374
	22.4.4 Pre-operational Simulation Studies	375
	22.4.5 Graphical User Interface, System Requirements, Availability and Manual	376
	22.5 Empirical Application	376
	22.6 Conclusion	378
	References	380
	Chapter 23: Development, Validation, and Application of a Competence Model for Mathematical Problem Solving by Using and Translating Representations of Functions	382
	23.1 Introduction	383
	23.2 Construction of a Theoretical Framework Model	384
	23.3 Development and Empirical Validation of Psychometric Models	386
	23.3.1 Basic Model: Representations and Translations Between and Within Situational, Numerical, and Graphical Representation	386
	23.3.2 Extension: Inclusion of Algebraic Representation/Cognitive Action	388
	23.3.3 Extension: Hierarchical Models	391
	23.4 Development and Evaluation of a Computerized Adaptive Testing Procedure	392
	23.4.1 Aims for the Development of the Adaptive Test	392
	23.4.2 Item Pool and Calibration	393
	23.4.3 Pre-operational Simulation Study	394
	23.5 Discussion	396
	References	397
	Chapter 24: Relating Product Data to Process Data from Computer-Based Competency Assessment	400
	24.1 Introduction	401
	24.2 Study 1: The Effect of Time on Task Success Differs Across Persons and Tasks	402
	24.2.1 Research Goal and Hypotheses	403
	24.2.2 Methods	403
	24.2.3 Results	404
	24.2.4 Discussion	404
	24.3 Study 2: Benefits for Task Completion from the Automatization of Subtasks	405
	24.3.1 Research Goal and Hypotheses	406
	24.3.2 Methods	406
	24.3.3 Results	407
	24.3.4 Discussion	408
	24.4 Study 3: Number of Interactions: More Is Not Always Better	408
	24.4.1 Research Goal and Hypotheses	409
	24.4.2 Methods	410
	24.4.3 Results	410
	24.4.4 Discussion	411
	24.5 Study 4: Problem Solver Types: Different Ways to Success in Information Problems	412
	24.5.1 Methods	412
	24.5.2 Results	412
	24.5.3 Discussion	413
	24.6 How to Handle Unstructured Process Data?: The Log File Data Extraction Tool	413
	24.7 Conclusions and Final Remarks	415
	References	416
	Chapter 25: Dynamic Problem Solving: Multiple-Item Testing Based on Minimally Complex Systems	419
	25.1 Introduction	420
	25.2 Modeling of Problem Solving Competencies	420
	25.3 Development of Computer-Based Assessment Instruments	422
	25.4 Empirical Tests of the Newly Developed Instruments	427
	25.5 Educational Application: PISA 2012	429
	25.5.1 Two Additional Issues: Optimization and Causal Diagrams	430
	25.6 Future Developments	431
	References	432
	Part VI: Feedback From Competency Assessment: Concepts, Conditions and Consequences	436
	Chapter 26: Formative Assessment in Mathematics Instruction: Theoretical Considerations and Empirical Results of the Co2CA Project	437
	26.1 Formative Assessment: A Promising Approach to Improving Teaching and Learning?	438
	26.1.1 Formative Assessment: State of the Art	438
	26.1.2 Components of Formative Assessment	439
	26.1.3 How Formative Assessment Affects Learning	439
	26.2 The Four Studies of the Co2CA Project	440
	26.2.1 Survey Study	440
	26.2.2 Experimental Study	441
	26.2.3 Intervention Study	442
	26.2.3.1 Aims and Research Questions of the Intervention Study	442
	26.2.3.2 Design of the Intervention Study	443
	26.2.3.3 Selected Results of the Intervention Study	449
	26.2.4 Transfer Study	449
	26.2.4.1 Aims and Research Questions of the Transfer Study	449
	26.2.4.2 Design of the Transfer Study	450
	26.2.4.3 First Results of the Transfer Study	452
	26.3 Summary	453
	References	455
	Chapter 27: Arguing Validity in Educational Assessment	458
	27.1 Introduction	459
	27.2 The Validity Concept	460
	27.2.1 The Place of Standard Setting in Educational Assessment	462
	27.3 The Argument Approach to Evaluating Validity	463
	27.3.1 A Structured Validity Argument	464
	27.3.1.1 Warrant of Well-Defined Content Standards	466
	27.3.1.2 Warrant of Test Alignment to the Content Domain	466
	27.3.1.3 Warrant of Well-Defined Performance Level Descriptors	467
	27.3.1.4 Warrant of Reliable Test-Score Measurement	468
	27.3.1.5 Warrant of Defensible Cut-Score Placements	468
	27.4 Discussion	469
	References	471
	Chapter 28: Evaluating Prerequisites for the Development of a Dynamic Test of Reading Competence: Feedback Effects on Reading Comprehension in Children	475
	28.1 Introduction	476
	28.2 The Idea of Dynamic Assessments	476
	28.3 Dynamic Assessments of Reading Competence: Existing Approaches and Challenges	478
	28.4 Experiments on the Effectiveness of Feedback on Reading Comprehension in a Train-Within-Test Setting	481
	28.4.1 Experiment 1	481
	28.4.2 Experiment 2	483
	28.4.3 Experiment 3	485
	28.5 Effects of Feedback on Reading Comprehension Within a Computer-Delivered Test: Lessons Learned	486
	28.6 Prospects of Dynamic Tests of Reading Competence	487
	References	488

Service

Info/Kontakt

Kontakt