An Expanding Awareness


As best I can reconstruct the time frame, in November 2012 I came across the draft of a expandreport that caught my attention–Expanding Evidence Approaches for Learning in a Digital World.  As I read the draft, I took notes and made comments.  Those notes and comments about the draft, not the final document, are being posted below under the title of An Expanding Awareness.  The final copy of this document was published in February 2013.  gritThe final copy can be downloaded by clicking here.  February 2013 is the same month another draft publication was released that caught the attention of many people across the country– Promoting Grit, Tenacity, and Perseverance: Critical Factors for Success in the 21st Century.

An Expanding Awareness

I recently stumbled upon a draft document hardly tangentially related to what I was looking for.  The title, Expanding Evidence Approaches for Learning in a Digital World, sufficiently piqued my curiosity to the point other pressing projects were put on hold while I read the document, took notes, made comments, and generated questions.  Rather than being a formal review by someone knowledgeable in educational data mining and learning analytics this is simply a sharing of notes, comments, concerns, and questions I had as a result of reading this document.

Even though this is a draft report, I would expect it to have a date on it.  That is one of the first pieces of data I seek to mine.  It is interesting that a report about data mining does not include that data.  Not including a date seems to be more common these days.  Does someone not want you to know when the report was generated?  Why?

The five chapters in the report address the following topics:

1.  Making Sure Learning Resources Promote Deeper Learner

2.  Building Adaptive Learning Systems That Support Personalized Learning

3.  Combining Data to Create Support Systems More Responsive to Student Needs

4.  Improving the Content and Process of Assessment with Technology

5.  Finding Appropriate Learning Resources and Making Informed Choices

1.  Making Sure Learning Resources Promote Deeper Learner

This chapter addresses data mining and research design.  It does not address current problems in the education system.  I wonder if this work is about creating a problem in order to generate solutions.

Is technology really advancing education or is education advancing technology?  All along, file000194126603good teachers have been doing many things technology is and will be doing.  We have had technology in education for years with computers entering classrooms and labs thirty or so years ago.  There has been a variety of software available over the years—stand alone, networked, etc. and some with good management systems.  We have had computers, smartboards, palm pilots, internet, powerpoint, digital blackboard, moodle, ipads, clickers, and more with more to come.  Do we have any good credible research that shows a good return on investment?  I imagine the corporate vendors get a good return but what about a return reflecting positive student achievement?  Where’s the evidence to support such an investment?

The report does mention the Common Core State Standards and the new science standards (still in draft form) and that the standards were designed and intended to develop “deeper learning”.  It is wonderful that we have the Hewlett Foundation to define “deeper learning” rather than classroom teachers.  I wonder how regular classroom teachers translate this into classroom instruction.  It may require technology.

These new standards were crafted to reflect “deeper learning,” defined by the Hewlett Foundation as the ability to acquire, apply, and expand academic content knowledge and also to think critically and solve complex problems, communicate effectively, work collaboratively, and learn how to learn.

Is all of this expanding evidence approach/data mining and technology really to promote deeper learner or to make quick adjustments to better indoctrinate students?

Just how useful will all of the data that is gathered and generated be?  It is questionable as to whether developers will even be able to interpret the data.

These data can be used to inform rapid cycles of testing and refinement, provided that developers have the expertise to interpret them.

Technology developers and venture capitalist are interested in education.  This report mentions four factors that are fueling that interest.  They fail to mention the potential profit in the education market as a fuel factor.  This interest does not seem be fueled by any interest in providing students with a solid academic education.

The last five years have been a time of unprecedented file000453200083interest in education by technology developers and venture capitalists. This interest is fueled by several factors: the availability of more powerful computers, advances in software and cloud computing, philanthropic and social business goals, and the belief that common standards could bring greater coherence to the education market.

An example is given in this chapter about some analysts noting a class no longer making progress.  They presented their data to the principal.  When the principal saw the data he was prompted to do what he should already have done—hire a regular teacher for that class instead of staffing it with a substitute.  Really?  If it took the analysts’ data to help this principal figure this out maybe he should be replaced.  Will we reach the point that principals will administer schools by sitting in front of an array of computer monitors filled with analyzed data rather than visiting classrooms and personally talking with teachers and students?

2.  Building Adaptive Learning Systems That Support Personalized Learning

As more and more technology is being used in education the “personal” aspect of file0001344599812education—that personal interaction between student and teacher–is becoming less personal as students focus more on computer/technology work rendering teachers more remote from students.  Are advanced digital learning systems depersonalizing education?  These systems may be able to model, diagnose, and respond but will they be able to deliver a solid well rounded academic education to our students as well as a skilled and knowledgeable teacher can?

Although one-on-one sessions with a skilled human tutor who dynamically understands and responds to the person being tutored offer the most personalized experience, digital learning systems have advanced greatly in their ability to model the knowledge and competencies students should acquire and to diagnose and respond dynamically to learner needs.

3.  Combining Data to Create Support Systems More Responsive to Student Needs

This chapter talks about combining data with other sources to better meet student needs.  The other sources referred to are largely social service agencies.  Will combining educational data with social service agency data provide better instruction and result in greater academic achievement?  I wonder if credible research findings support combining such data.

Is combining data with social service agencies outside the realm of what our schools, watchingeducation system, and government should be doing? Does this kind of cross agency data sharing/combining violate student privacy?  When a student interacts with a social agency for any reason, will they and their parents be informed and know the information will be “combined” (shared) beyond that social agency?  What if students and parents don’t want their data “combined”?  Combining seems to be a term used to replace data sharing.  When data will be combined are the sources of the various data elements lost in the new combination? This practice will invade student life outside school and is highly objectionable on many grounds—morally, ethically, and legally.

4.  Improving the Content and Process of Assessment with Technology

This chapter addresses issues some people will find very alarming while others will not understand why it should be alarming.  It starts off saying

Digital learning systems can collect data on important qualities not captured by achievement tests.

We haven’t done a good job of collecting data from achievement tests—NCLB put most states, if not the whole country, on a different path where subjective data has been collected by assessments rather than collecting objective test data.  The data we have collected has not always been well used.  Rather than focusing on effectively and intelligently using data already collected it seems like we are moving on to collect other data.  Will that be useful?

This chapter is talking about collecting data on “skills such as collaboration and innovation, and personal and affective qualities related to intellectual curiosity and persistence.”  Collecting data on fundamental basic skills in reading and math or content knowledge in science, history, or social studies gets no evident mention so apparently is of little to no importance.

The fine-grained information about students’ learning that newer digital learning systems collect can be used to construct measures of important learning outcomes and learning processes that have been difficult to capture with conventional state tests. What is more, these data can be mined to assess both cognitive and noncognitive skills—the latter being more oriented toward personal qualities such as conscientiousness and self-efficacy in college and the workplace. Specifically, digital learning systems provide the opportunity to measure these qualities on the basis of students’ behavior in a learning system rather than through self-report.

This takes us in the direction of collecting and assessing subjective data about personal securedownloadqualities, conscientiousness, self-efficacy, collaboration, innovation, intellectual curiosity, and persistence.  What do teachers want to know?  Wouldn’t they already know these things about students based on ongoing personal interaction with the students?  I guess not, especially the more remote the teachers role becomes.  What if a parent wants to know if a child can spell?  Use correct grammar?  Add, subtract, multiply, and divide efficiently with the standard algorithms?  What if a parent does not want this kind of subjective data about their child collected?

More of what educators really want to assess can be measured by mining the data produced when students interact with complex simulations and tasks presented in digital learning systems.

When it says educator here I doubt it is referring to the classroom teacher.  How useful will this be to classroom teachers?  Have they asked for this kind of assessment?  I would be surprised if they have.  I wouldn’t be surprised to find the majority of teachers have no interest in the kind of assessments being talked about here—if they even understand it.  This should not be taken as belittling teachers; most teachers are focused on more personal instructional interactions with their students.

A number of research groups are working on how to make data gathered from online learning systems useful within accountability contexts as well as for individual learners and teachers.

Good luck with that is what I say to those research groups.

5.  Finding Appropriate Learning Resources and Making Informed Choices

This chapter starts off saying:

Learning resources and materials play a critical role in achieving desired learning outcomes. Educators need better supports as they make decisions about which digital learning resources to adopt.

Is there a leap of logic or a lack of logic here?  This report on evidence approaches provides no evidence digital learning resources are better or more effective than print-based materials.

These digital resources give educators more choices, but they also raise the issue of how to ensure their quality and determine their effectiveness in achieving desired learning outcomes.

Besides the Internet, two other factors are driving the trend of teachers supplementing print-based textbooks and other materials with digital learning resources: easy-to-use creation and publishing tools that enable anyone to create, configure, aggregate, and modify learning materials and Internet supported resources such as online repositories and communities that make it easier for educators to find and evaluate resources that might meet their needs.

How do we know this makes it easier for educators to find and evaluate resources?  It’s possible it makes it more difficult as a result of a plethora of substandard, inefficient, and ineffective yet well packaged and promoted learning materials.

This chapter indicates the purpose for expanding the data mining and analytics.  Without directly saying so, it appears the purpose is for greater marketability, focus marketing, and most important—profitability.  Profitability lies in having a constant stream of new or “improved” products entering the market place.  Is this data mining and analytics more about improving profitability than improving student academic achievement?  R&D findings may give evidence a different version is needed.  Marketing that different version will result in greater profits.  Is this a continuous loop of finding evidence-based reasons to introduce different versions to generate that constant stream of products entering the market place?

In closing my comments I present a couple of recommendations from the report and pose a few questions.

Here is one recommendation made early in the report and again addressed in the summary and recommendations at the end of the report:

11. Institutional Review Board (IRB) documentation and approval processes for research involving digital learning systems and resources that carry minimal risk should be streamlined to accelerate R & D without compromising needed rights and privacy protections.

Streamline or bypass rights and privacy protections?  Recent changes in FERPA may have made this recommendation a moot point.  Regulations now seem to give more rights to any one wanting access to student data than to individuals concerned about their privacy and privacy protection.

Here is another recommendation:

12. R&D funding should be increased for studying the noncognitive aspects of 21st-century skills, namely, interpersonal skills (such as such as communication, collaboration, and leadership) and intrapersonal skills (such as persistence and self-regulation).

Where will the funds come from?  Our federal government should not fund any part of this.  This draft report does not indicate how it was funded but the U.S. Department of Education Office of Educational Technology publishes it.  Why is our federal government involved in something that should be in the private sector realm?  Should our federal government be involved?

Great importance is placed on getting immediate results from research.  The technology laid out in this report will generate results so quick the education community won’t know what to do.  The results generated are not student academic achievement results.  There will be a constant push pull effect.  The implementation of strategies based on results may be outmoded by new results with changes made before determining how effective the earlier strategy really was.

Oh, yes, we love our data.  We already have an adequate supply of data.  We don’t effectively or intelligently interpret or apply currently available data.  So let’s allow our schools and government to gather more data about things they have no business collecting, let alone combining/sharing and data mining.  Wrong.  Bad idea.

No evidence has been presented in this report convincing me that educational data mining and learning analytics will improve student academic achievement.  Is this more about applying a business/technology model to education on a massive scale?

This entry was posted in Data Mining / Privacy, Testing / Assessments. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s