Solved

Open discussion: Automated marking/grammar of submitted documents/text

Posted on 2011-03-04
5
525 Views
Last Modified: 2013-11-10
My client currently has an online system where students submit essays created in Word, or simply typed out as text in HTML text fields.

Currently my client has 600 essays to mark each month, which he is paying $25/hour for a person to do.  Therefore, he wants to automate this process ASAP.  We suggested just using multiple choice style questions, but he also has to obey educational demands meaning that essays are still required.

He is wanting me to create a program that performs some kind of text/grammar matching so that when the documents/text fields are submitted, the program can 'scan' the contents looking for certain included words/phrases/grammar, and provide feedback or a result.

Now I know how to do simple text matching, but the grammar thing has got me thinking...  As there are a hundred different ways to write the same sentence, I thought I'd ask on EE just to see if anything like this has been attempted before.

So, does anyone have any pointers, advice for how to achieve this?  Throw me your feedback & comments (both negative and positive!) and I'll split points accordingly.  Thanks :-)
0
Comment
Question by:Rouchie
5 Comments
 
LVL 18

Accepted Solution

by:
deighton earned 167 total points
ID: 35034982
well it is possible, because there are grammar checkers available on MS word etc.  It would be a problem incorporating heuristics, fuzzy logic and artificial intelligence - a big project in my opinion.  I can't tell you how to do it as such.

If I'd paid for my son to do a history course, and his essays were marked by a computer that gave him a mark for the phrase 'authoritarian rule' in an essay about Henry VIII, via a text search, I'd feel a bit ripped off that it was being marked like that.

But maybe I am naive about how essays are marked in practice.
0
 
LVL 25

Author Comment

by:Rouchie
ID: 35035054
>> If I'd paid for my son to do a history course....I'd feel a bit ripped off that it was being marked like that.

I completely agree, however, that decision is out of my hands - we're just being asked about the possibility of implementing it.  I've been researching this all morning and it seems quite a hot topic in universities, where lots of theses have been written regarding its possiblity.  I'd like to just say 'no' and be done with it.... :-)
0
 
LVL 12

Assisted Solution

by:Amick
Amick earned 167 total points
ID: 35035132
Essay grading probably awards part of the grade based on content and part of the grade based on spelling and grammar.  So far no computer has mastered either, so this is a huge undertaking.

For the spelling and grammar portion, you may want to take a look at using spelling and grammar engines like the one shipped with Microsoft Office as a starting point. Although it is far from 100% accurate, with nearly two decades of development it is getting better.  For the content portion you may want to research how IBM's Watson parses natural language, and note that even with a supercomputer and a multi-million dollar development budget it still wasn't 100% accurate in understanding language.

Although the commercial value of a successful and accurate human-language content and grammar module would be enormous, the investment requirement is unlikely to be supported in the current  business case.

If one is grading short responses looking only for key words, such as "Napoleon", "1812","Russia", the task becomes easier as a computer may be able to compare the current examinee's response with a database of previously graded responses.

Perhaps your client would be better served by creating a method to distribute the exam grading to a lower cost labor pool.  For example, allow remote graders to access completed exams from your client's website and pay them on a per graded piece basis. Have the same paper graded by multiple graders to help detect fraud or incompetence on the grader's behalf, keep grader merit-scores and adjust the number of times a paper is graded based on how competent the grader is.  Your client may end up with a better result at no more expense.
0
 
LVL 6

Assisted Solution

by:t-max
t-max earned 166 total points
ID: 35035134
I did some research on a speech recognition for my university degree, which might be somehow similar to what you need. It's no easy topic, because as you said there are several ways to say the same. Therefore what's usually done in speech/text applications like this, is to use math theory on statistics (mainly the Markov Chains) to check grammar. Here's a link to a google search on the matter:
http://www.google.com/search?q=markov+chains+&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a#sclient=psy&hl=en&client=firefox-a&hs=4aT&rls=org.mozilla:en-US%3Aofficial&q=markov+chains+grammar&aq=f&aqi=&aql=&oq=&pbx=1&psj=1&bav=on.2,or.&fp=82be9d6adeb3c13c
Another topic that might be important is AI, if you want to put some feedback into the system from the input you receive, as language might slightly change over time.
There are a few things more I could mention, but the above is the main thing to start looking at, at least from what I know. I'm not sure which qualifications you have, but I'm sure it's not an easy task, so I wish you the best. Regards!
0
 
LVL 25

Author Closing Comment

by:Rouchie
ID: 35066413
Thank you.  Those were the responses I was looking for.  "Expensive and inaccurate" are the key points I will communicate back.  The IBM Watson thing was particularly interesting.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Suggested Solutions

This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…
The viewer will learn how to use the return statement in functions in C++. The video will also teach the user how to pass data to a function and have the function return data back for further processing.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now