Maximum PC

Hello everyone, I have been doing extensive research into effective programming aptitude testing for prospective students- assume students with little or no experience in computer programming. A lot of work has been done, from design modeling to template work, and yet little correlation seems to have been observed.
Have any of you come across any pre-course assessment tests? I have been thinking of designing an exam based upon a simplified regular expression pattern matching model, but have little supporting evidence beyond the sense that if a prospective student can assemble a regular expression to filter a select group of characters, then she can progress further into software development. Before I launch into design and testing of this concept, I endeavored to uncover any existing work in this direction. Has anyone heard of a regular expression-based aptitude test?

I think more work should be put into teaching students than choosing them, but I'm a cynical old bastard.

Schools have a moral obligation to instruct students that are capable of learning the subject, given the onerous amount of debt taken on by the students. Government regulations require that schools do not simply screw prospectives that cannot grasp the concepts. Western Association of Schools and Colleges accrediting body requires that schools adequately assess student ability coming in. Is there anyone that visits Max PC that has anything fruitful to offer in the first question?

I'm sure someone might .. we have quite a few academics and professionals that frequent this forum.

You'll want to can the snarkiness, though .. it doesn't help you find someone answer your question.

Offhand, I would think that standard Algebra word problems, analytical questions similar to the ones you would find on the LSAT, and especially, geometry problems (including proofs) would be good indicators. However, these might be better indicators of success in a computer science program and not just success as a programmer -- those can be two very different things. Plus, programming varies considerably... "web programming" has nothing in common with say implementing a computational geometry algorithm.


By "design modeling", are you referring to software design? Things like OA/D and design patterns? In my experience, this is area seems to correlate highly with general software development experience. You have to spend quite a bit of time writing software before you can really appreciate some of this stuff. I don't see how you would test for this ability amongst experience programmers, much less someone without programming experience.

By "template work" are you referring to C++ templates? Maybe templates suck? =)



Nada. But I would say that if someone can absorb the contents of the first two or three lectures of SICP, they're probably going to be a pretty damn good programmer.


Hmm... I'm not sure that would work. First, the syntax would put off a lot of people. It would just seem too foreign. Second, you give anyone that has happened to use regexs an incredible advantage. Of course, you could limit the grammar to just . * + and ? and use the standard write a grammar that accepts aaabaabaabaab type of questions. I'm not sure that would cover enough though.

I would encourage you to look at an old copy of the GRE CS subject test and try to "dumb-down" the questions for people without a CS background. By dumb-down, I mean that you'll have to provide a lot of "assumed knowledge" by rewording the questions. Also, you might want to look at some of the easiest questions used in TopCoder competitions and see if you can rework those questions into something useful.

Just throwing my two cents into the issue of why you'd want to do this... it seems to me that if you could create an aptitude test that works, you would have something immensely useful (especially in the US where MTV seems to have invaded our memes, and universities, with their dumb is better mindset). I totally forgot... Russia has done this type of thing for decades and produces some of the best competitors in a number of "mental sports" like chess, mathematic competitors and programming competitors. Maybe you can gleam something from their system. Of course, it might not be the system that is responsible for their success either.

I also agree with Jipstyle along the lines that people shouldn't be excluded from pursuing their interests either. Fortunately, nature seems to have been programmed somewhat sensibly in that people tend to be passionate about things that they're somewhat good at... most of the time.

I had not heard of grammars till I took Autamata... How many 18 year olds other than "extreme nerds" do you know of that knew what a RegEx was pre-college? You are thinking about all of the wrong things to test. The Top Coder thing was a decent idea. Don't assume that you will suck in a CS&E related field just because they come in without any programming knowledge.

I don't think you need to worry about pre-testing for programming honestly. The first semester in my program (Intro to programming, Calculus I, Chemistry, some other gen ed classes, and an intro to EECS course, which was basically designed to show you what you were getting into) was enough to weed out those that simply couldn't and the second or third was for those that didn't really want to.

There is a lot more to succeeding in CS&E than just being capable. I know lots of people who are capable who would be terrible at it.

Exactly.

Let's add the fact that teaching people to think logically is part of the job of an educator. A private school can choose its own students, of course, but I think that does everyone a disservice.



Either that or those people have been pursuing those interests for a while and have practiced those skills. It is a question of nature vs. nurture.

My experience shows that most people of moderate intelligence can be taught to think logically. The idea that only some people can do it is silly and based on years of the subject being taught by and within an insular subculture.

Essentially ... geeks and nerds like to think that they are special. Combine that with an inability to communicate effectively and you have a subculture that thinks that the ability to reason is rare and difficult. It is neither.

/rant

Thanks for the feedback. In terms of templates, I am referring to
http://jite.org/documents/Vol5/v5p271-2 ... amy115.pdf
a study performed by Imamy et al.
Regarding regular expressions, I am less interested in determining if potentials are familiar with regex, and curious as to whether the pattern matching process implicit in regex could divine an aptitude in programming. I have done further research and come to the conclusion mentioned by Gadget. Computer programming is essentially a mathematical system (Dehnadi and Bornat, http://www.eis.mdx.ac.uk/research/PhDArea/saeed/), so I decided to pursue pedagogical techniques used to teach math. In fact, Saeed Dehnadi mentions mental models as a more fundamental description of how students acquire knowledge: acquire chunks of information, learn to identify a given chunk in different scenarios, extrapolate general models or rules for that chunk, and imagine new combinations of those chunks to synthesize new purposes. Apologies for the pedantry.
Good grades in applicable subjects show no correlation, unfortunately. So I am uncertain that simply relying upon math exposure is sufficient.
In regard to letting students pursue their interests, I agree 100%- however, loan guarantors expect that the investment they make in a student will result in marketable skills that enable the student to pay those loans back. And as of this year, the sole guarantor is the U.S. government, so flexibility is removed for students that lack the funding out of pocket. Schools that ignore the ability of the student risk future funding opportunities for their students, to leave ethics aside.

Again, thanks for your input, it suggests new avenues for me to pursue.I like to describe method implementations as "planning ahead" to first term students. The inference is that the student must consider what circumstances could arise in a hypothetical scenario, and plan explicit responses to the variety. Sounds like a successful MBA grad, in hindsight.

Nerds certainly don't have a lock on logical thinking. =)

I think it comes down to people having slightly different sets of skills and cognitive abilities. The average person would certainly find programming in C or assembly (sorry about the redundancy there) to be a very difficult task, and even people with above average intelligence will find proving that an algorithm is correct and complete to be very daunting (except for trivial cases). Then again, the average programmer rarely does these things during the normal workday.

Interesting discussion... =)

Regarding templates, I think the idea mentioned in the paper is interesting and it might help with reinforce ideas. However, I also wonder if the authors forgot some of their own points while creating their tool. For example, they mentioned too much time is spent on syntax in beginning programming classes -- C++ is damn near the worst language to learn from a "syntax" point of view. This is a big part of the reason that schools like MIT used Scheme in their introductory programming course (IIRC, MIT students can now choose between Scheme or Python). Now I realize that C++ is an important language in industry and won't be going away any time soon, but it is neither a good introductory language nor a very good teaching language. Students spend too much time debugging incredibly misleading compiler errors when they should be spending time thinking about other things (ie you left off a semi-colon on line 4, so I'm going to suggest there is a memory alignment error on line 400 then print fail).

And that brings me to another point... unfortunately, many instructors do a poor job at teaching programming from a pragmatic programming perspective. Many undergraduate CS classes include a "group project" because with the rational of "they'll have to program in teams" in the "real world" by an instructor that hasn't adequately prepared them for such a task. This is frankly a bullshit argument. Yes, you do a lot of programming on teams in the "real world", but you can also be fired for performing badly, quit and join a new employer, etc.

The biggest problem is that group projects require a substantial amount of preparation by the instructor which often isn't done. If a programming task is either large enough or difficult enough as to require a group of people to finish it, you had better prepare them for working on a team (otherwise your setting them up for failure and/or a horrible experience). At a minimum, they should be taught how to use an appropriate debugger and either have some kind of collaborative environment setup for them. The Berkeley and USC operating systems courses get this (mostly) right. You're given the code for a toy OS that you have to fix/improve. You're also taught how to use the debugger, environment, and provided suggestions for working together. You're not just left to "figure it out" on your own. You actually spend most of your time thinking about how to write operating system code, and not, wanting to shoot yourself in the head. Well, you may still want to shoot yourself, but it won't be due to someone writing incompatible code in VS when everyone else was using a Unix environment.


Often, it will depend on the domain. For example, someone working on robot path-finding will almost certainly have a fair bit of math background (at least at any company that is planning on surviving); however, I'm almost always dismayed at the LACK of mathematical rigor displayed by "database programmers" (where mathematical in this context refers to relational algebra, normalization, etc). It's unfortunate.



Frankly, I wish that academia were somewhat more insulated from the "demands of industry". Companies are often very "fadish" jumping on the next big thing like Six Sigma (pronounced Sick Sigma), CMMI, OOP, etc. If companies had their way, they'd probably rewrite the average CS curriculum to be more like...

Algorithms: How to determine if you should write a "better" algorithm (answer: rarely). No math required.

Compilers: We use these compilers. Read the instructions.

Programming Language Theory: See compilers.

Software engineering: Here are our software development processes. Memorize them. Yes, we renamed Waterfall model to the XYZ Software Development process.

</rant>

There were only two programming group projects I had when I was in school. One was software engineering (and that was our Junior year). We learned software engineering, development models, etc, etc... we were then given a list of requirements, verbally, that we had to write down. We could of course ask questions, but the professor did not provide typed up requirements. I liked that. We were given a list of ins and outs, and had to come up with the rest. The other was senior design which was all up to use to work on. We had three people in our group and it worked really well. Then again, we all had Software Engineering together and worked through that really well and LEARNED something.

Now, I have seen other group projects go horribly wrong.

One thing I have noticed with the program where I went (for easy of explanation, a satellite of the main campus that was hosted at a community college) is that each class (cohort group) that goes through the program is VERY different.

The class above mine was pretty independent of each other and had a couple small groups with several loaner types. Ours was a huge group with a couple idiots that managed to fake their way through, our problem is that all of us were very smart (as told my a professor) so we butted heads a lot. Like my other post, you get a lot of smart people in a room and you get lots of conflict.

The group that is going through right now, set to graduate this semester or next is also a big group, but they have lots of people that like to take over and talk over others. It doesn't work out well. They needed more structure as a class to work well together. Their software engineering teacher hasn't provided that. No clear definition of the assignment, no assigning of groups, no requirements, no specific deadlines... and no real lecture on the software engineering process. So you have a few people who are code monkeys that go all caveman and code, and then everybody else is kind of shut out and talked over. The end results is that nobody really learns anything about Software Engineering, working together with others, or anything for that matter. The class is a joke, and the project fails on so many levels.

I would have certainly preferred fewer group projects at the undergrad level. Of course, a certain amount of programming should be required in each course. There is no substitute.

In my experience, the biggest problem is that too many of the professors don't really know the tools, environments and (sometimes) the languages they're supposed to be teaching. With all of the online instruction available now, all of the top schools appear to be providing their students with the necessary background information (eg how to use emacs, NetBeans, whatever) and decent programming exercises. Its a bit sad that this level of instruction doesn't appear to filter down very far (eg SICP has been around for >10 years, but most AI/Lisp teaching is incredibly misguided).


Based on my experience working at one university and being a research assistant at another, you won't ever see that at the larger universities. The professors actually spend quite a bit of time creating/revising their classes and everything (AFAIK) has to be in writing and approved by some quality control type people prior to the start of instruction. I've been a (paid) guinea pig for this type of thing.

I imagine what happened in your case is that the professor did have all of the requirements written up in advance. He then graded you based on how many of those requirements you were able to extract. Having you extract the requirements from him verbally is certainly an excellent exercise (esp for small / medium sized projects). I just wish their was more "exploratory programming" (or extreme programming) going on outside of research labs.


If life provides you the opportunity to attend a top school at some point, I would highly recommend taking it! Not only do the top schools provide you with more and better job/career opportunities, but the course content and selection is vastly superior. When I started at USC, I thought that I knew the basic outline of the rabbit hole (using the Alice in Wonderland cliche/analogy). I certainly didn't everything in detail, but I felt that I could at least be conversant in most areas of compsci especially in my main area of interest (algorithms / algorithm analysis). I had a general familiarity with most of it.

Boy was I wrong. Linear programming, semi-definite programming, PSPACE-Complete complexity class... I hadn't even heard of these terms... and this was only the first class! This wasn't supposed to happen after I had taken 20 or more CS courses, spent two years working part-time at a university, and three years in "industry". I had barely stuck my head into the cave thinking that I knew something... humility is sometimes thrust upon us (or perhaps through us)!


Actually, I think the notion that too many smart people leads to conflict is wrong. I've worked with groups where I was certainly the smartest guy (>90% of the time at Boeing!) and where I was not even close to the smartest -- especially when Adleman and/or Kempe are part of the group! In my experience, a group of "truly" smart people tend to get along just fine (at least as well as the average group).

The problem is when you have a group with a couple of smart people, two or three people who think they're smart, and the rest are average or just hanging on to a job (and probably don't care). The problem here is that you can't convince the "I think I'm so damn smart" types that they're wrong, and invariably, they will somehow be the tech lead or the manager's friend. And if you haven't spent some time dealing with this type of person, you can often see problems with their idea (or you might have a better idea), but you are unable to convince people to change things. This is especially true when first starting out because you haven't spent much time articulating the pros and cons of various approaches or playing devils advocate.

Of course, all of this is exacerbated on your first big project because the diamonds in the rough are still rough.