If you’re reading this, chances are you have strong Excel skills and already know what INDEX/MATCH formulas do. For the rest of you, here’s a short introduction:

INDEX/MATCH formulas, created by combining Excel’s built-in INDEX function and its built-in MATCH function into a single compound formula, are ideal when you need to:

• Merge data from one Excel list into another Excel list by matching records from the two lists; or
• Use a common field from two Excel lists to lookup a second (or third or fourth) field by matching records from the two lists.

For instance, suppose you had two Excel worksheets for the same group of customers. The first worksheet contains columns for Customer ID and Email Address. The second worksheet contains columns for Customer ID, Phone Number and Age. With Customer ID as the common column, you could use an INDEX/MATCH formula to add each customer’s phone number and age to the email worksheet.

For SQL experts, you can think of INDEX/MATCH formulas as a way to use Excel to do inner joins.

Quick Sidebar

At this point, someone is undoubtedly thinking: I could do the same thing faster in Microsoft Access with a lookup query in Design View or in Crystal Reports with the link tab of the Database Expert. You are probably correct, but this post is intended for everyday users who only have or know Microsoft Excel or situations where setting up an Access DB or a new Crystal Report is just not warranted. But I digress.

A standard INDEX/MATCH formula is written like this:

Index( value_array, Match( lookup_value, lookup_array, match_type ), column_number )

The MATCH portion returns a position in a list. The INDEX portion returns a value in a cell. So combining them together allows you to lookup a value in a cell based on the position of an item in a list. (What the formula actually does is use a MATCH function as the second argument of an INDEX function.)

Here’s the Trick

Instead of trying to digest all of the above, just rewrite the formula in the following way and replace the double-bracketed portions with your actual data or cell references.

=INDEX([[find this kind of value]],MATCH([[for this cell within the **first** list]], [[with a match within this **second** list]],0))

A few parting notes that might be additionally helpful:

• The MATCH portion of the formula is processed before the INDEX portion.
• If you plan to use AutoFill to copy the formula down a column, ensure that the lookup array is either a named range or an absolute reference to a range.
• You cannot refer to an entire column as the lookup array for the MATCH function; You must specify an exact cell range.
• The 0 at the end of the MATCH portion is optional and one of three possible choices (1,0,-1). 0 means find an exact match. 1 means find the highest value that matches. -1 means find the lowest value that matches. If you omit this argument, it defaults to 1, which is almost always what you want.

The first reason is that nothing happens in the movie. That’s not a spoiler or a clever play on words. Sadly, it’s a plot synopsis. If you can believe it, The Happening is an apocalyptic thriller without an apocalypse. Sure, there’s a cataclysmic “event” in the beginning. But then no chaos, excitement or real suspense ensues. Even though humanity is facing possible extinction, everyone stays calm, rational and orderly. In addition to being wildly unrealistic, such a storyline is boring.

If the “event” in the movie happened in real life, there would be widespread panic. Hazmat crews would be everywhere. Cell phone networks would crash. Children would be ripped from their mothers’ fingertips in the inevitable chaos. But nothing remotely like this happens in the movie. Actually, after the first 10 minutes, not much of anything happens, making the film’s title the ultimate oxymoron.

As if a dreary plot was not harm enough, the special effects, dialogue and editing are film-school amateurish. As many other reviewers have decried, I can confirm that there are indeed two scenes in the film where the microphone grip accidentally falls into the frame.

The second reason I’m so disappointed with The Happening is that, like many of his former fans, I really wanted to see Shyamalan redeem himself after the dreadful Lady in the Water. As bad as that movie was (and it was very, very bad), at least it came with the entertaining distraction that was the behind-the-scenes political battle between Shyamalan and the Disney executives who didn’t want to lose their jobs by distributing such a dud. The infighting culminated in a notorious scene at the Four Seasons Hotel in Philadelphia where an angry Shyamalan threw down his napkin and stormed away from a dinner meeting with “the suits” who, after previewing the final cut, had flown across the country to personally beg him to rewrite the script. (Historical Note: The film eventually came out without a re-write and the executives indeed lost their jobs when it flopped.)

If I met M. Night Shyamalan today, I would ask him one question: Where did we go wrong? The Sixth Sense is one of the 10 best films of all times. Unbreakable is an underrated classic. He wouldn’t admit it, but the answer is The Village, his third major release. Shyamalan was stung by the heavy criticism of the movie and, in my opinion, started to believe the experts who said that his film’s hallmark surprise endings had inevitably turned into an Achilles’ heel because it was impossible to surprise an audience who came to the theater expecting a surprise. Both Signs and Lady in the Water, neither of which included a twist ending, proved that the experts were wrong. Twist endings are not Shyamalan’s problem. But I digress.

Back to The Happening. It sucks and nothing happens. Wait, did I say that already?

If after reading this review you still spend good gas money to see this film, you will wish that a certain wind blows by the home of everyone who played a part in green-lighting it. You will go from wondering if Shyamalan has lost his touch to being sure of it. Your date will be pissed, not just because you saw a bad movie, but also because the bad movie lacked enough content for interesting dinner conversation.

Here’s a happening for you: On my way out the theater I thought I saw Bruce Willis crouched on one knee talking to a little boy. As I passed by, it sounded like the boy whispered, “I see dead movies.”

Although I really, really hate to see the series come to an end, it’s high time that it does. No matter how enthralling and well-written, any mystery series centered on the detective capers of a middle-aged grandfather (which is what Easy has become by the last book) is bound to become a tough sell sooner or later. I’m sure Mosley was more cognizant of that than anyone, which is why Blonde Faith ended with such finality.

Each book in the series has a color in its title. I had read four of the books and hadn’t consciously noticed this pattern until it came up in a magazine article I read.

A Little Yellow Dog was my favorite book in the series, followed closely by Cinnamon Kiss.

Jackson Blue was my favorite secondary character. Every time his character made an appearance I was reminded that a beautiful mind and an ugly soul can coexist in one person.

I understand about Easy and Bonnie Shay, I do.

Mosley’s depictions of postwar Los Angeles are so vivid and historically accurate it’s like time-traveling.

I discovered the series when a good friend of mine insisted I read Black Betty. I loved it just like she said I would and can’t thank her enough for the introduction.

Most people don’t know that Walter Mosley is bi-racial (his father is black and his mother is Jewish.) I only mention that fact at all to say that Mosley’s striking ability—and choice—to chronicle the joys and pains of the black experience in America is even more interesting that I originally thought.

Of the several themes that Mosley weaved throughout the series, the most fascinating is this: Your good deeds today will not absolve you of the evil you did yesterday. No matter how much Christmas Black loves and cares for Easter Dawn, they can’t ever be a normal family because of what he once did on the other side of the world. No matter how grand Jewelle’s real estate empire becomes, she can’t forget what she did to Mofass to get it started. No matter how much of a safe haven Laselle Latour’s female-only boarding house provides for women in L.A., it will never make up for the bordello that she ran back in Houston.

The only character who was exempt from the gravity of this moral black hole is Raymond “Mouse” Alexander, the diminutive killer who was Easy’s right-hand man. Mouse would shoot you through the eye for breakfast and go dancing for dinner. As Easy discovered on many occasions, having Mouse as a best friend was like keeping a full-grown tiger for a pet.

I will miss having another Easy Rawlins mystery to look forward to. Mosley is among the best writers of his generation and the life of my mind has been enriched by these stories. One of my favorite passages from the series is this:

Proof is a funny thing. For policeman and for lawyers it depends on tangible evidence: fingerprints, eyewitnesses, irrefutable logic, or self-incrimination. But for me evidence is like morning mist over a complex terrain. You see the landscape and then it’s gone.

And then it’s gone.

I suppose that explains why I was on a mailing list announcing the official release of Crystal Reports 2008 this past October. (I’ve waited until now to post about the release because the Crystal Reports people have a history of releasing new editions where something important is either missing or wrong. UPDATE: Sure enough, the License released in October had a major error and had to be re-issued.) While it would have been more interesting to be on a mailing list that announced that SAP was about to buy the company behind Crystal Reports for $6 billion just a few days later, I take what I can get.

Ken Hamady at The Crystal Reports Underground has a good writeup about all the new bells and whistles (and the ones that are gone), so I won’t retread that ground.

I remember buying Crystal Reports at the Georgia Tech bookstore when the company was still called Crystal Services. And then it became Seagate. And then Crystal Decisions. And then Business Objects.

All that turnover in the corner office might be why certain basic features consistently get overlooked—like the ability to save a report in an earlier format. Who knows, maybe after each buyout the new engineers can’t reach the old engineers on their new yachts.

Happy Reporting.

The idea behind SIF is timely and necessary. And the protocol is well designed. But the whole initiative is badly in need of a good PR consultant. In my opinion, SIF is the poster child for Standards Gone Wild. The technical lexicon surrounding a SIF implementation is denser than London fog. I’m aware that technical people often like the esoteric acronyms and terminology that we think separates us from mere mortals. But in this case, when you’re trying to convince a whole industry to change a fundamental practice, much of it may be counterproductive.

First off, the term “School Interoperability Framework” by itself is enough to intimidate many people. Upon hearing it, non-IT people run for the hills.

If anybody had asked me, I would’ve suggested a simpler, more descriptive name that could readily be turned into a catchy acronym. I think something like the Open Protocol for Education Networks (OPEN) or more simply, the School Data Standard (SDS) would have worked better from a marketing standpoint.

If you want the whole world to adopt your standard, then make it adoption-friendly.

Okay, I better get back to writing that non-normative RFC 4122 XML schema for our vertical zone integration server.

If you Google “missing data” you will be barraged with complicated statistical techniques. That’s par for the course, but none of these sites ever seem to tell you how to actually implement these techniques in the real world with real data. So I’ll give it a try.

Not that it’s best, but this is the approach that I use:

• If less than 5% of data points are missing, I use plain old Listwise Deletion.
• If less than 10% of data points are missing, I use Mean Imputation. Yes, I know it artificially inflates central tendency and affects standard error. But since such a small amount of data is missing, I can live with it.
• If more than 10% of data points are missing and I’m confident the missing data are MAR or MCAR, then I use Multiple Imputation.
• If more than 10% of data points are missing and I believe the missing data are NMAR, then I pull out the big guns and use Heckman Selection Modeling.
• If the missing variable is Race, I simply assign the record to the “Other” category and forget about it.
• If the data are longitudinal and it’s not the first wave, then I just use the mean of the subject’s previous observations and forget about it.

And here’s how I go about it:

For Listwise Deletion, I import the data in an Excel worksheet then use Autofill to select the blank records or variables.

For Mean Imputation, I use Excel’s AVERAGE function to calculate the mean and then use an IF statement to insert that mean into all missing records. If your dataset has tens of thousands of rows, this can be excruciatingly slow in Excel. So you should know in those cases it can be done faster in SAS with PROC MEAN.

For Multiple Imputation, I use SAS’ PROC MI function to run the imputations and its PROC MIANALYSE function to calculate the summary statistics on the regressors. I always run five sets of imputations because studies by very smart people have shown this is enough. Excel gurus know that the summary statistics can be done in Excel by searching for the text string _imputation_ (including the underscores) and separating each imputation into a separate worksheet. The string _imputation_ is the delimiter that PROC MI inserts into the dataset.

For Heckman Selection Modeling, I use Stata’s HECKMAN command, because it lets you choose between maximum likelihood and two-stage estimation. I always use the two-stage option, although there’s probably some complicated rule somewhere in the manual explaining when to use which. SAS lovers should know that SAS (version 9 and later) can do Heckman estimation with its PROC QLIM function, but keep in mind it’s limited to the maximum likelihood version of the model. Either way, people who don’t know about these functions will be dazzled with your skills. And it sounds fabulous when the footnotes of your survey say something like, “Missing values were imputed using two-stage Heckman Correction Estimation.”

For what it’s worth:

If somebody asks me why I use Multiple Imputation over alternate methods, I just say, “If it’s good enough for the Census Bureau, then it’s good enough for me.” Then I walk away and go to Starbucks.

If somebody asks me why I use Heckman Selection Modeling, I just say, “If it’s good enough for the Nobel Prize Committee, then it’s good enough for me.” Then I turn my iPod back on and spin back towards my computer screen.

I’ve heard that Stata’s ICE command is better than SAS’s PROC MI function, but I’m so accustomed to using SAS and Excel for this that I’ve never tried it.

I know I won’t win any goodwill points from the American Statistical Association for saying this, but unless your missing data could affect something really important like, say, nuclear missile targeting, I wouldn’t lose sleep over the choice of technique you choose because every one of them has legitimate weaknesses and the probability that no one really cares is 99.99%.

Jamel Cato
The Blue Collar Data Analyst
2007

Here are the things I like:

• You can sign-up with just a username and password. No email required. No forms to fill out.
• It’s easy to find people with similar literary interests.
• The excellent search and browsing features.
• The best use of tagging on the Internet.
• The recommendation engine is way better than Amazon’s version.
• The Zeitgeist page, which features every imaginable statistic about the LT community. As a data analyst, I have to love this. Some of the statistics are remarkable, like the guy with over 14,000 books in his library. One guy. 14,000 books.

Here are the things I don’t like:

• It can be a lot of work, at least initially. Part of the reason this post doesn’t link to my LT profile is because I haven’t finished inputting the hundreds of books I own.
• The site has a lot of features, but there’s no Help section. And the FAQ page is almost impossible to find.
• While I’m enamored with the idea of networking with fans of my favorite authors/books, I’m not sure I want to let the whole world know what’s in my personal library in order to do that. I realize I could take the Spiderman approach and hide my identity behind a clever username like Not-Jamel-007, but then I can’t link my LT profile to any site that includes my real name—like this blog.

Notwithstanding these few minor annoyances, I have to say that LibraryThing is my favorite site right now.

But it’s not a panacea. Over 200,000 members and only 5 of them thought A Little Yellow Dog was worthy of five stars. Go figure.

There are three main ways to calculate percentile ranks in Oracle 9i or later:

(a) Calculate them manually with Joins and Sub-queries;
(b) Use the CUME_DIST function and format the results as a percentage;
(c) Use the PERCENT_RANK function.

In this post I’m going to focus on options (b) and (c) because the only people who would reinvent the wheel and use (a) are SQL programmers who don’t know (b) and (c) already exist.

CUME_DIST and PERCENT_RANK are built-in Oracle mathematical functions that allow you to rank a value based on its relative standing within a set of values. For example, if you hear someone say that a 1600 SAT score was in the 99th percentile (meaning 99% of all the other scores in that administration of the test were lower) a ranking formula is what tells you so.

The two functions take different approaches to determining a percentile rank and you should understand the basic difference.

CUM_DIST determines a percentile rank by calculating the ratio of the number of rows that have a lesser or equal ranking to the total number of rows in the partition.

PERCENT_RANK determines a percentile rank by setting the lowest value (that is, the first row returned by the query) equal to 0 and assigning all the remaining rows with this formula:

(n-1)/(m-1) where n is the nth row in a partition of m records.

There are other differences between the two functions but a detailed review is way beyond the promised scope of this post. However, you will do well to remember four points:

• The two functions are similar, but (unlike many books lead you to believe) they are not identical. That’s why they return different answers when you use them side-by-side in the same query.
• CUME_DIST returns a position of a row and PERCENT_RANK returns a rank of a row.
• CUME_DIST always excludes 0 and PERCENT_RANK always includes it.
• In most cases where your goal is to rank records by percentile, PERCENT_RANK is the function you want to use.

Both functions can be used in two forms: aggregate or analytic. Use the aggregate form when you want to find the percentile rank of one particular record in the database according to some criteria you specify. Use the analytic form when you want to find the percentile ranks of a group of records in the database. You can spot the aggregate form because the SELECT statement will contain a WITHIN GROUP outer table join. The analytic form will use a PARTITION BY clause instead.

The aggregate form uses this syntax:

PERCENT_RANK (expression) WITHIN GROUP
(ORDER BY order_by_clause [ASC|DESC] [NULLS FIRST|LAST] );

Here’s a simple example:

SELECT PERCENT_RANK (100000000, 1000000) WITHIN GROUP (ORDER BY total_gross, star_salary) "Percentile Rank" from movies WHERE movie_year IN ('2006');

The above SQL statement will return (from a table called Movies) the percentile rank of a particular 2006 movie that grossed $100 million and paid its starring actor $1 million.

The analytic form uses this syntax:

PERCENT_RANK () OVER
([PARTITION BY query_partition_clause] ORDER BY order_by_clause);

Here’s an example of that:

SELECT movie_name, movie_year, movie_type, total_gross,
PERCENT_RANK () OVER (PARTITION BY movie_type
ORDER BY total_gross DESC) "Percentile Rank"
FROM movies
WHERE movie_year IN ('2006');

The above SQL statement will return a table listing the percentile rankings of all movies released in 2006 according to their total box office gross.

As a closing note, remember that when using the aggregate form the number and datatypes of expressions inside the first parenthesis must match the number and datatypes of expressions inside the second parenthesis.

Now go rank some data.