If you’re reading this, chances are you have strong Excel skills and already know what INDEX/MATCH formulas do. For the rest of you, here’s a short introduction:

INDEX/MATCH formulas, created by combining Excel’s built-in INDEX function and its built-in MATCH function into a single compound formula, are ideal when you need to:

• Merge data from one Excel list into another Excel list by matching records from the two lists; or
• Use a common field from two Excel lists to lookup a second (or third or fourth) field by matching records from the two lists.

For instance, suppose you had two Excel worksheets for the same group of customers. The first worksheet contains columns for Customer ID and Email Address. The second worksheet contains columns for Customer ID, Phone Number and Age. With Customer ID as the common column, you could use an INDEX/MATCH formula to add each customer’s phone number and age to the email worksheet.

For SQL experts, you can think of INDEX/MATCH formulas as a way to use Excel to do inner joins.

Quick Sidebar

At this point, someone is undoubtedly thinking: I could do the same thing faster in Microsoft Access with a lookup query in Design View or in Crystal Reports with the link tab of the Database Expert. You are probably correct, but this post is intended for everyday users who only have or know Microsoft Excel or situations where setting up an Access DB or a new Crystal Report is just not warranted. But I digress.

A standard INDEX/MATCH formula is written like this:

Index( value_array, Match( lookup_value, lookup_array, match_type ), column_number )

The MATCH portion returns a position in a list. The INDEX portion returns a value in a cell. So combining them together allows you to lookup a value in a cell based on the position of an item in a list. (What the formula actually does is use a MATCH function as the second argument of an INDEX function.)

Here’s the Trick

Instead of trying to digest all of the above, just rewrite the formula in the following way and replace the double-bracketed portions with your actual data or cell references.

=INDEX([[find this kind of value]],MATCH([[for this cell within the **first** list]], [[with a match within this **second** list]],0))

A few parting notes that might be additionally helpful:

• The MATCH portion of the formula is processed before the INDEX portion.
• If you plan to use AutoFill to copy the formula down a column, ensure that the lookup array is either a named range or an absolute reference to a range.
• You cannot refer to an entire column as the lookup array for the MATCH function; You must specify an exact cell range.
• The 0 at the end of the MATCH portion is optional and one of three possible choices (1,0,-1). 0 means find an exact match. 1 means find the highest value that matches. -1 means find the lowest value that matches. If you omit this argument, it defaults to 1, which is almost always what you want.

If you Google “missing data” you will be barraged with complicated statistical techniques. That’s par for the course, but none of these sites ever seem to tell you how to actually implement these techniques in the real world with real data. So I’ll give it a try.

Not that it’s best, but this is the approach that I use:

• If less than 5% of data points are missing, I use plain old Listwise Deletion.
• If less than 10% of data points are missing, I use Mean Imputation. Yes, I know it artificially inflates central tendency and affects standard error. But since such a small amount of data is missing, I can live with it.
• If more than 10% of data points are missing and I’m confident the missing data are MAR or MCAR, then I use Multiple Imputation.
• If more than 10% of data points are missing and I believe the missing data are NMAR, then I pull out the big guns and use Heckman Selection Modeling.
• If the missing variable is Race, I simply assign the record to the “Other” category and forget about it.
• If the data are longitudinal and it’s not the first wave, then I just use the mean of the subject’s previous observations and forget about it.

And here’s how I go about it:

For Listwise Deletion, I import the data in an Excel worksheet then use Autofill to select the blank records or variables.

For Mean Imputation, I use Excel’s AVERAGE function to calculate the mean and then use an IF statement to insert that mean into all missing records. If your dataset has tens of thousands of rows, this can be excruciatingly slow in Excel. So you should know in those cases it can be done faster in SAS with PROC MEAN.

For Multiple Imputation, I use SAS’ PROC MI function to run the imputations and its PROC MIANALYSE function to calculate the summary statistics on the regressors. I always run five sets of imputations because studies by very smart people have shown this is enough. Excel gurus know that the summary statistics can be done in Excel by searching for the text string _imputation_ (including the underscores) and separating each imputation into a separate worksheet. The string _imputation_ is the delimiter that PROC MI inserts into the dataset.

For Heckman Selection Modeling, I use Stata’s HECKMAN command, because it lets you choose between maximum likelihood and two-stage estimation. I always use the two-stage option, although there’s probably some complicated rule somewhere in the manual explaining when to use which. SAS lovers should know that SAS (version 9 and later) can do Heckman estimation with its PROC QLIM function, but keep in mind it’s limited to the maximum likelihood version of the model. Either way, people who don’t know about these functions will be dazzled with your skills. And it sounds fabulous when the footnotes of your survey say something like, “Missing values were imputed using two-stage Heckman Correction Estimation.”

For what it’s worth:

If somebody asks me why I use Multiple Imputation over alternate methods, I just say, “If it’s good enough for the Census Bureau, then it’s good enough for me.” Then I walk away and go to Starbucks.

If somebody asks me why I use Heckman Selection Modeling, I just say, “If it’s good enough for the Nobel Prize Committee, then it’s good enough for me.” Then I turn my iPod back on and spin back towards my computer screen.

I’ve heard that Stata’s ICE command is better than SAS’s PROC MI function, but I’m so accustomed to using SAS and Excel for this that I’ve never tried it.

I know I won’t win any goodwill points from the American Statistical Association for saying this, but unless your missing data could affect something really important like, say, nuclear missile targeting, I wouldn’t lose sleep over the choice of technique you choose because every one of them has legitimate weaknesses and the probability that no one really cares is 99.99%.

Jamel Cato
The Blue Collar Data Analyst
2007

There are three main ways to calculate percentile ranks in Oracle 9i or later:

(a) Calculate them manually with Joins and Sub-queries;
(b) Use the CUME_DIST function and format the results as a percentage;
(c) Use the PERCENT_RANK function.

In this post I’m going to focus on options (b) and (c) because the only people who would reinvent the wheel and use (a) are SQL programmers who don’t know (b) and (c) already exist.

CUME_DIST and PERCENT_RANK are built-in Oracle mathematical functions that allow you to rank a value based on its relative standing within a set of values. For example, if you hear someone say that a 1600 SAT score was in the 99th percentile (meaning 99% of all the other scores in that administration of the test were lower) a ranking formula is what tells you so.

The two functions take different approaches to determining a percentile rank and you should understand the basic difference.

CUM_DIST determines a percentile rank by calculating the ratio of the number of rows that have a lesser or equal ranking to the total number of rows in the partition.

PERCENT_RANK determines a percentile rank by setting the lowest value (that is, the first row returned by the query) equal to 0 and assigning all the remaining rows with this formula:

(n-1)/(m-1) where n is the nth row in a partition of m records.

There are other differences between the two functions but a detailed review is way beyond the promised scope of this post. However, you will do well to remember four points:

• The two functions are similar, but (unlike many books lead you to believe) they are not identical. That’s why they return different answers when you use them side-by-side in the same query.
• CUME_DIST returns a position of a row and PERCENT_RANK returns a rank of a row.
• CUME_DIST always excludes 0 and PERCENT_RANK always includes it.
• In most cases where your goal is to rank records by percentile, PERCENT_RANK is the function you want to use.

Both functions can be used in two forms: aggregate or analytic. Use the aggregate form when you want to find the percentile rank of one particular record in the database according to some criteria you specify. Use the analytic form when you want to find the percentile ranks of a group of records in the database. You can spot the aggregate form because the SELECT statement will contain a WITHIN GROUP outer table join. The analytic form will use a PARTITION BY clause instead.

The aggregate form uses this syntax:

PERCENT_RANK (expression) WITHIN GROUP
(ORDER BY order_by_clause [ASC|DESC] [NULLS FIRST|LAST] );

Here’s a simple example:

SELECT PERCENT_RANK (100000000, 1000000) WITHIN GROUP (ORDER BY total_gross, star_salary) "Percentile Rank" from movies WHERE movie_year IN ('2006');

The above SQL statement will return (from a table called Movies) the percentile rank of a particular 2006 movie that grossed $100 million and paid its starring actor $1 million.

The analytic form uses this syntax:

PERCENT_RANK () OVER
([PARTITION BY query_partition_clause] ORDER BY order_by_clause);

Here’s an example of that:

SELECT movie_name, movie_year, movie_type, total_gross,
PERCENT_RANK () OVER (PARTITION BY movie_type
ORDER BY total_gross DESC) "Percentile Rank"
FROM movies
WHERE movie_year IN ('2006');

The above SQL statement will return a table listing the percentile rankings of all movies released in 2006 according to their total box office gross.

As a closing note, remember that when using the aggregate form the number and datatypes of expressions inside the first parenthesis must match the number and datatypes of expressions inside the second parenthesis.

Now go rank some data.