Filtering Data

Source: vignettes/filters.Rmd

Previous: analyzing data

Sometimes you only want to work with a subset of your data. With the crunch package, you can both filter the views of data you work with in your R session and manage the filters that you and your collaborators see in the web application.

Filtering and subsetting in R

As we’ve seen in previous vignettes, making logical expressions with Crunch datasets and variables is natural. We showed how to update a selection of values on the server, as well as how to crosstab in a subset of a dataset. Other applications work just as intuitively.

Filtering like this works by creating a dataset or variable object that has the filter embedded in it:

dems <- ds[ds$pid3 == "Democrat",]
dems
## Dataset “Economist/YouGov Weekly Survey”
## 
## Contains 96 rows of 38 variables:
## Filtered by pid3 == "Democrat"
## 
## $newsint2: newsint2 (categorical)
## $track: Direction of country (categorical)
## $manningpenalty: manningpenalty (categorical)
## $imiss: Issue importance (multiple_response)
## $imissf: imissf (categorical)
## $obamaapp: obamaapp (categorical)
## $boap: Approval of Obama on issues (multiple_response)
## $congapp: congapp (categorical)
## $ideo5: ideo5 (categorical)
## $ideoobama: ideoobama (categorical)
## $saysobama: saysobama (categorical)
## $likeobama: likeobama (categorical)
## $manningknowledge: manningknowledge (categorical)
## $manningfavorability: manningfavorability (categorical)
## $manningguilt: manningguilt (categorical)
## $snowdenfav: Favorability of Edward Snowden (categorical)
## $snowdenleakapp: Approval of Snowden's Leak (categorical)
## $snowdenpros: Support for Prosecution of Snowden (categorical)
## $snowdenpenalty: Penalty for Snowden (categorical)
## $perc_skipped: perc_skipped (numeric)
## $birthyr: birthyr (numeric)
## $gender: gender (categorical)
## $pid3: pid3 (categorical)
## $pid7: pid7 (categorical)
## $pid7others: pid7others (categorical)
## $race: race (categorical)
## $educ: educ (categorical)
## $marstat: marstat (categorical)
## $phone: phone (categorical)
## $faminc: faminc (categorical)
## $region: region (numeric)
## $state: state (categorical)
## $weight: weight (numeric)
## $votereg_new: votereg_new (numeric)
## $is_voter: is_voter (numeric)
## $votereg_old: votereg_old (numeric)
## $votereg: votereg (numeric)
## $age: age (numeric)
round(crtabs(mean(track) ~ educ + gender, data=dems), 2)
##                       gender
## educ                   Male Female
##   No HS                   0      3
##   High school graduate    7     17
##   Some college            8     16
##   2-year                  1      6
##   4-year                  9     19
##   Post-grad               5      5

When you extract a variable from a filtered dataset, it too is filtered. So

table(dems$educ)
## educ
##                No HS High school graduate         Some college               2-year               4-year 
##             5.990781            23.817206            19.000165             4.962549            18.238301 
##            Post-grad 
##             6.278870

is the same as

table(ds$educ[ds$pid3 == "Democrat",])
## educ
##                No HS High school graduate         Some college               2-year               4-year 
##             5.990781            23.817206            19.000165             4.962549            18.238301 
##            Post-grad 
##             6.278870

As an aside, if you prefer using the subset function, that works just the same as the [ extract method on datasets:

identical(subset(ds, ds$pid3 == "Democrat"), dems)
## [1] TRUE

Filter entities

In the web application, you can save filter definitions with names for easy reuse. You can also share these filter definitions with other viewers of the dataset.

To do so, we work with the dataset’s filter catalog. To start, our filter catalog will be empty:

## data frame with 0 columns and 0 rows

Create a filter by assigning a Crunch logical expression to the catalog by the name we want to give it, using $ or [[:

filters(ds)[["Young males"]] <- ds$gender == "Male" & ds$age < 30
filters(ds)[["Young males"]]
## Crunch filter “Young males”
## Expression: gender == "Male" & age < 30

This new filter now appears in our filter catalog.

##          name                               id is_public
## 1 Young males 257567b10b4d46e19c703cf2de0b7cd4     FALSE

You could also have made the filter with the newFilter function:

f <- newFilter("Young males", ds$gender == "Male" & ds$age < 30)

This filter is now available for you to use in the web application. If you want to make the filter available to all viewers of the dataset, make it “public”:

is.public(filters(ds)[["Young males"]]) <- TRUE
filters(ds)
##          name                               id is_public
## 1 Young males 257567b10b4d46e19c703cf2de0b7cd4      TRUE

You can also edit the filter expressions by assigning a new one in, like:

filters(ds)[["Young males"]] <- ds$gender == "Male" & ds$age < 35
filters(ds)[["Young males"]]
## Crunch filter “Young males”
## Expression: gender == "Male" & age < 35

Exclusion filters

One other application for filtering is the dataset exclusion filter. The exclusion allows you to suppress from view rows that match a certain condition. Exclusions are set on the dataset with a Crunch logical expression:

dim(ds)
## [1] 250  38
exclusion(ds) <- ds$perc_skipped > 15
exclusion(ds)
## Crunch logical expression: perc_skipped > 15
dim(ds)
## [1] 229  38

All viewers of the dataset will see the dataset as if those rows do not exist; however, as the editor of the dataset, you can remove the exclusion filter to see them if you need:

exclusion(ds) <- NULL
dim(ds)
## [1] 250  38

Alternative: dropRows

If you do know that you never want to see those rows again, you can permanently delete them with dropRows:

## Not run
ds <- dropRows(ds, ds$perc_skipped > 15)

Next: exporting data