dplyr Gap Analysis
What of dplyr’s verb set has a real AdvancedAnalyticsTools equivalent today, what doesn’t, and which gaps are sensible
to shim directly on an array versus better left to the SQL/query layer that already sits in front of
AdvancedAnalyticsCache. Companion to
Data Frames vs the Packed Array, which covers the conceptual mapping; this is the
practical punch list.
The shape to shim against
Every existing table-shaped tool in Jube.Service/Agent/AdvancedAnalyticsTools/DataPreparationAndResampling/
works on a named jagged table — double[][] data (one array per row) plus a parallel string[]
columnNames — not the flat, unnamed, row-major double[] that AdvancedAnalyticsCache/
AdvancedAnalyticsPacking.Pack/Unpack actually stores. SelectTableColumns, DropSparseRowsAndColumns
and FindDuplicateRows (FeatureEngineeringTools.cs, MissingValueTools.cs, DataQualityTools.cs) all take this
shape and share two guard helpers: DataPreparationGuard.Table(data, ...) (validates rectangularity, returns column
count) and DataPreparationGuard.Names(columnNames, columns, ...) (validates or defaults names to column_0,
column_1, …). A shim needs column identity to do anything dplyr-shaped at all — the flat packed array alone has
none — so every recommendation below targets this same double[][] + string[]
shape, returning FeatureTableResult(string[] ColumnNames, double[][] Table) where a full table comes back, so it
composes with SelectTableColumns and the rest of the family without a new result type per verb.
Verb by verb
| dplyr verb | Current coverage | Gap | Verdict |
|---|---|---|---|
select() |
Exact match: SelectTableColumns (FeatureEngineeringTools.cs:448) — keep named columns, in a given order |
None | Covered |
rename() |
None as a table operation | Renaming is metadata-only (relabel columnNames, touch no cell) |
Not needed as a separate tool: the caller already holds columnNames and can replace an entry directly |
relocate() |
Partial: SelectTableColumns with keep in the wanted order already reorders columns, but it can’t express “move X after Y, leave the rest as-is” without naming every column |
Reordering when you don’t want to also filter down to a subset | Not covered beyond what SelectTableColumns’s reordering already provides |
mutate() |
Partial, verb-shaped tools exist for specific transforms (ApplyColumnFunction, GenerateColumnPairFeatures, GenerateLagFeatures, EncodeCyclicalFeature, all in FeatureEngineeringTools.cs/ScalingTools.cs) |
No generic “compute a new column from an arbitrary expression over existing columns” — every existing tool is a fixed, named transform, not an expression evaluator | Not covered as a generic escape hatch: an arbitrary-expression evaluator over the array is a different kind of tool, closer to the rule DSL than a data-prep tool |
filter() |
Partial: several tools identify rows (FindDuplicateRows returns indices to drop, DropSparseRowsAndColumns drops by a missingness rule, sampling tools return an index subset) but none takes an arbitrary boolean predicate and returns the compacted table |
A generic “keep these row indices, return the smaller table” step — every existing tool stops at indices and leaves recombination to the caller | The single highest-leverage gap: every row-selecting tool stops at indices and leaves recombination into a smaller table to the caller |
arrange() |
Partial: SortValuesWithOriginalIndices/SortValuesByKeys (NumericsAndOptimisation/CombinatoricsAndUtilityTools.cs:447,463) sort one vector, optionally by a single parallel key vector |
No whole-table, multi-key, stable sort — you’d have to sort one column and manually permute every other array yourself | No whole-table, multi-key sort exists; sorting one column and permuting every other array by hand is left entirely to the caller |
distinct() |
Partial: FindDuplicateRows (DataQualityTools.cs:66) already computes exactly this — it returns the row indices to keep if you drop repeats — but it stops at reporting, no tool actually removes them |
Nothing missing conceptually, just the last step | Nothing missing conceptually beyond the final step: FindDuplicateRows already identifies which rows to keep, but no tool removes the duplicates itself |
slice() / head() / tail() |
None | Row-range subsetting by position | Not covered as a distinct tool; row-range subsetting by position has no dedicated tool |
group_by() %>% summarise() |
Partial: SummariseByGroup (DescriptiveStatistics/TableAndTransformTools.cs:390) groups one numeric column by one label array, returning a fixed set of stats (count, mean, median, sd, min, max, sum) — not a user-chosen aggregation, not multiple simultaneous group/value columns |
Multi-column group-by, a caller-chosen aggregation per column, more than one grouping key | SummariseByGroup covers a single key and a single value column with a fixed statistic set; a full relational group-by (multiple keys, a caller-chosen aggregation) is a genuinely different and larger thing than this single-key, single-value grouped summary |
bind_rows() / bind_cols() |
None | Concatenating two already-shaped tables | Combining two independently-produced tables is ordinarily something the SQL query that produced them should already have done before packing |
left_join() / inner_join() / anti_join() / semi_join() |
None anywhere in AdvancedAnalyticsTools (checked the whole tree, not just this group) |
Key-based row matching across two tables | Not possible on the array: a join needs a key concept the packed array has no notion of at all (rows are bare positions, not identified by any column), and matching keys across two independently-produced tables is exactly what the SQL layer upstream of AdvancedAnalyticsCache already does before anything is packed, as already noted in DataFramesVsAccordArrays.md |
pivot_longer() / pivot_wider() |
None | Reshaping between wide and long form | Same call as joins: reshape before packing, at the query layer, where column identity still exists in a form worth reshaping. Not sensible on a positional array |