Columns

Columns are the heart of the VisiData computation engine.

Each column can calculate a value from a row object; and it might also be able to put a different value into the row object (for a later calculate to re-derive).

A Column subclass can override calcValue and putValue to define its fundamental interaction with the row object. This is often the only thing a Column subclass has to do.

calcValue and putValue should generally not be called by application code. Instead, apps and plugins should call getValue and setValue, which provide appropriate layers of caching.

class visidata.Column(name=None, *, type=<function anytype>, cache=False, **kwargs)[source]

Base class for all column types.

  • name: name of this column.

  • type: anytype str int float date or other type-like conversion function.

  • cache: cache behavior

    • False (default): getValue never caches; calcValue is always called.

    • True: getValue maintains a cache of options.col_cache_size.

    • "async": getValue launches thread for every uncached result, returns invalid value until cache entry available.

  • width: == 0 if hidden, None if auto-compute next time.

  • height: max height, None/0 to auto-compute for each row.

  • fmtstr: format string as applied by column type.

  • getter: default calcValue calls getter(col, row).

  • setter: default putValue calls setter(col, row, val).

  • kwargs: other attributes to be set on this column.

Column.name

Name of this column.

Column.type

Type of this column.

Column.width

Width of this column in characters. 0 or negative means hidden. None means not-yet-autocomputed.

Column.visibleWidth

Width of column as is displayed in terminal

New in version 2.1.

Column.fmtstr

Format string to use to display this column.

Column.hidden

Return True if width of this column is 0 or negative.

visidata.Column.calcValue(self, row)

Calculate and return value for row in this column.

visidata.Column.putValue(self, row, val)

Change value for row in this column to val immediately. Does not check the type. Overridable; by default calls .setter(row, val).

visidata.Column.getValue(self, row)

Return value for row in this column, calculating if not cached.

visidata.Column.getTypedValue(self, row)

Return the properly-typed value for the given row at this column, or a TypedWrapper object in case of null or error.

visidata.Column.getDisplayValue(self, row)

Return string displayed in this column for given row.

visidata.Column.formatValue(self, typedval, width=None)

Return displayable string of typedval according to Column.fmtstr.

visidata.Column.getValueRows(self, rows)

Generate (value, row) for each row in rows at this column, excluding null and error values.

visidata.Column.getValues(self, rows)

Generate value for each row in rows at this column, excluding null and error values.

visidata.Column.setValue(self, row, val, setModified=True)

Change value for row in this column to val. Call putValue immediately if not a deferred column (added to deferred parent at load-time); otherwise cache until later putChanges. Caller must add undo function.

visidata.Column.setValues(self, rows, *values)

Set values in this column for rows to values, recycling values as needed to fill rows.

visidata.Column.setValuesTyped(self, rows, *values)

Set values on this column for rows to values, coerced to column type, recycling values as needed to fill rows. Abort on type exception.

visidata.Column.setValuesFromExpr(self, rows, expr)

Set values in this column for rows to the result of the Python expression expr applied to each row.

visidata.BaseSheet.evalExpr(self, expr, **kwargs)

Evaluate Python expression expr in the context of kwargs (may vary by sheet type).

visidata.Column.recalc(self, sheet=None)

Reset column cache, attach column to sheet, and reify column name.

visidata.TableSheet.recalc(self)

Clear caches and set the sheet attribute on all columns.

visidata.Column.isError(col, row)

Return True if the computed or typed value for row in this column is an error.

  • calcValue may be arbitrarily expensive or even asynchronous, so once the value is calculated, it is cached until Column.recalc() is called.

  • putValue may modify the source data directly (for instance, if the row object represents a row in a database table). VisiData will never modify source data without an explicit save command. So applications (and all other code) must call setValue to change the value of any cell.

  • delete-cell actually just calls setValue with None.

Columns are added to Sheets with addColumn.

Example

class SpecialColumn(Column):
    'Column for special fields on special rows.'
    def calcValue(self, row):
        return row.special()[self.expr]

    def putValue(self, row, val):
        row.special()[self.expr] = val

c = SpecialColumn('field1', expr='field1_key')
sheet.addColumn(c)

Column Subclasses

class visidata.ItemColumn(name=None, expr=None, **kwargs)[source]

Column using getitem/setitem with expr.

class visidata.AttrColumn(name=None, expr=None, **kwargs)[source]

Column using getattr/setattr with attr.

class visidata.ExprColumn(name, expr=None, **kwargs)[source]

Column using expr to derive the value from each row.

class visidata.SettableColumn(*args, **kwargs)[source]

Column using rowid to store and retrieve values internally.

class visidata.SubColumnFunc(name='', origcol=None, expr=None, subfunc=<function getitemdef>, **kwargs)[source]

Column compositor; preprocess row with subfunc*(row, *expr) before passing to origcol.getValue and origcol.setValue.

Key Columns

visidata.TableSheet.setKeys(self, cols)

Make all cols into key columns.

visidata.TableSheet.unsetKeys(self, cols)

Make all cols non-key columns.

visidata.TableSheet.rowkey(self, row)

Return tuple of the key for row.

visidata.TableSheet.keystr(self, row)

Return string of the key for row.

TableSheet.keyCols

List of visible key columns.

TableSheet.nonKeyVisibleCols

List of visible non-key columns.

Types

The value returned by getValue could be many different things:

  • a string

  • numerically typed

  • a list or dict

  • None

  • a null value (according to options.null_value)

  • an Exception (error getting)

  • a Thread (async pending)

  • any python object

This value may need to be parsed and/or converted to a consistent type. So, every column has a type attribute, which affects how it is parsed, displayed, grouped, sorted, and more.

The default column type is anytype, which lets the underlying value pass through unaltered. This is the only type for which Column.getTypedValue can return arbitrary types.

The classic VisiData column types are:

type

description

numeric

command

keystrokes

anytype

pass-through

type-anytype

z~

str

string

type-str

~

date

date/time

Y

type-date

@

int

integer

Y

type-int

#

float

decimal

Y

type-float

%

currency

decimal with units

Y

type-currency

$

vlen

sequence length

Y

type-vlen

z#

The default keybindings for setting types are all on the shifted top left keys on a US keyboard.

User-defined Types

Fundamentally, a type is a function which takes the underlying value and returns an object of a specific type. This function should accept a string and do a reasonable conversion, like Python int and float do. And like those builtin types, this function should produce a reasonable baseline zero (arithmetic identity) when passed no parameters or None.

Computations should generally call getTypedValue, so that the values being used are consistently typed.

If the underlying value is None, the result will be a TypedWrapper, which provides the baseline zero value for purposes of comparison, but a stringified version of the underlying value for display. For a calcValue which raises or returns an Exception, getTypedValue will return a TypedExceptionWrapper with similar behavior.

In the following, TYPE is the type (like int, date, etc), and typedval is an instance of that type.

A VisiData type function or constructor must have certain properties:

  • TYPE() must return a reasonable default value for TYPE.

  • TYPE(typedval) must return an exact copy of typedval.

  • TYPE(str) must convert from a reasonable string representation into TYPE.

  • TYPE.__name__ must be set to the official name of the type function or constructor.

Objects returned by TYPE(...) must be:

  • comparable (for sorting)

  • hashable

  • formattable

  • roundable (if numeric, for binning)

  • idempotent (TYPE(TYPE(v)) == TYPE(v))

Types API

class visidata.vd.addType(typetype=None, icon=None, fmtstr='', formatter=<bound method numericFormatter of <visidata.vdobj.VisiData object>>, key='', name=None)

Add type to type map.

  • typetype: actual type class TYPE above

  • icon: unicode character in column header

  • fmtstr: format string to use if fmtstr not given

  • formatter: formatting function to call as formatter(fmtstr, typedvalue)

visidata.vd.isNumeric(col)

New in version 2.1.

visidata.isNumeric(col)[source]

Deprecated since version 2.1: Use vd.isNumeric instead.

Examples

::

# Add an ip_address type. vd.addType(ipaddress.ip_address, icon=’:’, formatter=lambda fmt,ip: str(ip))

TableSheet.addCommand(None, ‘type-ipaddr’, ‘cursorCol.type=ipaddress.ip_address’, ‘set type of current column to IP address’

Nulls

VisiData has a crude concept of null values. These interact with:

  1. aggregators: the denominator counts only non-null values

  2. the Describe Sheet: only null values count in the nulls column

  3. the fill-nulls command

  4. the melt command only keeps non-null values

  5. the prev-null and next-null commands (z< and z>)

  • The null values are Python None and options.null_value if set.

Note

options.null_value can be used to specify a null value in addition to Python None. The CLI can only set this option to a string like '' (empty string) or 'NA'. The option can be set to a different type in .visidatarc or other code, though:

options.null_value = 0.0
options.null_value = date("1980-01-01")
visidata.BaseSheet.isNullFunc(sheet)

Return func(value) which returns whether or not value is null.

There is no direct isNull function, because the possible null values can change at runtime via the above option, and getting an option value is very expensive to do in a bulk operation.

Aggregators

Aggregators allow you to gather the rows within a single column, and interpret them using descriptive statistics. VisiData comes pre-loaded with a default set like mean, stdev, and sum.

visidata.TableSheet.addAggregators(sheet, cols, aggrnames)

Add each aggregator in list of aggrnames to each of cols.

visidata.vd.aggregator(name, funcValues, helpstr='', *args, type=None)

Define simple aggregator name that calls funcValues(values, *args) to aggregate values. Use type to force the default type of the aggregated column.

The type parameter is optional. It allows you to define the default type of the aggregated column.

Example

Add aggregator for numpy’s internal rate of return module:

import numpy as np
vd.aggregator('irr', np.irr, type=float)