Memory Management

When extracting and loading data with Tenup there are important memory considerations to keep in mind. Remember, whatever computer Tenup is run from acts as an intermediary between the original data source and 1010data. This means that your computer's memory (RAM) will be used as a staging area as data is pulled from the origin source of data and loaded into 1010data. Tenup provides several options to make sure your computer doesn't have issues during the extract and load process. This section will detail the options and best practices for managing local memory while using Tenup.

Setting the Number of Rows

When extracting data from a data source, Tenup uses the local memory of the computer it's running on to store the extracted data and prepare it to be loaded in 1010data. The -R option gives you control over how much data is requested from the data source of origin.

By default, the number of rows stored in local memory is set to 32768. This number is very small compared to the usual size of data tables handled by 1010data, but it is a safe starting point.

Before raising the limit, keep in mind that the number of rows aren't the only important factor in how much memory is taken up. The number of columns, and the width of those columns, are also factors in the amount of memory that will be used by the data.

Testing may be necessary to know the optimum size of the window for a given table size and origin database, but you can usually safely raise the window size to 2000000 rows without experiencing issues. This is done as follows:
Here is an example with a value provided for the [NUMBER_OF_ROWS] variable:
$ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -R 1500000

Note that the value provided must be a whole number (integer) and must not contained any punctuation such as commas or periods.

Setting the Upload Window Size

The "window size" in Tenup is the amount of data held in Tenup's buffer between each load task. This setting is controlled with the -W option. -W is intended to be used when -R is changed to be either very small or very large. If –R is set to be very small, then –W will prevent excessive round trip communications with 1010data. If –R is set to be very large, it will prevent tenup from sending too much data to 1010data and prevent errors from occurring.

The default value for -W is 32768.
$ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -R 1000000

It should be noted that it is possible to have -R and -W set to different values. However, 1010data strongly recommends that these values be the same. By default, both settings are set to 32768.

Setting the Maximum Column Size Allowed

In some databases there is no way to get the maximum display size of a column. The -m option sets the maximum number of display characters permissible in a single variable-width column when loading data. The default value is 1024. Here is the syntax:
And here is the same example with a value provided for -m
$ tenup64 -u [USERNAME] -p [PASSWORD] -C [CONNSTR] [PATH_TO_NEW_TABLE] [QUERY] -m 2048