Skip to main content

Databases and SQL

"Database" is a catch-all term referring to traditional RDBMS as well as K/V stores, document databases, and other "NoSQL" storages. There are many external database systems as well as browser APIs like WebSQL and localStorage

This demo discusses general strategies and provides examples for a variety of database systems. The examples are merely intended to demonstrate very basic functionality.

Key-value stores, unstructured use of Document Databases, and other schema-less databases are covered in the NoSQL demo.

Structured Tables

Database tables are a common import and export target for spreadsheets. One common representation of a database table is an array of JS objects whose keys are column headers and whose values are the underlying data values. For example,

NameIndex
Barack Obama44
Donald Trump45
Joseph Biden46

is naturally represented as an array of objects

[
{ Name: "Barack Obama", Index: 44 },
{ Name: "Donald Trump", Index: 45 },
{ Name: "Joseph Biden", Index: 46 }
]

The sheet_to_json and json_to_sheet helper functions work with objects of similar shape, converting to and from worksheet objects. The corresponding worksheet would include a header row for the labels:

XXX|      A       |   B   |
---+--------------+-------+
1 | Name | Index |
2 | Barack Obama | 44 |
3 | Donald Trump | 45 |
3 | Joseph Biden | 46 |

Building Worksheets from Structured Tables

There are NodeJS connector libraries for many popular RDBMS systems. Libraries have facilities for connecting to a database, executing queries, and obtaining results as arrays of JS objects that can be passed to json_to_sheet. The main differences surround API shape and supported data types.

For example, better-sqlite3 is a connector library for SQLite. The result of a SELECT query is an array of objects suitable for json_to_sheet:

var aoo = db.prepare("SELECT * FROM 'Presidents' LIMIT 100000").all();
var worksheet = XLSX.utils.json_to_sheet(aoo);

Other databases will require post-processing. For example, MongoDB results include the Object ID (usually stored in the _id key). This can be removed before generating a worksheet:

const aoo = await db.collection('coll').find({}).toArray();
aoo.forEach((x) => delete x._id);
const ws = XLSX.utils.json_to_sheet(aoo);

Building Schemas from Worksheets

When a strict schema is needed, the sheet_to_json helper function generates arrays of JS objects that can be scanned to determine the column "types".

note

Document databases like MongoDB tend not to require schemas. Arrays of objects can be used directly without setting up a schema:

const aoo = XLSX.utils.sheet_to_json(ws);
await db.collection('coll').insertMany(aoo, { ordered: true });

This example will fetch https://sheetjs.com/data/cd.xls, scan the columns of the first worksheet to determine data types, and generate 6 PostgreSQL statements.

Explanation (click to show)

The relevant generate_sql function takes a worksheet name and a table name:

// define mapping between determined types and PostgreSQL types
const PG = { "n": "float8", "s": "text", "b": "boolean" };

function generate_sql(ws, wsname) {

// generate an array of objects from the data
const aoo = XLSX.utils.sheet_to_json(ws);

// types will map column headers to types, while hdr holds headers in order
const types = {}, hdr = [];

// loop across each row object
aoo.forEach(row =>
// Object.entries returns a row of [key, value] pairs. Loop across those
Object.entries(row).forEach(([k,v]) => {

// If this is first time seeing key, mark unknown and append header array
if(!types[k]) { types[k] = "?"; hdr.push(k); }

// skip null and undefined
if(v == null) return;

// check and resolve type
switch(typeof v) {
case "string": // strings are the broadest type
types[k] = "s"; break;
case "number": // if column is not string, number is the broadest type
if(types[k] != "s") types[k] = "n"; break;
case "boolean": // only mark boolean if column is unknown or boolean
if("?b".includes(types[k])) types[k] = "b"; break;
default: types[k] = "s"; break; // default to string type
}
})
);

// The final array consists of the CREATE TABLE query and a series of INSERTs
return [
// generate CREATE TABLE query and return batch
`CREATE TABLE \`${wsname}\` (${hdr.map(h =>
// column name must be wrapped in backticks
`\`${h}\` ${PG[types[h]]}`
).join(", ")});`
].concat(aoo.map(row => { // generate INSERT query for each row
// entries will be an array of [key, value] pairs for the data in the row
const entries = Object.entries(row);
// fields will hold the column names and values will hold the values
const fields = [], values = [];
// check each key/value pair in the row
entries.forEach(([k,v]) => {
// skip null / undefined
if(v == null) return;
// column name must be wrapped in backticks
fields.push(`\`${k}\``);
// when the field type is numeric, `true` -> 1 and `false` -> 0
if(types[k] == "n") values.push(typeof v == "boolean" ? (v ? 1 : 0) : v);
// otherwise,
else values.push(`'${v.toString().replaceAll("'", "''")}'`);
})
if(fields.length) return `INSERT INTO \`${wsname}\` (${fields.join(", ")}) VALUES (${values.join(", ")})`;
})).filter(x => x); // filter out skipped rows
}
Result
Loading...
Live Editor

DSV Interchange

Many databases offer utilities for reading and writing CSV, pipe-separated documents, and other simple data files. They enable workflows where the library generates CSV data for the database to process or where the library parses CSV files created by the database.

Worksheet to CSV

CSV data can be generated from worksheets using XLSX.utils.sheet_to_csv.

// starting from a worksheet object
const csv = XLSX.utils.sheet_to_json(ws);

// whole workbook conversion
const csv_arr = wb.SheetNames.map(n => XLSX.utils.sheet_to_json(wb.Sheets[n]));

CSV to Worksheet

XLSX.read can read strings with CSV data. It will generate single-sheet workbooks with worksheet name Sheet1.

Where supported, XLSX.readFile can read files.

// starting from a CSV string
const ws_str = XLSX.read(csv_str, {type: "string"}).Sheets.Sheet1;

// starting from a CSV binary string (e.g. `FileReader#readAsBinaryString`)
const ws_bstr = XLSX.read(csv_bstr, {type: "binary"}).Sheets.Sheet1;

// starting from a CSV file in NodeJS or Bun or Deno
const ws_file = XLSX.readFile("test.csv").Sheets.Sheet1;

Databases

SQLite

Most platforms offer a simple way to query SQLite databases.

The following example shows how to query for each table in an SQLite database, query for the data for each table, add each non-empty table to a workbook, and export as XLSX.

The Northwind database is available in SQLite form.

The better-sqlite3 module provides a very simple API for working with SQLite databases. Statement#all runs a prepared statement and returns an array of JS objects.

1) Install the dependencies:

npm i --save https://cdn.sheetjs.com/xlsx-latest/xlsx-latest.tgz better-sqlite3

2) Save the following to node.mjs:

node.mjs
/* Load SQLite3 connector library */
import Database from "better-sqlite3";

/* Load SheetJS library */
import * as XLSX from 'xlsx/xlsx.mjs';
import * as fs from 'fs';
XLSX.set_fs(fs);

/* Initialize database */
var db = Database("northwind.db");

/* Create new workbook */
var wb = XLSX.utils.book_new();

/* Get list of table names */
var sql = db.prepare("SELECT name FROM sqlite_master WHERE type='table'");
var result = sql.all();

/* Loop across each name */
result.forEach(function(row) {
/* Get first 100K rows */
var aoo = db.prepare("SELECT * FROM '" + row.name + "' LIMIT 100000").all();
if(aoo.length > 0) {
/* Create Worksheet from the row objects */
var ws = XLSX.utils.json_to_sheet(aoo, {dense: true});
/* Add to Workbook */
XLSX.utils.book_append_sheet(wb, ws, row.name);
}
});

/* Write File */
XLSX.writeFile(wb, "node.xlsx");

3) Run node node.mjs and open node.xlsx

WebSQL

danger

This information is included for legacy deployments. Web SQL is deprecated.

https://caniuse.com/sql-storage has up-to-date info on browser support. Firefox never supported Web SQL. Safari 13 dropped support. As of the time of writing, the current Chrome version (103) supports WebSQL.

WebSQL was a popular SQL-based in-browser database available on Chrome. In practice, it is powered by SQLite, and most simple SQLite-compatible queries work as-is in WebSQL.

The public demo http://sheetjs.com/sql generates a database from workbook.

Importing data from spreadsheets is straightforward using the generate_sql helper function from "Building Schemas":

const db = openDatabase('sheetql', '1.0', 'SheetJS WebSQL Test', 2097152);
const stmts = generate_sql(ws, wsname);
// NOTE: tx.executeSql and db.transaction use callbacks. This wraps in Promises
for(var i = 0; i < stmts.length; ++i) await new Promise((res, rej) => {
db.transaction(tx =>
tx.executeSql(stmts[i], [],
(tx, data) => res(data), // if the query is successful, return the data
(tx, err) => rej(err) // if the query fails, reject with the error
));
});

The result of a SQL SELECT statement is a SQLResultSet. The rows property is a SQLResultSetRowList. It is an "array-like" structure that has length and properties like 0, 1, etc. However, this is not a real Array object. A real Array can be created using Array.from:

const db = openDatabase('sheetql', '1.0', 'SheetJS WebSQL Test', 2097152);
db.readTransaction(tx =>
tx.executeSQL("SELECT * FROM DatabaseTable", [], (tx, data) => {
// data.rows is "array-like", so `Array.from` can make it a real array
const aoo = Array.from(data.rows);
const ws = XLSX.utils.json_to_sheet(aoo);
// ... it is recommended to perform an export here OR wrap in a Promise
})
);

The following demo generates a database with 5 fixed SQL statements. Queries can be changed in the Live Editor. The WebSQL database can be inspected in the "WebSQL" section of the "Application" Tab of Developer Tools:

WebSQL view in Developer Tools

Result
Loading...
Live Editor

LocalStorage and SessionStorage

The Storage API, encompassing localStorage and sessionStorage, describes simple key-value stores that only support string values and keys.

Arrays of objects can be stored using JSON.stringify using row index as key:

const aoo = XLSX.utils.sheet_to_json(ws);
for(var i = 0; i < aoo.length; ++i) localStorage.setItem(i, JSON.stringify(aoo[i]));

Recovering the array of objects is possible by using JSON.parse:

const aoo = [];
for(var i = 0; i < localStorage.length; ++i) aoo.push(JSON.parse(localStorage.getItem(i)));
const ws = XLSX.utils.json_to_sheet(aoo);

This example will fetch https://sheetjs.com/data/cd.xls, fill localStorage with rows, then generate a worksheet from the rows and write to a new file.

caution

This example is for illustration purposes. If array of objects is available, it is strongly recommended to convert that array to a worksheet directly.

Result
Loading...
Live Editor

IndexedDB

localForage is a IndexedDB wrapper that presents an async Storage interface.

Arrays of objects can be stored using JSON.stringify using row index as key:

const aoo = XLSX.utils.sheet_to_json(ws);
for(var i = 0; i < aoo.length; ++i) await localForage.setItem(i, JSON.stringify(aoo[i]));

Recovering the array of objects is possible by using JSON.parse:

const aoo = [];
for(var i = 0; i < localForage.length; ++i) aoo.push(JSON.parse(await localForage.getItem(i)));
const wb = XLSX.utils.json_to_sheet(aoo);

Other SQL Databases

The generate_sql function from "Building Schemas from Worksheets" can be adapted to generate SQL statements for a variety of databases, including:

PostgreSQL

The pg connector library was tested against the generate_sql output as-is.

The rows property of a query result is an array of objects that plays nice with json_to_sheet:

const aoa = await connection.query(`SELECT * FROM DataTable`).rows;
const worksheet = XLSX.utils.json_to_sheet(aoa);

MySQL / MariaDB

The mysql2 connector library was tested. The differences are shown below, primarily stemming from the different quoting requirements and field types.

Differences (click to show)
// define mapping between determined types and MySQL types
const PG = { "n": "REAL", "s": "TEXT", "b": "TINYINT" };

function generate_sql(ws, wsname) {

// generate an array of objects from the data
const aoo = XLSX.utils.sheet_to_json(ws);

// types will map column headers to types, while hdr holds headers in order
const types = {}, hdr = [];

// loop across each row object
aoo.forEach(row =>
// Object.entries returns a row of [key, value] pairs. Loop across those
Object.entries(row).forEach(([k,v]) => {

// If this is first time seeing key, mark unknown and append header array
if(!types[k]) { types[k] = "?"; hdr.push(k); }

// skip null and undefined
if(v == null) return;

// check and resolve type
switch(typeof v) {
case "string": // strings are the broadest type
types[k] = "s"; break;
case "number": // if column is not string, number is the broadest type
if(types[k] != "s") types[k] = "n"; break;
case "boolean": // only mark boolean if column is unknown or boolean
if("?b".includes(types[k])) types[k] = "b"; break;
default: types[k] = "s"; break; // default to string type
}
})
);

// The final array consists of the CREATE TABLE query and a series of INSERTs
return [
// generate CREATE TABLE query and return batch
`CREATE TABLE ${wsname} (${hdr.map(h =>
`${h} ${PG[types[h]]}`
).join(", ")});`
].concat(aoo.map(row => { // generate INSERT query for each row
// entries will be an array of [key, value] pairs for the data in the row
const entries = Object.entries(row);
// fields will hold the column names and values will hold the values
const fields = [], values = [];
// check each key/value pair in the row
entries.forEach(([k,v]) => {
// skip null / undefined
if(v == null) return;
fields.push(`${k}`);
// when the field type is numeric, `true` -> 1 and `false` -> 0
if(types[k] == "n") values.push(typeof v == "boolean" ? (v ? 1 : 0) : v);
// otherwise,
else values.push(`"${v.toString().replaceAll('"', '""')}"`);
})
if(fields.length) return `INSERT INTO \`${wsname}\` (${fields.join(", ")}) VALUES (${values.join(", ")})`;
})).filter(x => x); // filter out skipped rows
}

The first property of a query result is an array of objects that plays nice with json_to_sheet:

const aoa = await connection.query(`SELECT * FROM DataTable`)[0];
const worksheet = XLSX.utils.json_to_sheet(aoa);

Query Builders

Query builders are designed to simplify query generation and normalize field types and other database minutiae.

Knex

The result of a SELECT statement is an array of objects:

const aoo = await connection.select("*").from("DataTable");
const worksheet = XLSX.utils.json_to_sheet(aoa);

Knex wraps primitive types when creating a table. generate_sql takes a knex connection object and uses the API:

Generating a Table (click to show)
// define mapping between determined types and Knex types
const PG = { "n": "float", "s": "text", "b": "boolean" };

async function generate_sql(knex, ws, wsname) {

// generate an array of objects from the data
const aoo = XLSX.utils.sheet_to_json(ws);

// types will map column headers to types, while hdr holds headers in order
const types = {}, hdr = [];

// loop across each row object
aoo.forEach(row =>
// Object.entries returns a row of [key, value] pairs. Loop across those
Object.entries(row).forEach(([k,v]) => {

// If this is first time seeing key, mark unknown and append header array
if(!types[k]) { types[k] = "?"; hdr.push(k); }

// skip null and undefined
if(v == null) return;

// check and resolve type
switch(typeof v) {
case "string": // strings are the broadest type
types[k] = "s"; break;
case "number": // if column is not string, number is the broadest type
if(types[k] != "s") types[k] = "n"; break;
case "boolean": // only mark boolean if column is unknown or boolean
if("?b".includes(types[k])) types[k] = "b"; break;
default: types[k] = "s"; break; // default to string type
}
})
);

await knex.schema.dropTableIfExists(wsname);
await knex.schema.createTable(wsname, (table) => { hdr.forEach(h => { table[PG[types[h]] || "text"](h); }); });
for(let i = 0; i < aoo.length; ++i) {
if(!aoo[i] || !Object.keys(aoo[i]).length) continue;
try { await knex.insert(aoo[i]).into(wsname); } catch(e) {}
}
return knex;
}

MongoDB Structured Collections

MongoDB is a popular document-oriented database engine.

It is straightforward to treat collections as worksheets. Each object maps to a row in the table.

The official NodeJS connector is mongodb on NPM.

Worksheets can be generated from collections by using Collection#find. A projection can suppress the object ID field:

/* generate a worksheet from a collection */
const aoo = await collection.find({}, {projection:{_id:0}}).toArray();
const ws = utils.json_to_sheet(aoo);

Collections can be populated with data from a worksheet using insertMany:

/* import data from a worksheet to a collection */
const aoo = XLSX.utils.sheet_to_json(ws);
await collection.insertMany(aoo, {ordered: true});
Complete Example (click to show)
caution

When this demo was last tested, the mongodb module did not work with Node 18. It was verified in Node 16.16.0.

1) Install the dependencies:

npm i --save https://cdn.sheetjs.com/xlsx-latest/xlsx-latest.tgz mongodb

2) Start a MongoDB server on localhost (follow official instructions)

3) Save the following to SheetJSMongoCRUD.mjs (the key step is highlighted):

SheetJSMongoCRUD.mjs
import { writeFile, set_fs, utils } from 'xlsx/xlsx.mjs';
import * as fs from 'fs'; set_fs(fs);
import { MongoClient } from 'mongodb';

const url = 'mongodb://localhost:27017/sheetjs';
const db_name = 'sheetjs';

(async() => {
/* Connect to mongodb server */
const client = await MongoClient.connect(url, { useUnifiedTopology: true });

/* Sample data table */
const db = client.db(db_name);
try { await db.collection('pres').drop(); } catch(e) {}
const pres = db.collection('pres');
await pres.insertMany([
{ name: "Barack Obama", idx: 44 },
{ name: "Donald Trump", idx: 45 },
{ name: "Joseph Biden", idx: 46 }
], {ordered: true});

/* Export database to XLSX */
const wb = utils.book_new();
const aoo = await pres.find({}, {projection:{_id:0}}).toArray();
const ws = utils.json_to_sheet(aoo);
utils.book_append_sheet(wb, ws, "Presidents");
writeFile(wb, "SheetJSMongoCRUD.xlsx");

/* Close connection */
client.close();
})();

4) Run node SheetJSMongoCRUD.mjs and open SheetJSMongoCRUD.xlsx