Chunk array of objects by their size in JSON
$ yarn add @shelf/array-chunk-by-size
Useful if you want to split large array into smaller, but limited by JSON size.
Each array chunk will be up to specified amount of bytes when stringified into JSON.
import {chunkArray} from '@shelf/array-chunk-by-size';
const bigArray = [{a: 1}, {b: 2}, {c: 3}];
const twoKilobytes = 2 * 1024;
const smallerArrays = chunkArray({input: bigArray, bytesSize: twoKilobytes});
// => [ [ ... ], [ ... ] ] and so onAlternatively, you might pass a custom size calculation function. For example, to chunk array by LLM tokens size:
import {chunkArray} from '@shelf/array-chunk-by-size';
import {encode} from 'gpt-3-encoder';
const bigArray = ['msg-1', 'msg-2'];
const gpt3MaxTokens = 4000;
const smallerArrays = chunkArray({
input: bigArray,
bytesSize: gpt3MaxTokens,
sizeCalcFunction: item => encode(item).length,
});$ git checkout master
$ yarn version
$ yarn publish
$ git push origin master --tagsMIT © Shelf
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent