When an input to an answer extractor or universal classifier is received that is longer
than the maximum input length that a model can process in a single go, we will, unless chunking is disabled, automatically split that input up into smaller
chunks and will then process each chunk separately.
We use our own semchunk algorithm to chunk inputs in such a way that
they are unlikely to cut off in the middle of an important sentence or paragraph.
Although semchunk is a deterministic algorithm, it can still be difficult
to predict exactly how many chunks will be created for any given input, due to the fact that
the algorithm is designed to create chunks that are semantically meaningful as possible rather
than to create chunks of a fixed size.
The default chunk size is the maximum input length of the model less overhead, which
includes not only boilerplate tokens but also, if a model that takes a query as input is
being used, the number of tokens in the longest statement in that query. For every chunk
that is created, the number of tokens inputted into a model will increase by the number of
tokens in that chunk in addition to the number of boilerplate tokens that are added to the
input.
Additionally, if the chunks are being passed to a model that also takes a statement as
input, such as a universal classifier, that
statement will have to be added to each chunk, which will therefore increase the number of
tokens inputted into the model by the number of tokens in the statement multiplied by the
number of chunks. The use of a chunk overlap ratio will also increase the number of tokens
and oftentimes the number of chunks being inputted to a model.