Configuration

This module contains functionality related to the the configuration module for embedding.splitters.basic_markdown.

Configuration

BasicMarkdownSplitterConfiguration

Bases: SplitterConfiguration

Configuration for the BasicMarkdownSplitter. This class defines the parameters needed to split markdown documents into chunks with specific token sizes and overlaps.

Source code in src/embedding/splitters/basic_markdown/configuration.py
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class BasicMarkdownSplitterConfiguration(SplitterConfiguration):
    """
    Configuration for the BasicMarkdownSplitter. This class defines the parameters needed to split markdown documents into chunks
    with specific token sizes and overlaps.
    """

    chunk_overlap_in_tokens: int = Field(
        ..., description="The number of tokens that overlap between chunks."
    )
    chunk_size_in_tokens: int = Field(
        ..., description="The size of each chunk in tokens."
    )
    name: SplitterName = Field(
        SplitterName.BASIC_MARKDOWN, description="The name of the splitter."
    )