> For the complete documentation index, see [llms.txt](https://snet-ambassadors.gitbook.io/singularitynet-archive/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://snet-ambassadors.gitbook.io/singularitynet-archive/development/llm-development/data-loading-and-preprocessing.md).

# Data Loading and Preprocessing

{% hint style="info" %}
This page is currently in draft
{% endhint %}

A generic approach may use an online Python interpreter such as [Google Colab](https://colab.research.google.com).

A code example may be found at this location -

{% embed url="<https://github.com/SingularityNET-Archive/LLM-Development/blob/main/Colab/Data_Loading_and_Preprocessing.ipynb>" %}

Here's what this code does:

1. We use the built-in json module to load the JSON data directly from the file, instead of using the `JSONLoader`.
2. We iterate over the root-level array data, and for each item (workgroup meeting object), we use the CharacterTextSplitter to split the string representation of the item into chunks. We extend the chunks list with the resulting chunks.
3. We convert each chunk in the chunks list into a Document object.
4. Finally, we iterate over the `docs` list and print the content of each document.

This solution assumes that your JSON data is a root-level array of workgroup meeting objects. If your JSON data has a different structure, you may need to modify the code accordingly.

Please note that this solution doesn't handle nested JSON structures or complex data types within the JSON objects.

If you need more advanced JSON parsing capabilities, you may want to consider using a dedicated JSON processing library like `jsonpath-ng` or `jq` directly.

### Sample code for loading JSON files into Langchain <a href="#sample-code-for-loading-json-files-into-langchain" id="sample-code-for-loading-json-files-into-langchain"></a>

Typically a strucured text source will be used such as JSON.

An example of how to load JSON files into Langcahin may be found at this location -

{% embed url="<https://github.com/SingularityNET-Archive/LLM-Development/blob/main/Colab/JSON_Loader.ipynb>" %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://snet-ambassadors.gitbook.io/singularitynet-archive/development/llm-development/data-loading-and-preprocessing.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.