Skip to content

DX-1941/DX-1942: add docs for search migrator #476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -1464,7 +1464,12 @@
}
]
},

{
"group": "Tools",
"pages": [
"search/tools/databasemigrator"
]
},
{
"group": "Help",
"pages": ["search/help/faq"]
Expand Down
96 changes: 96 additions & 0 deletions search/tools/databasemigrator.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
title: Database Migrator
description: a CLI tool to migrate your data to Upstash Search
---

## Introduction

This tool helps you to migrate your data from other service providers (e.g. Algolia, Meilisearch)
to your Upstash Index. This data will be stored in an upstash vector, feeding the search component
for you
### Migrate Using npx
The command below prompts you to provide credentials for the indexes that you want to do the migration between.

```sh
npx @upstash/search-migrator
```

### Using Flags

You can also provide your credentials and other information as command-line flags.
Here are some examples:

#### Algolia to Upstash

```sh
npx @upstash/search-migrator \
--upstash-url "YOUR_UPSTASH_URL" \
--upstash-token "YOUR_UPSTASH_TOKEN" \
--algolia-app-id "YOUR_ALGOLIA_APP_ID" \
--algolia-api-key "YOUR_ALGOLIA_WRITE_API_KEY"
```

#### Meilisearch to Upstash

```sh
npx @upstash/search-migrator \
--upstash-url "YOUR_UPSTASH_URL" \
--upstash-token "YOUR_UPSTASH_TOKEN" \
--meilisearch-host "YOUR_MEILISEARCH_HOST" \
--meilisearch-api-key "YOUR_MEILISEARCH_API_KEY"
```

## Obtaining Credentials

### Upstash

1. Go to your [Upstash Console](https://console.upstash.com/).
2. Select your Search index.
3. Under the **Details** section, you will find your `UPSTASH_SEARCH_REST_URL` and `UPSTASH_SEARCH_REST_TOKEN`.
- `--upstash-url` corresponds to `UPSTASH_SEARCH_REST_URL`.
- `--upstash-token` corresponds to `UPSTASH_SEARCH_REST_TOKEN`.

- You may want to check out
[@upstash/search-migrator](https://www.npmjs.com/package/@upstash/search-migrator)
to see how to find credentials for other service providers

## Migration Process

The migrator will:

1. **Connect** to your source database (Algolia or Meilisearch)
2. **Fetch** all documents from the specified index
3. **Transform** the data to match Upstash Search's format
4. **Upload** the documents to your Upstash Search index
5. **Verify** the migration by comparing document counts

### Data Transformation

The migrator automatically handles the transformation of your data:

- **Document IDs**: Preserved from the source
- **Content**: Mapped to Upstash Search's content field
- **Metadata**: Preserved as metadata in Upstash Search
- **Searchable fields**: All fields become searchable by default


<Note>
For free tier, 10000 requests can be made daily,
so a database with more than 10000 entries could
be interrupted.
</Note>

### Getting Help

If you encounter any issues during migration:

1. Check the error messages for specific details
2. Verify your credentials are correct
3. Ensure your source database is accessible
4. Contact support at [[email protected]](mailto:[email protected])

## Final Remarks
If you've come to this point without any issues, congratulations! you may resume your work
with upstash search, an advanced, developer frinedly search product.

For further insights, please visit [@upstash/search-migrator](https://www.npmjs.com/package/@upstash/search-migrator)