Skip to content

feat: Add upsert and unique checking in local mode #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

feat: Add upsert and unique checking in local mode #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
133147f
Add initial upsert support in local mode
matthewmcneely Jun 10, 2025
b284c82
Add edge traversal limit option
matthewmcneely Jun 10, 2025
853564a
Update README
matthewmcneely Jun 10, 2025
c155a3c
Add some mutation logging at verbosity level 3
matthewmcneely Jun 10, 2025
e24da88
Harden the upsert logic
matthewmcneely Jun 10, 2025
a13ae57
Add/update tests
matthewmcneely Jun 10, 2025
b0ae279
Update README
matthewmcneely Jun 13, 2025
30dac40
Add config value
matthewmcneely Jun 13, 2025
aeceb52
Add support for upsert and unique mutation checks
matthewmcneely Jun 13, 2025
74c725f
Use UTC time in tests
matthewmcneely Jun 13, 2025
ea83d46
Handle unique mutations of slices of elements
matthewmcneely Jun 16, 2025
4865264
Alias the dgman unique error type
matthewmcneely Jun 16, 2025
f704edd
Handle slice-based upserts
matthewmcneely Jun 16, 2025
1d196a4
Remove environment setting in test
matthewmcneely Jun 17, 2025
9046eec
Update dgman pkg
matthewmcneely Jun 18, 2025
2375374
Unify client map and key from full defined options; add tests
matthewmcneely Jun 18, 2025
cb19877
Update README
matthewmcneely Jun 19, 2025
424eb98
Remove test env setting
matthewmcneely Jun 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 85 additions & 31 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,8 @@ Connects to a database stored locally on the filesystem. This mode doesn't requi
database server and is perfect for development, testing, or embedded applications. The directory
must exist before connecting.

File-based databases do not support concurrent access from separate processes.
File-based databases do not support concurrent access from separate processes. Further, there can
only be one file-based client per process.

```go
// Connect to a local file-based database
Expand All @@ -131,6 +132,8 @@ Connects to a Dgraph cluster. For more details on the Dgraph URI format, see the
client, err := mg.NewClient("dgraph://hostname:9080")
```

You can have multiple remote clients per process provided the URIs are distinct.

### Configuration Options

modusGraph provides several configuration options that can be passed to the `NewClient` function:
Expand All @@ -155,6 +158,15 @@ connections.
client, err := mg.NewClient(uri, mg.WithPoolSize(20))
```

#### WithMaxEdgeTraversal(int)

Sets the maximum number of edges to traverse when querying. The default is 10 edges.

```go
// Set max edge traversal to 20 edges
client, err := mg.NewClient(uri, mg.WithMaxEdgeTraversal(20))
```

#### WithLogger(logr.Logger)

Configures structured logging with custom verbosity levels. By default, logging is disabled.
Expand Down Expand Up @@ -187,6 +199,9 @@ Every struct that represents a node in your graph should include:
```go
type MyNode struct {
// Your fields here with appropriate tags
Name string `json:"name,omitempty" dgraph:"index=exact"`
Description string `json:"description,omitempty" dgraph:"index=term"`
CreatedAt time.Time `json:"createdAt,omitempty" dgraph:"index=day"`

// These fields are required for Dgraph integration
UID string `json:"uid,omitempty"`
Expand All @@ -198,38 +213,38 @@ type MyNode struct {

modusGraph uses struct tags to define how each field should be handled in the graph database:

| Directive | Option | Description | Example |
| ----------- | -------- | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| **index** | exact | Creates an exact-match index for string fields | Name string `json:"name" dgraph:"index=exact"` |
| | hash | Creates a hash index (same as exact) | Code string `json:"code" dgraph:"index=hash"` |
| | term | Creates a term index for text search | Description string `json:"description" dgraph:"index=term"` |
| | fulltext | Creates a full-text search index | Content string `json:"content" dgraph:"index=fulltext"` |
| | int | Creates an index for integer fields | Age int `json:"age" dgraph:"index=int"` |
| | geo | Creates a geolocation index | Location `json:"location" dgraph:"index=geo"` |
| | day | Creates a day-based index for datetime fields | Created time.Time `json:"created" dgraph:"index=day"` |
| | year | Creates a year-based index for datetime fields | Birthday time.Time `json:"birthday" dgraph:"index=year"` |
| | month | Creates a month-based index for datetime fields | Hired time.Time `json:"hired" dgraph:"index=month"` |
| | hour | Creates an hour-based index for datetime fields | Login time.Time `json:"login" dgraph:"index=hour"` |
| | hnsw | Creates a vector similarity index | Vector \*dg.VectorFloat32 `json:"vector" dgraph:"index=hnsw(metric:cosine)"` |
| **type** | geo | Specifies a geolocation field | Location `json:"location" dgraph:"type=geo"` |
| | datetime | Specifies a datetime field | CreatedAt time.Time `json:"createdAt" dgraph:"type=datetime"` |
| | int | Specifies an integer field | Count int `json:"count" dgraph:"type=int"` |
| | float | Specifies a floating-point field | Price float64 `json:"price" dgraph:"type=float"` |
| | bool | Specifies a boolean field | Active bool `json:"active" dgraph:"type=bool"` |
| | password | Specifies a password field (stored securely) | Password string `json:"password" dgraph:"type=password"` |
| **count** | | Creates a count index | Visits int `json:"visits" dgraph:"count"` |
| **unique** | | Enforces uniqueness for the field (remote Dgraph only) | Email string `json:"email" dgraph:"index=hash unique"` |
| **upsert** | | Allows a field to be used in upsert operations (remote Dgraph only) | UserID string `json:"userID" dgraph:"index=hash upsert"` |
| **reverse** | | Creates a bidirectional edge | Friends []\*Person `json:"friends" dgraph:"reverse"` |
| **lang** | | Enables multi-language support for the field | Description string `json:"description" dgraph:"lang"` |
| Directive | Option | Description | Example |
| ----------- | -------- | ----------------------------------------------- | ------------------------------------------------------------------------------------ |
| **index** | exact | Creates an exact-match index for string fields | Name string `json:"name" dgraph:"index=exact"` |
| | hash | Creates a hash index (same as exact) | Code string `json:"code" dgraph:"index=hash"` |
| | term | Creates a term index for text search | Description string `json:"description" dgraph:"index=term"` |
| | fulltext | Creates a full-text search index | Content string `json:"content" dgraph:"index=fulltext"` |
| | int | Creates an index for integer fields | Age int `json:"age" dgraph:"index=int"` |
| | geo | Creates a geolocation index | Location `json:"location" dgraph:"index=geo"` |
| | day | Creates a day-based index for datetime fields | Created time.Time `json:"created" dgraph:"index=day"` |
| | year | Creates a year-based index for datetime fields | Birthday time.Time `json:"birthday" dgraph:"index=year"` |
| | month | Creates a month-based index for datetime fields | Hired time.Time `json:"hired" dgraph:"index=month"` |
| | hour | Creates an hour-based index for datetime fields | Login time.Time `json:"login" dgraph:"index=hour"` |
| | hnsw | Creates a vector similarity index | Vector \*dg.VectorFloat32 `json:"vector" dgraph:"index=hnsw(metric:cosine)"` |
| **type** | geo | Specifies a geolocation field | Location `json:"location" dgraph:"type=geo"` |
| | datetime | Specifies a datetime field | CreatedAt time.Time `json:"createdAt" dgraph:"type=datetime"` |
| | int | Specifies an integer field | Count int `json:"count" dgraph:"type=int"` |
| | float | Specifies a floating-point field | Price float64 `json:"price" dgraph:"type=float"` |
| | bool | Specifies a boolean field | Active bool `json:"active" dgraph:"type=bool"` |
| | password | Specifies a password field (stored securely) | Password string `json:"password" dgraph:"type=password"` |
| **count** | | Creates a count index | Visits int `json:"visits" dgraph:"count"` |
| **unique** | | Enforces uniqueness for the field | Email string `json:"email" dgraph:"index=hash unique"` |
| **upsert** | | Allows a field to be used in upsert operations | UserID string `json:"userID" dgraph:"index=hash upsert"` |
| **reverse** | | Creates a bidirectional edge | Friends []\*Person `json:"friends" dgraph:"reverse"` |
| **lang** | | Enables multi-language support for the field | Description string `json:"description" dgraph:"lang"` |

### Relationships

Relationships between nodes are defined using struct pointers or slices of struct pointers:

```go
type Person struct {
Name string `json:"name,omitempty" dgraph:"index=exact"`
Name string `json:"name,omitempty" dgraph:"index=exact upsert"`
Friends []*Person `json:"friends,omitempty"`
Manager *Person `json:"manager,omitempty"`

Expand Down Expand Up @@ -268,6 +283,10 @@ Advanced querying is required to properly bind reverse edges in query results. S

modusGraph provides a simple API for common database operations.

Note that in local-mode, unique fields are limited to the top-level object. Fields marked as unique
in embedded or lists of embedded objects that have `unique` tags will not be checked for uniqueness
when the top-level object is inserted.

### Inserting Data

To insert a new node into the database:
Expand All @@ -292,9 +311,40 @@ if err != nil {
fmt.Println("Created user with UID:", user.UID)
```

### Upserting Data

modusGraph provides a simple API for upserting data into the database.

Note that in local-mode, upserts are only supported on the top-level object. Fields in embedded or
lists of embedded objects that have `upsert` tags will be ignored when the top-level object is
upserted.

```go
ctx := context.Background()

user := User{
Name: "John Doe", // this field has the `upsert` tag
Email: "[email protected]",
Role: "Admin",
}

// Upsert the user into the database
// If "John Doe" does not exist, it will be created
// If "John Doe" exists, it will be updated
err := client.Upsert(ctx, &user)
if err != nil {
log.Fatalf("Failed to upsert user: %v", err)
}

```

### Updating Data

To update an existing node, first retrieve it, modify it, then save it back:
To update an existing node, first retrieve it, modify it, then save it back.

Note that in local-mode, unique update checks are only supported on the top-level object. Fields in
embedded or lists of embedded objects that have `unique` tags will not be checked for uniqueness
when the top-level object is updated.

```go
ctx := context.Background()
Expand Down Expand Up @@ -534,10 +584,14 @@ These operations are useful for testing or when you need to reset your database
modusGraph has a few limitations to be aware of:

- **Unique constraints in file-based mode**: Due to the intricacies of how Dgraph handles unique
fields and upserts in its core package, unique field checks and upsert operations are not
supported (yet) when using the local (file-based) mode. These operations work properly when using
a full Dgraph cluster, but the simplified file-based mode does not support the constraint
enforcement mechanisms required for uniqueness guarantees.
fields in its core package, when using file-based mode, unique field checks are only supported at
the top level object that is being in/upserted or updated. Embedded or lists of embedded objects
that have unique tags will NOT be checked for uniqueness when the top-level object is in/upserted
or updated.

- **Upsert operations**: Upsert operations are only supported on the top-level object. Fields in
embedded or lists of embedded objects that have upsert tags will be ignored when the top-level
object is upserted.

- **Schema evolution**: While modusGraph supports schema inference through tags, evolving an
existing schema with new fields requires careful consideration to avoid data inconsistencies.
Expand Down
2 changes: 2 additions & 0 deletions buf_server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ func (s *serverWrapper) Query(ctx context.Context, req *api.Request) (*api.Respo
s.engine.logger.V(2).Info("Query using namespace", "namespaceID", ns.ID())

if len(req.Mutations) > 0 {
s.engine.logger.V(3).Info("Mutating", "mutations", req.Mutations)

uids, err := ns.Mutate(ctx, req.Mutations)
if err != nil {
return nil, fmt.Errorf("engine mutation error: %w", err)
Expand Down
Loading