Added OSM road detection feature for consumer selection and simulatio… #138

CelinaKellinghaus · 2025-09-10T08:27:20Z

This PR adds the ability to display main roads from OpenStreetMap on the maps used in both
the consumer selection and simulation results pages.

Features:

Fetch roads from Overpass API via a new Django endpoint.
Display roads on the Leaflet map.
Roads are automatically loaded based on the current map view.
Existing consumers, poles, and links are not affected.
Road ID is shown when clicking on it.
Possible future enhancement: Roads layer can optionally be toggled.

Closes #50

paulapreuss

This is looking really nice already 🚀 Congratulations on your first feature :) I left some comments on some things that could be structured a bit differently, let me know what you think.

paulapreuss · 2025-09-11T10:54:29Z

offgridplanner/optimization/osm_roads.py

I don't think it's really necessary to create a new file for this code, I would rather integrate it within identify_consumers_on_map.py, rename that file and have it all live within one file instead of multiple very specific ones. What do you think about renaming identify_consumers_on_map.py to osm_utils.py?

paulapreuss · 2025-09-11T10:56:08Z

offgridplanner/optimization/urls.py

@@ -1,5 +1,6 @@
 from django.urls import path

+from . import views


All functions defined in views are already being imported with the line below, so you don't need this

offgridplanner/optimization/urls.py

paulapreuss · 2025-09-11T11:04:40Z

offgridplanner/static/js/integrate_map.js

 initializeMap();

+map.on("moveend", () => {
+  loadAndShowOSMRoads();


I am having the issue that if I start looking around the map and exploring (in the case that I was creating a new project, for instance) at some point I run into requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://overpass-api.de/api/interpreter. I think to avoid this we should not display the roads on map drag but do it like we do with the buildings, where we only load/display the roads once a bbox is selected for the consumers.

paulapreuss · 2025-09-11T11:07:30Z

offgridplanner/static/js/integrate_map.js

+  osmRoadsLayer.clearLayers();
+
+  const geojsonLayer = L.geoJSON(geojson, {
+    style: () => ({ color: "#cc99ff", weight: 2, opacity: 0.8 }),


Can we add the roads symbol (pink line) to the map legend aswell?

paulapreuss · 2025-09-25T06:23:57Z

I think it might be a good idea to save the roads data in the database after fetching it once for a project, that way it can just be displayed when loading the consumer selection step or the results page for an existing project without having to worry about the bounding box. The roads data should just be updated when a user creates a new bounding box/consumer selection.

For the database, you can use the existing BaseJsonData model as a base class. Some things to look at when starting to work with the database:

Essentially, you will need to create a new model Roads in the database. When the roads data is retrieved for a project, you create an instance of the model and save the data to it (or, alternatively, you get the existing instance and update the data, if it already exists - hint: look at the get_or_create method). You can have a look at how the saving to the database and retrieval for the map is currently done for the nodes in consumer_to_db and db_nodes_to_js, for instance.

paulapreuss · 2025-10-21T10:13:55Z

offgridplanner/static/js/backend_communication.js

+
+            if (res.executed) {
+                responseMsg.innerHTML = "";
+                Array.prototype.push.apply(road_elements, res.new_roads);


This line is what's causing your behavior - you are appending the new roads to the road_elements array, so you end up with an array that has both the old and new roads being saved to your database (the saving behavior seems to work fine, it just looks confusing because the old roads are not getting removed). Instead of pushing the new roads to the existing array, you should just overwrite it.

paulapreuss · 2025-10-21T10:15:36Z

offgridplanner/optimization/views.py

+            Roads.objects.filter(project=project).delete()
+            return JsonResponse({"message": "No data provided"}, status=200)
+
+        df = pd.DataFrame.from_records(road_elements)
+        if df.empty:
+            Roads.objects.filter(project=project).delete()
+            return JsonResponse({"message": "No valid data"}, status=200)
+
+        df = df.drop_duplicates(subset=["road_id"], keep="first")
+        required_columns = ["road_id", "coordinates", "how_added", "road_type"]
+        df = df[required_columns]
+        df["how_added"] = df["how_added"].fillna("automatic")
+        df["road_type"] = df["road_type"].fillna("osm")
+
+        Roads.objects.filter(project=project).delete()


Here it seems like you are deleting the existing instance no matter what happens, so no need to do it on each of the conditions :)

paulapreuss · 2025-10-21T10:19:02Z

offgridplanner/optimization/views.py

+
+        Roads.objects.filter(project=project).delete()
+
+        Roads.objects.create(project=project, data=df.to_json(orient="records"))


...but in general I would overwrite the existing object instead of creating a new one with

roads, created = Roads.objects.get_or_create(project=project) # --> this is a very useful function because it will get you a roads instance regardless of if it already exists or has to be created first roads.data = df.to_json(orient="records") roads.save()

paulapreuss · 2025-12-15T18:07:39Z

@CelinaKellinghaus please push your latest local changes and also the migration file, then I will take care of rebasing to solve the merge conflicts. When I do that I will have to force push the branch, so you should have no unpushed changes to avoid conflicts or lost code.

paulapreuss

Unfortunately, this PR currently contains reversals for most of the changes that have happened in the repository since the branch for the feature was created. I'm not sure what went wrong with rebasing or if this is just not the most current version? In any case, all the changes that are unrelated to the roads feature need to be taken out of the PR.

paulapreuss · 2025-12-17T09:30:22Z

offgridplanner/optimization/views.py

-    if isinstance(markers_only, str):
-        markers_only = True if markers_only == "true" else False  # noqa:SIM210


Is there a reason you deleted this code?

paulapreuss · 2025-12-17T09:32:06Z

offgridplanner/optimization/views.py

-            df = df.fillna("null")
+            df["latitude"] = df["latitude"].astype(float)
+            df["longitude"] = df["longitude"].astype(float)
+            df["shs_options"] = df["shs_options"].fillna(0)
+            df["custom_specification"] = df["custom_specification"].fillna("")
+            df["shs_options"] = df["shs_options"].astype(int)
+            df["is_connected"] = df["is_connected"].astype(bool)
+            nodes_list = df.to_dict("records")
+            is_load_center = True


I think you might have accidentally brought some deleted code back into this PR when doing the rebase, I see no reason why there should be changes to db_nodes_to_js, as it isn't really relevant to the roads functionality, right?

paulapreuss · 2025-12-17T09:33:04Z

offgridplanner/optimization/views.py

-        df, msg = check_imported_consumer_data(df, proj_id)
+        df, msg = check_imported_consumer_data(df)


I"m assuming these are also changes that got undone while rebasing

paulapreuss · 2025-12-17T09:37:24Z

offgridplanner/optimization/views.py

-    sim_res = data.get("results", {})
-    grid_processor = GridProcessor(proj_id=proj_id, results_json=sim_res.get("grid"))
+    results = data.get("results", {})
+    grid_processor = GridProcessor(proj_id=proj_id, results_json=results.get("grid"))
    grid_processor.grid_results_to_db()
    supply_processor = SupplyProcessor(
-        proj_id=proj_id, results_json=sim_res.get("supply")
+        proj_id=proj_id, results_json=results.get("supply")
    )
-    supply_processor.process_supply_optimization_results()
    supply_processor.supply_results_to_db()
-    # Process shared results (after both grid and supply have been processed)
-    results = Results.objects.get(simulation__project__id=proj_id)
-    results.lcoe_share_supply = (
-        (results.epc_total - results.cost_grid) / results.epc_total * 100
-    )
-    results.lcoe_share_grid = 100 - results.lcoe_share_supply
-    assets = ["grid", "diesel_genset", "inverter", "rectifier", "battery", "pv"]
-    results.upfront_invest_total = sum(
-        [getattr(results, f"upfront_invest_{key}") for key in assets]
-    )
-    results.save()
+


This is also causing your simulation to fail, since you accidentally deleted a lot of the results processing

paulapreuss · 2025-12-17T09:38:14Z

offgridplanner/optimization/views.py

+    project = get_object_or_404(Project, id=proj_id)
+    opts = project.options
+    res = project.simulation.results
+    df = pd.Series(model_to_dict(res))
+    infeasible = bool(df["infeasible"]) if "infeasible" in df else False
+    if df.empty:
+        return JsonResponse({})
+    # TODO figure out this logic - I changed it so it would run through but it doesnt make so much sense to me
+    # if df['lcoe'] is None and opts.do_es_design_optimization is True:
+    #     return JsonResponse({})
+    # elif df['n_poles']is None and opts.do_grid_optimization is True:
+    #     return JsonResponse({})
+    if opts.do_grid_optimization is True:
+        df["average_length_distribution_cable"] = (
+            df["length_distribution_cable"] / df["n_distribution_links"]
+        )
+        df["average_length_connection_cable"] = (
+            df["length_connection_cable"] / df["n_connection_links"]
+        )
+        df["gridLcoe"] = float(df["cost_grid"]) / float(df["epc_total"]) * 100
+    else:
+        df["average_length_distribution_cable"] = None
+        df["average_length_connection_cable"] = None
+        df["gridLcoe"] = 0
+    df[["time_grid_design", "time_energy_system_design"]] = df[
+        ["time_grid_design", "time_energy_system_design"]
+    ].fillna(0)
+    df["time"] = df["time_grid_design"] + df["time_energy_system_design"]
+    unit_dict = {
+        "n_poles": "",
+        "n_consumers": "",
+        "n_shs_consumers": "",
+        "length_distribution_cable": "m",
+        "average_length_distribution_cable": "m",
+        "length_connection_cable": "m",
+        "average_length_connection_cable": "m",
+        "cost_grid": "USD/a",
+        "lcoe": "",
+        "gridLcoe": "%",
+        "esLcoe": "%",
+        "res": "%",
+        "max_voltage_drop": "%",
+        "shortage_total": "%",
+        "surplus_rate": "%",
+        "time": "s",
+        "co2_savings": "t/a",
+        "total_annual_consumption": "kWh/a",
+        "average_annual_demand_per_consumer": "W",
+        "upfront_invest_grid": "USD",
+        "upfront_invest_diesel_gen": "USD",
+        "upfront_invest_inverter": "USD",
+        "upfront_invest_rectifier": "USD",
+        "upfront_invest_battery": "USD",
+        "upfront_invest_pv": "USD",
+        "upfront_invest_converters": "USD",
+        "upfront_invest_total": "USD",
+        "battery_capacity": "kWh",
+        "pv_capacity": "kW",
+        "diesel_genset_capacity": "kW",
+        "inverter_capacity": "kW",
+        "rectifier_capacity": "kW",
+        "co2_emissions": "t/a",
+        "fuel_consumption": "liter/a",
+        "peak_demand": "kW",
+        "base_load": "kW",
+        "max_shortage": "%",
+        "cost_fuel": "USD/a",
+        "epc_pv": "USD/a",
+        "epc_diesel_genset": "USD/a",
+        "epc_inverter": "USD/a",
+        "epc_rectifier": "USD/a",
+        "epc_battery": "USD/a",
+        "epc_total": "USD/a",
+    }


Also when we talked about it yesterday on teams it was clear that this code was deleted, from my understanding it was also not there in your local repository, so I'm not quite sure what happened here...

…n results

Fixed rebase issues.

paulapreuss

Nice job! 🚀

I know that this was a comprehensive feature with some challenges, but you pushed through it :) I think it works and looks very nice, thank you for implementing this. Feel free to merge the PR.

CelinaKellinghaus requested a review from paulapreuss September 10, 2025 08:27

CelinaKellinghaus added the enhancement 🚀 New feature or request label Sep 10, 2025

paulapreuss reviewed Sep 11, 2025

View reviewed changes

paulapreuss reviewed Oct 21, 2025

View reviewed changes

CelinaKellinghaus force-pushed the feature/osm-main-roads branch from 3adcde8 to d76cc94 Compare October 30, 2025 23:45

CelinaKellinghaus marked this pull request as ready for review October 30, 2025 23:49

CelinaKellinghaus force-pushed the feature/osm-main-roads branch from d76cc94 to 36b07e1 Compare December 15, 2025 22:29

paulapreuss previously requested changes Dec 17, 2025

View reviewed changes

CelinaKellinghaus added 9 commits December 17, 2025 11:00

Added OSM road detection feature for consumer selection and simulatio…

6045d9e

…n results

Fixed requests.exceptions.HTTPError

bd51302

Added functionality to save roads to database

5c3a12d

Added functionality to load roads from db in consumer selection

65910b5

Roads are now saved correctly to db

649bbb7

Added functionality to show roads in simulation results

048cd84

Removed unused osm view

f4b4146

Minor cleanup for better readability

b06d2b7

Removed unused function

4530e49

CelinaKellinghaus force-pushed the feature/osm-main-roads branch from 2453524 to 4530e49 Compare December 17, 2025 10:50

CelinaKellinghaus and others added 2 commits December 17, 2025 11:51

Removed Schema Validation

9f9508c

Add migration file

a93dcee

paulapreuss self-requested a review December 18, 2025 08:57

paulapreuss approved these changes Dec 18, 2025

View reviewed changes

		@@ -1,5 +1,6 @@
		from django.urls import path

		from . import views


		Roads.objects.filter(project=project).delete()

		Roads.objects.create(project=project, data=df.to_json(orient="records"))

		if isinstance(markers_only, str):
		markers_only = True if markers_only == "true" else False # noqa:SIM210

		df, msg = check_imported_consumer_data(df, proj_id)
		df, msg = check_imported_consumer_data(df)

Added OSM road detection feature for consumer selection and simulatio… #138

Are you sure you want to change the base?

Added OSM road detection feature for consumer selection and simulatio… #138

Uh oh!

Conversation

CelinaKellinghaus commented Sep 10, 2025

Uh oh!

paulapreuss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulapreuss commented Sep 25, 2025

Uh oh!

paulapreuss Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulapreuss commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paulapreuss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulapreuss left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

paulapreuss Oct 21, 2025 •

edited

Loading

paulapreuss commented Dec 15, 2025 •

edited

Loading