View Source

This page is for discussing ways in which we can accommodate multiple projects using same sensors without duplicating data.

Approach 1- Aggregation and Filtering on backend

Have the backends make requests between them and filter the data in the backend before returning to geodashboard.
The api returns aggregated data from own database and data recieved from API requests to other projects.

The geodashboard communicates with the geostreams-api instances directly. The filtering and aggregation of data takes place in the geodashboard app.

Use postgres partitioning and foreign data wrapper to connect tables of services with each other.

Scenario: Project A needs to show all datapoints of a source in Project B.

Partition the Project B database tables on sources. i.e. datapoints is partitioned into datapoints_sourceX, datapoints_sourceY.
- Queries can still be run on the main datapoints table returning data for all the sources
Create a partition table in Project A's database `datapoints_sourceX`.
Use foreign data wrapper to connect `datapoints_sourceX` partition table to Project B's datapoints_sourceX table.
Now, all the data of datapoints_sourceX will be available in datapoints table on Project A as well without it being copied into Project A database.

Will require restructuring the database schema.
The databases on each machine will have to be exposed so that they can communicate. Currently, they are only accessible on the machines locally.