This document provides a comprehensive technical reference for the allocation implementation in Ribasim. It bridges the gap between the mathematical formulation described in the concept documentation and the actual code implementation using JuMP.jl.
The allocation algorithm solves linear optimization problems to distribute water among competing demands. This document explains:
How mathematical formulations translate to JuMP code
Problem Building (placeholder initialization → real value updates)
Integration between the physical layer and optimization layer
Data structures and indexing patterns
Implementation details for each node type
2 Architecture Overview
2.1 Problem Building
The allocation optimization problems are build using a two-phase approach:
2.1.1 Phase 1: Problem Structure Initialization (allocation_init.jl)
During initialization, JuMP variables and constraints are created with placeholder values. These placeholders define the structure and relationships but not the actual values:
Why placeholders? JuMP requires the problem structure (variables, constraints, objective) to be defined upfront. The actual physical values (water levels, flows, demands) are not yet known during initialization and will change at each allocation timestep.
2.1.2 Phase 2: Real Value Updates (allocation_optim.jl)
Before each optimization, the set_simulation_data! functions update constraints with real values from the physical layer:
# From set_simulation_data! for Basinfor basin_id in basin_ids_subnetwork basin_idx = basin_id.idx current_storage_basin = current_storage[basin_idx]# Update the variable bounds with actual storage values storage_change_variable = storage_change[basin_id] JuMP.set_lower_bound( storage_change_variable,-current_storage_basin / scaling.storage )# ... more updatesend
This pattern allows efficient repeated optimization: the problem structure remains fixed while only the constraint coefficients and bounds are updated.
2.2 Physical Layer Integration
The allocation layer operates on a linearized version of the physical layer. Here’s how physical quantities map to optimization variables:
2.2.1 Basin Levels and Storage
In the physical layer, basin level \(h\) is a nonlinear function of storage \(S\) determined by the basin profile (area-storage relationship). For optimization, we linearize around the current state:
where \(A^n\) is the basin area at the current timestep.
Code implementation:
# In set_simulation_data! for Basin# Get linearization pointh_n = basin.current_level[basin_idx]A_n = current_area[basin_idx] # Area at current levelS_n = current_storage[basin_idx]# Define the decision variable for storage change# ΔS = S^{n+1} - S^n (scaled)storage_change = problem[:basin_storage_change]# The level at end of timestep is:# h^{n+1} = h^n + (1/A^n) * ΔS# This relationship is embedded in constraint coefficients (see below)
2.2.2 Connector Nodes (Flow as Function of Levels)
Nodes like TabulatedRatingCurve, LinearResistance, and ManningResistance have flow \(Q\) that depends on upstream level \(h_a\) and downstream level \(h_b\):
For a basin downstream: \(h_b^{n+1} - h_b^n \approx \frac{1}{A_b}(S_b^{n+1} - S_b^n)\)
Code implementation:
# From linearize_connector_node!, where t_level = t + Δt_allocation is the end of the# allocation timestep# Evaluate flow and derivatives at the linearization pointh_a =get_level(p, inflow_id, t_level)h_b =get_level(p, outflow_id, t_level)q0 =flow_function(connector_node, node_id, h_a, h_b, p, t_level)# Compute partial derivatives with automatic differentiation∂q∂h_a =forward_diff( level_a ->flow_function(connector_node, node_id, level_a, h_b, p, t_level), h_a,)∂q∂h_b =forward_diff( level_b ->flow_function(connector_node, node_id, h_a, level_b, p, t_level), h_b,)# Update constraint: Q = q0 + ∂q/∂h_a * Δh_a + ∂q/∂h_b * Δh_b# When h_b is a Basin level, substitute the linearized profile:# Q = q0 + (∂q/∂h_b / A_b) * ΔS_b + ...
The function set_partial_derivative_wrt_level! handles converting level derivatives to storage derivatives:
functionset_partial_derivative_wrt_level!( allocation_model::AllocationModel, node_id::NodeID, ∂q∂h::Float64, # Partial derivative of flow w.r.t. level p::Parameters, constraint::JuMP.ConstraintRef,)::Nothing (; problem, scaling) = allocation_model (; current_area) = p.current_basin_properties# Convert ∂Q/∂h to ∂Q/∂S using ∂h/∂S = 1/A, and account for the fact that the# flow and storage variables are scaled differently storage_change = problem[:basin_storage_change][node_id] JuMP.set_normalized_coefficient( constraint, storage_change,-∂q∂h * scaling.storage / (scaling.flow * current_area[node_id.idx]), )returnnothingend
3 Mathematical Formulation to Code Mapping
3.1 Decision Variables
3.1.1 Flow Variables
Mathematical formulation: For each link \((i,j)\) in the network, a flow variable \(Q_{ij}\) represents the volumetric flow rate.
Code implementation:
# From add_flow!problem[:flow] = JuMP.@variable(problem, flow[link = flow_links_subnetwork])
Key points: - Indexed by link::Tuple{NodeID, NodeID} representing (source, destination) - Declared without bounds; the capacities depend on the allocation timestep and the current physical state, and are therefore set before every optimization by update_flow_variable_bounds! (see Flow capacity bounds) - Scaled by scaling.flow for numerical stability - Stored in sparse array for efficient access: problem[:flow][link]
3.1.2 Basin Storage Change Variables
Mathematical formulation: For each basin \(b\), a storage change variable \(\Delta S_b = S_b^{n+1} - S_b^n\) represents the change in storage over the allocation timestep.
Key points: - Doubly-indexed by (node_id, demand_priority) - DemandPriorityIterator(node_id, p_independent) yields only the priorities for which this node actually has a demand, so no variables are created for unused combinations - Bounded by the demand value (a placeholder d, updated before optimization by set_demands!) - Similar structures for FlowDemand and for secondary network connections
Key points: - Triple-indexed: (node_id, demand_priority, objective_ord) - objective_ord = :first: relative error \(E^p_v\) used by the first (demand) objective - objective_ord = :second: fairness error \(\overline{E}^p_v\) used by the second objective - Bounded between 0 and 1 because these are relative errors
Key points: - Simple equality constraint between two flow variables - Applied to Pump, Outlet, LinearResistance, ManningResistance, TabulatedRatingCurve - Ensures mass conservation through the node
Discretized with a semi-implicit (linearized) scheme, not a fully implicit Euler step. \(f_{\text{pos}}\) (precipitation, drainage) is state-independent and thus explicit; \(Q_k\) and \(f_{\text{neg}}\) (evaporation, infiltration) depend on the state and are linearized around their end-of-step values via the low_storage_factor decision variable: \[\frac{S^{n+1} - S^n}{\Delta t} = \sum_{k} Q_k^{n+1} + f_{\text{pos}} - f^{n+1}_{\text{neg}}\]
Or in terms of storage change \(\Delta S = S^{n+1} - S^n\): \[\Delta S = \Delta t \left(\sum_{k} Q_k^{n+1} + f_{\text{pos}} - f^{n+1}_{\text{neg}}\right)\]
Key points: - The low_storage_factor linearizes the state-dependent negative forcing (evaporation, infiltration) to prevent negative storage, without requiring a nonlinear (fully implicit) solve - Forcing terms (f_pos, f_neg) represent precipitation, drainage, surface runoff, evaporation and infiltration - The coefficient scaling.flow / scaling.storage converts the scaled flow rates to a scaled storage change over the allocation timestep - Updated each optimization with current forcing values
3.2.3 UserDemand Allocated Sum
Mathematical formulation: The sum of allocated flows over all priorities equals the total inflow, summed over all inflow links of the UserDemand node: \[\sum_{b \in \text{in}(v)} F_{(b, v)} = \sum_{p \in P_v} F^p_v\]
Code implementation:
# From add_user_demand!problem[:user_demand_allocated_sum_constraint] = JuMP.@constraint( problem, [node_id = user_demand_ids_subnetwork],sum(flow[link_metadata.link] for link_metadata in inflow_links[node_id.idx]) ==sum( user_demand_allocated[node_id, demand_priority] for demand_priority inDemandPriorityIterator(node_id, p_independent) ); base_name ="user_demand_allocated_sum_constraint")
Key points: - Ensures only demanded water enters UserDemand nodes - A UserDemand node can have multiple inflow links, so the left-hand side is a sum - Sum on the right is over priorities where the node has a demand - Links the flow variables to the allocated flow variables
# From add_user_demand!problem[:user_demand_relative_error_constraint] = JuMP.@constraint( problem, [ node_id = user_demand_ids_subnetwork, demand_priority =DemandPriorityIterator(node_id, p_independent), ], d * user_demand_error[node_id, demand_priority, :first] ≥ d - user_demand_allocated[node_id, demand_priority], base_name ="user_demand_relative_error_constraint")
Key points: - Multiplication by demand d makes this effectively an absolute error - The error variable is forced to be at least the unmet demand fraction - Updated before each optimization with current demands
Key points: - The error represents the storage deficit below the minimum level - level_demand_error is indexed by (node_id, demand_priority, side, objective_ord) with side ∈ (:lower, :upper) - Separate constraints exist for the upper side, and the demands themselves only appear as constraint constants (they are not decision variables)
3.2.5 Return Flow Constraints
Mathematical formulation: For UserDemand with return factor \(r_i(t)\): \[Q_{\text{out}} = r_i \sum_{b \in \text{in}(v)} Q_{(b, v)}\]
Code implementation:
# From add_user_demand!problem[:user_demand_return_flow] = JuMP.@constraint( problem, [node_id = user_demand_ids_subnetwork], flow[outflow_link[node_id.idx].link] == return_factor *sum(flow[lm.link] for lm in inflow_links[node_id.idx]), base_name ="user_demand_return_flow")
Key points: - return_factor is a placeholder, updated before optimization with the value at the start of the allocation timestep (a rate) - Links the total inflow and the outflow of a UserDemand node - Models consumptive use (when the return factor < 1)
Key points: - q0, ∂q∂h_upstream, ∂q∂h_downstream are placeholders - Derivatives divided by area when basin is involved (converts \(\partial Q/\partial h\) to \(\partial Q/\partial S\)) - Updated in set_simulation_data! before each optimization - Handles various combinations (basin-basin, basin-boundary, etc.)
3.3 Objectives
The allocation optimization solves multiple objectives in sequence using lexicographic goal programming. Each objective is optimized; before the next non-empty objective is solved, the preceding result is constrained with an inequality in the direction of improvement.
For level demands at priority \(p\): \[\min \sum_{b; p\in P^\min_b} E^p_{b, \text{lower}} + \sum_{b; p\in P^\max_b} E^p_{b, \text{upper}}\]
Code implementation:
# From add_demand_objectives!# For flow demandsfirst_objective_expression = JuMP.AffExpr(0.0)for node_id in user_demand_ids_subnetworkif demand_priority inDemandPriorityIterator(node_id, p_independent) demand =get_demand(user_demand, node_id, demand_priority, t) JuMP.add_to_expression!( first_objective_expression, demand * user_demand_error[node_id, demand_priority, :first] )endend# Store both demand goals for later optimizationsecond_objective_expression = JuMP.AffExpr(0.0) # Fairness-error expressionpush!( objectives,AllocationObjective( AllocationObjectiveType.demand_flow, demand_priority, demand_priority_idx, [first_objective_expression, second_objective_expression], ))
Key points: - Error variables are weighted by their relative demand. This is equivalent to minimizing the total absolute unmet demand because the objective is the original demand-weighted error multiplied by the positive constant \(1 / \sum_{v \in V_p} d_v^p\). - The normalization is applied after all demand coefficients for a priority have been updated, including demands collected from secondary networks. It keeps each first objective and its retained lexicographic constraint numerically well scaled when the flow scale is much larger than the active demands. - Separate objectives for each priority - Objectives are stored and optimized in order by optimize_multi_objective!
Mathematical formulation: After minimizing the total error, minimize deviations from the average allocation rate.
For flow demands: \[\min \sum_{v,p} \overline{E}_v^p\]
where \(\overline{E}_v^p \ge E_v^p - G^p\) and \(G^p\) is the demand-weighted mean relative error: \[
G^p = \sum_{v \in V_p} w_v^p E_v^p.
\]
Code implementation:
# From add_demand_objectives!# Compute average error (constraint on average error variable)average_flow_unit_error_constraint =@constraint( problem, [demand_priority = demand_priorities_all; ...], average_flow_unit_error[demand_priority] ==sum(weight * error[node_id, demand_priority, :first]for node_id, weight in demand_weights), base_name ="average_flow_unit_error" )# Fairness error: overline_E >= E - G# (Implicitly defined through constraints)# Objective sums fairness errorsobjective_expression_fairness =sum( user_demand_error[node_id, demand_priority, :second]for node_id in user_demand_ids_subnetwork if...)
Key points: - average_flow_unit_error represents \(G^p\) - Fairness errors (the :second error variables) penalize being worse than average - Optimized after the primary objective is satisfied
3.3.3 LevelDemand Fairness Normalization
LevelDemand first-objective errors represent storage volumes. For a priority \(p\) and side \(s\), the implementation derives the mean current area over participating Basins,
then defines the mean storage error with \(n_{p,s}\bar E_{p,s} = \sum_b E^p_{b,s}\). The second objective penalizes storage errors after converting them by the relative area scale:
The ratios are recomputed for every allocation solve from current Basin areas. This gives the same relative level-error comparison as dividing each storage error by its area, but avoids mixing very large raw-area and very small reciprocal-area coefficients in the LP.
where \(w_i\) is the route priority weight (cost) for node \(i\).
Code implementation:
# From add_route_priority_objective!objective_expression = JuMP.AffExpr(0.0)for link in flow_links# Get the route priority weight for the source node weight = route_priority_weight[link[1]]if weight >0 JuMP.add_to_expression!( objective_expression, weight * flow[link] )endend
Key points: - Optimized after all demand objectives are satisfied - Only affects routing, not how much is allocated - Higher weight = less preferred route (higher cost)
over the allocation controlled nodes \(i\), ordered by node type and node ID.
Code implementation:
# From add_flow_change!, which update_control_states! calls every allocation timestepflow_change = JuMP.@variable(problem, [node_ids], lower_bound =0.0)JuMP.@constraint( problem, [node_id = node_ids, sign = (-1, 1)], flow_change[node_id] ≥ sign * allocation_controlled_flows[node_id],)# From set_flow_change_rhs!, called in set_simulation_data!:# flow_change - sign * flow ≥ -sign * previous_flowJuMP.set_normalized_rhs(constraints[node_id, sign], -sign * previous_flow)
Key points: - Optimized last, so it only chooses between solutions that are equally good for all other objectives - The previous flow is flow_rate of the node, which is where allocation writes its result - The slightly increasing weights make the objective itself generically have a unique optimum - The structure only depends on which nodes are allocation controlled; the previous flows are right hand sides
where \(\alpha_b\) is the low storage factor for basin \(b\).
Code implementation:
# From add_low_storage_factor_objective!# Note: Maximizing low_storage_factor is equivalent to minimizing its negativeexpression =-sum( low_storage_factor[node_id]for node_id in basin_ids_subnetwork)push!( objectives,AllocationObjective(type= AllocationObjectiveType.low_storage_factor, expressions = [expression], ),)
Key points: - Maximizes the low storage factor, so outflows from Basins are only reduced to prevent negative storage - Optimized first, before the demand objectives. Otherwise the demand objectives can lower the low storage factors to hold back water, for instance to meet a LevelDemand by throttling a weir, which the physical layer cannot do
4 Updating Constraints Before Optimization
Before each optimization run, constraints must be updated with current values from the physical layer. This is done through a series of set_simulation_data! functions.
4.1 Basin Updates
functionset_simulation_data!( allocation_model::AllocationModel, basin::Basin, p::Parameters, Δt_allocation::Float64,)::Bool# ... setup code ...for basin_id in basin_ids_subnetwork idx = basin_id.idx storage_now = current_storage[idx]# Storage may never become negative. No upper bound: Basin profiles extrapolate# above their top JuMP.set_lower_bound(storage_change[basin_id], -storage_now / scaling.storage)# ... rest of constraint updates ...endreturn errorsend
Key concepts: - Variable bounds are updated each timestep based on the current Basin storage - Only the lower bound is set, so the storage cannot become negative. There is no upper bound, since Basin profiles extrapolate above their top - Forcing terms (precipitation, evaporation) are computed and added to constraints - Errors are detected (e.g. storage outside the valid range) and reported
4.2 Connector Node Updates (Linearization)
functionlinearize_connector_node!( allocation_model::AllocationModel, connector_node::AbstractParameterNode, flow_constraint, flow_function::Function, p::Parameters, t_level::Float64,) (; problem, scaling) = allocation_model (; inflow_link, outflow_link) = connector_node# Evaluate at t_level = t + Δt_allocation, the end of the allocation timestep. This is# semi-implicit, not a fully implicit Euler step: the partial derivatives below are# frozen at the start of the timestep, only the level unknowns being solved for are the# end-of-step values.for node_id inonly(flow_constraint.axes) inflow_id = inflow_link[node_id.idx].link[1] outflow_id = outflow_link[node_id.idx].link[2]# Levels at the linearization point h_a =get_level(p, inflow_id, t_level) h_b =get_level(p, outflow_id, t_level) constraint = flow_constraint[node_id]# Raw (unreduced) flow: the physical low-storage scalar must not be baked in here,# otherwise it would be double-applied on top of the `low_storage_factor` variable# set below. This raw value becomes the coefficient of that variable (or, when# neither side of the connector is a Basin, the plain constraint right-hand side). q0_raw =flow_function( connector_node, node_id, h_a, h_b, p, t_level; apply_low_storage_factor =false, )# The basin whose low_storage_factor governs this flow, matching the# upstream/downstream selection used in the physical layer's# low_storage_factor_resistance_node. relevant_id = q0_raw >0 ? inflow_id : outflow_id lsf =get_low_storage_factor(problem, relevant_id)# Reset both candidate coefficients every call: flow direction (and hence which# basin's low_storage_factor applies) can flip between allocation steps.for basin_id in (inflow_id, outflow_id) basin_id.type== NodeType.Basin ||continue JuMP.set_normalized_coefficient(constraint, get_low_storage_factor(problem, basin_id), 0.0)endif lsf isa JuMP.VariableRef JuMP.set_normalized_coefficient(constraint, lsf, -q0_raw / scaling.flow) JuMP.set_normalized_rhs(constraint, 0.0)else JuMP.set_normalized_rhs(constraint, q0_raw / scaling.flow)end# Only linearize if the level comes from a Basin, since only then it is a# decision variable of the problem. The derivatives keep using the# factor-included flow function (default `apply_low_storage_factor = true`):# since the physical low-storage scalar is a constant w.r.t. level at the# linearization point, this yields the correctly-scaled ∂q/∂h for free.if inflow_id.type== NodeType.Basin ∂q∂h_a =forward_diff( level_a ->flow_function(connector_node, node_id, level_a, h_b, p, t_level), h_a, )set_partial_derivative_wrt_level!( allocation_model, inflow_id, ∂q∂h_a, p, constraint, )end# ... same for outflow_id ...endend
Key concepts: - Derivatives are computed with automatic differentiation (forward_diff) of the same flow functions the physical layer uses, so the allocation layer cannot drift from the physics - Only Basin levels are decision variables; LevelBoundary and Terminal levels enter through the low_storage_factor coefficient or the constant right-hand side - Semi-implicit, not fully implicit: Basin levels and LevelBoundary levels are evaluated at the end of the timestep (\(t + \Delta t\)), but the linearization coefficients (∂q/∂h, and the q0_raw used for the low_storage_factor coefficient) are all computed from state known at the start of the timestep — no nonlinear solve is performed. Other inputs have their own evaluation times: schedule rates (UserDemand, FlowDemand, return factor) are read at the start of the timestep, FlowBoundary flow rates are read at the intervention time t_known, and LevelDemand levels at the end of the timestep. For Basins get_level returns the level at the start of the timestep, which is exactly the point we want to linearize around - This same mechanism is used for LinearResistance, ManningResistance, TabulatedRatingCurve, Pump and Outlet, so the low_storage_factor variable uniformly governs the low-storage reduction of every connector node type instead of being baked into a stale physical-layer scalar
4.3 Demand Updates
functionset_demands!( allocation_model::AllocationModel, node::Union{UserDemand, FlowDemand},# ... other parameters)::Nothingfor node_id in demand_node_ids_subnetworkfor demand_priority inDemandPriorityIterator(node_id, p_independent)# Get current demand from the physical layer, in scaled units demand =get_demand(node, node_id, demand_priority, t) / scaling.flow# Update error constraints with the current demandfor objective_ord in (:first, :second) error_constraint = node_relative_error_constraint[node_id, demand_priority, objective_ord] JuMP.set_normalized_coefficient( error_constraint, node_error[node_id, demand_priority, objective_ord], demand, ) JuMP.set_normalized_rhs(error_constraint, demand)end# Update upper bound on allocated variable JuMP.set_upper_bound(node_allocated[node_id, demand_priority], demand)endendreturnnothingend
Key concepts: - Demands are time-varying and interpolated from input tables - Constraint coefficients are updated to reflect current demands - The demands are stored in the problem already divided by scaling.flow, so the bounds of the allocated variables are directly usable as scaled flow bounds elsewhere (see Flow capacity bounds) - Upper bounds on allocation variables ensure allocated ≤ demanded
4.4 Flow capacity bounds
Flow variables are declared without bounds, because their capacities depend on the current state of the physical layer and on the allocation timestep. They are set before every optimization, in this order:
update_flow_variable_bounds! sets the physical capacity of every non-fixed flow variable;
update_user_demand_flow_bounds! tightens the bounds of the links around UserDemand nodes to their total demand;
update_flow_demand_variable_bounds! derives the bounds of the FlowDemand helper variables from the now finalized flow bounds.
All of these use the helper set_variable_bounds!, where an infinite bound means that no JuMP bound is set at all:
Each flow link is bounded by both of the nodes it connects, so the bounds are the intersection of the intervals returned by connector_flow_capacity_bounds for either node:
functionupdate_flow_variable_bounds!( allocation_model::AllocationModel, integrator::DEIntegrator, Δt_allocation::Float64,)::Nothing (; p, t) = integrator (; problem, scaling, flow_links_subnetwork) = allocation_model flow = problem[:flow]for flow_link in flow_links_subnetwork flow_var = flow[flow_link]# Flows out of FlowBoundary nodes are fixed to the boundary flow JuMP.is_fixed(flow_var) &&continue lower, upper =-Inf, Inffor node_id in flow_link node_lower, node_upper =connector_flow_capacity_bounds( node_id, p, t, Δt_allocation, scaling.flow, ) lower =max(lower, node_lower) upper =min(upper, node_upper)end lower ≤ upper ||error("Empty flow capacity interval [$lower, $upper] for $flow_link.")set_variable_bounds!(flow_var, lower / scaling.flow, upper / scaling.flow)endreturnnothingend
connector_flow_capacity_bounds dispatches on the node type:
Pump and Outlet: the tightest min_flow_rate and max_flow_rate over \([t, t + \Delta t]\), since allocation assigns a single flow rate for the whole timestep. Both are multiplied by level_reduction_factor: the product of the level based reduction factors of the physical layer (min_upstream_level, max_downstream_level and for an Outlet the level difference), evaluated at the levels at the start of the allocation timestep. If this leaves less than MIN_SCALED_CAPACITY (in units of the flow scaling), the node is treated as closed: a level that rests at its limit leaves a capacity too small for the solver tolerances, which makes the problem infeasible next to a FlowDemand;
LinearResistance, ManningResistance and non-allocation-controlled TabulatedRatingCurve: linearized_flow_bounds, see below;
allocation controlled TabulatedRatingCurve: \([0, Q(h_{a,\max})]\) where \(h_{a,\max}\) is the highest reachable upstream level;
UserDemand: \([0, \infty)\), tightened afterwards by update_user_demand_flow_bounds!;
anything else: \((-\infty, \infty)\), i.e. no bound from this node.
4.4.2 Bounds from the linearized flow relation
For nodes whose flow follows from a linearized flow-level relation, the tightest possible bounds follow from evaluating that same linearization over the reachable level changes:
The reachable level change follows from the bounds on the storage change decision variable, converted with the same linearized profile \(\Delta h = \Delta S / A^n\) the constraints use:
functionallocation_level_change_bounds( allocation_model::AllocationModel, p::Parameters, node_id::NodeID,)::Tuple{Float64, Float64} (; problem, scaling, node_ids_in_subnetwork) = allocation_model# Levels of all other node types are data rather than decision variables,# so they cannot change within one optimization node_id.type== NodeType.Basin ||return (0.0, 0.0)# `flow_links_subnetwork` can contain links to Basins outside this subnetwork (node_id ∈ node_ids_in_subnetwork.basin_ids_subnetwork) ||return (-Inf, Inf) storage_change = problem[:basin_storage_change][node_id] level_per_storage = scaling.storage / p.current_basin_properties.current_area[node_id.idx]return ( JuMP.lower_bound(storage_change) * level_per_storage, JuMP.upper_bound(storage_change) * level_per_storage, )end
Because these bounds are derived from the same linearization that appears in the flow constraint, they can never conflict with the constraint and therefore never make the problem infeasible.
Important
For allocation controlled Pump and Outlet nodes there is no fallback bound, so a finite max_flow_rate is required. This is checked by valid_allocation_flow_capacity in core/src/validation.jl, both for the current parameters and for all DiscreteControl states, so a control state change cannot make the problem unbounded later in the simulation.
4.4.3 Demand-based bounds
update_user_demand_flow_bounds! bounds all inflow links and the outflow link of a UserDemand node between zero and its total demand over all demand priorities. For the outflow the return factor is conservatively assumed to be 1. Since the demand values stored in the problem are already divided by scaling.flow, the upper bounds of the allocated variables can be used directly.
update_flow_demand_variable_bounds! then takes the (now final) bounds of the flow variable of the node with a flow demand and transfers them to the helper variables: the allocated variable of the earliest demand priority may go down to \(\min(0, Q^\min)\) to represent negative flow, and the ‘demand priority 0’ surplus variable is bounded above by \(\max(0, Q^\max)\). If the flow variable has no bound on a side, the corresponding helper variable bound is deleted as well.
5 The Optimization Loop
5.1 Orchestration: update_allocation!
update_allocation! is the entry point that is called every allocation timestep. It first refreshes the physical state (water_balance!) and the scaling factors, and then per subnetwork performs the update sequence:
update_control_states!(allocation_model, p_independent)set_simulation_data!(allocation_model, p, t, Δt, u, du, t_known)reset_demand_coefficients(allocation_model)set_demands!(allocation_model, p, t, Δt)update_flow_variable_bounds!(allocation_model, p, t, Δt)update_user_demand_flow_bounds!(allocation_model, p_independent)update_flow_demand_variable_bounds!(allocation_model, p_independent)normalize_flow_demand_objectives!(allocation_model)
The order matters: set_demands! must run before the flow bounds are set, because update_user_demand_flow_bounds! reads the demands from the bounds of the allocated variables, and update_flow_demand_variable_bounds! in turn reads the finalized flow variable bounds.
Secondary networks are updated first. If a primary network is present, its update sequence is interrupted after reset_demand_coefficients to run the demand collection for every secondary network.
Afterwards, for every subnetwork in turn the problem is optimized (optimize_multi_objective!), the results are parsed and saved, the primary network allocation is distributed over the secondary networks (allocate_flows_to_subnetwork), and the results are applied to the physical layer with apply_control_from_allocation!.
5.2 Lexicographic optimization
The main optimization loop in optimize_multi_objective! solves objectives in sequence:
Key concepts: - Lexicographic optimization: Each objective is optimized in order - Before optimizing objective \(i + 1\), add constraint: \(\text{obj}_i \le \text{optimal}_i\) - This ensures later objectives don’t degrade earlier ones - An expression can be non-retained. It is optimized as a deterministic tie-break, but is not constrained for the next expression. - Temporary constraints are removed after all objectives are solved - The termination status is checked after every objective optimization; an infeasible or other non-optimal solve stops allocation immediately - If all objective expressions are zero, for instance when all demands are zero in a subnetwork without Basins, the problem is solved once without an objective, so that there are results to read
6 Scaling for Numerical Stability
The optimization uses scaling factors to improve numerical conditioning:
@kwdefmutable struct ScalingFactors flow::Float64 =1.0e3# Typical flow rate (m³/s) storage::Float64 =1.0e6# Typical storage value (m³) mean_half_storage::Float64 =0.0# Initialization-time reference storage (m³)endfunctionupdate_scaling!(p::Parameters, t::Float64, Δt::Float64, t_known::Float64) (; p_independent, current_basin_properties) = p (; allocation_models) = p_independent.allocation (; current_storage) = current_basin_propertiesfor allocation_model in allocation_models (; node_ids_in_subnetwork, scaling) = allocation_model (; basin_ids_subnetwork) = node_ids_in_subnetworkifisempty(basin_ids_subnetwork)# Without a Basin the storage scale is not used, so scale by the flows. If these# are all zero, keep the previous scaling. flow =typical_boundary_flow(p_independent, node_ids_in_subnetwork, t, t_known)isnothing(flow) || (scaling.flow = flow) scaling.storage = scaling.flow * Δtelse storage_sum =sum(current_storage[id.idx] for id in basin_ids_subnetwork) scaling.storage = (scaling.mean_half_storage + storage_sum /length(basin_ids_subnetwork)) /2 scaling.flow = scaling.storage / Δtendendreturnend
mean_half_storage is computed once when the allocation problem is built, as the mean over the Basins in the subnetwork of half their maximum storage. A subnetwork without a Basin is scaled by its flows instead: typical_boundary_flow gives the order of magnitude (power of 10) of the geometric mean of the nonzero FlowBoundary flow rates and UserDemand and FlowDemand demands in the subnetwork. Unlike the arithmetic mean, the geometric mean is not dominated by a single large value, like a demand that is much larger than the available flow. As in the optimization, the FlowBoundary flow rates are read at the intervention time t_known, so that a future change of a boundary flow doesn’t change the scaling.
Key concepts: - Storage variables are divided by scaling.storage - Flow variables are divided by scaling.flow - The storage scale is updated for each allocation solve from the current mean storage and the initialization-time mean half-storage - The flow scale equals scaling.storage / Δt, so that a typical flow over an allocation timestep amounts to a typical storage change - Per-priority flow-demand objectives are normalized independently, because a global flow scale can still make a small active demand numerically insignificant - Improves numerical stability and solver performance - Must scale/unscale when communicating with physical layer
6.1 Example: Scaled Volume Conservation
Unscaled formulation:\[\Delta S \text{ [m³]} = \Delta t \text{ [s]} \cdot \left( Q_{\text{in}} - Q_{\text{out}} \text{ [m³/s]} \right)\]
Since scaling.flow = scaling.storage / Δt, this single coefficient carries both the unit conversion from flow rate to volume over the allocation timestep and the ratio of the two scaling factors. It is updated together with the rest of the constraint in set_simulation_data!.
7 Data Structures and Indexing
7.1 NodeID
Nodes are identified by the NodeID struct:
struct NodeIDtype::NodeType.T # e.g., NodeType.Basin, NodeType.UserDemand idx::Int32 # Index into the type-specific data arrays value::Int32 # User-facing ID from input fileend
Key concepts: - idx is used to index into type-specific arrays (e.g., basin.storage[basin_id.idx]) - value is the ID from the input file (for error messages, output) - type distinguishes different node types
Key concepts: - Only creates variables for valid index combinations - Filtered by iterating over DemandPriorityIterator(node_id, p_independent), which yields only the demand priorities the node actually has - Efficient for sparse problems (not all nodes have all priorities)
7.4 AllocationModel Struct
The main data structure:
@kwdefmutable struct AllocationModel subnetwork_id::Int32 node_ids_in_subnetwork::NodeIDsInSubnetwork problem::JuMP.Model # The JuMP optimization problem# Time since the last time results were written Δt_since_last_record::Float64 =0.0 has_demand_priority::Vector{Bool}# Optimization objectives objectives::Vector{AllocationObjective} = AllocationObjective[]# Forcing volumes (precipitation, drainage, evaporation, infiltration) explicit_positive_forcing_volume::OrderedDict{NodeID, Float64} =OrderedDict() implicit_negative_forcing_volume::OrderedDict{NodeID, Float64} =OrderedDict()# Cumulative tracking for output cumulative_supplied_volume::OrderedDict{Tuple{NodeID, NodeID}, Float64} =OrderedDict() sources::OrderedDict{Int32, NodeID} =OrderedDict() secondary_network_demand::OrderedDict{Tuple{NodeID, NodeID}, Vector{Float64}} =OrderedDict()# All links with at least one node in this subnetwork flow_links_subnetwork::Vector{Tuple{NodeID, NodeID}} =Tuple{NodeID, NodeID}[] scaling::ScalingFactors =ScalingFactors()# Current LevelDemand fairness normalization by demand priority level_demand_area_sum::Dict{Int32, Float64} =Dict() level_demand_count::Dict{Int32, Int} =Dict() level_demand_area_scale::Dict{Int32, Float64} =Dict()# Temporary constraints for lexicographic optimization temporary_constraints::Vector{JuMP.ConstraintRef} = JuMP.ConstraintRef[]end
Note
The allocation timestep Δt_allocation is not a field of AllocationModel. Because the timestep is adaptive, it is passed as a function argument to all functions that need it. Δt_since_last_record is unrelated: it accumulates the time since results were last written.
8 Secondary Networks and Primary-Secondary Connections
8.1 Primary Network
The primary network (subnetwork_id = 1) represents the main water system. It:
Contains its own demand nodes
Connects to secondary networks via Pump or Outlet nodes
Treats each secondary network as a single demand node
8.2 Secondary Networks
Secondary networks (subnetwork_id > 1):
Can only connect to the primary network
Cannot connect to each other directly
Have their own internal allocation optimization
8.3 The Two-Stage Process
8.3.1 Stage 1: Demand Collection
For each secondary network:
Allow inflow from the primary network up to the physical capacity of the connecting links
Solve the demand and fairness objectives for each priority
Minimize total primary-to-secondary inlet flow as a non-retained tie-break
Use that minimum inlet flow as the secondary network’s request for the priority
The tie-break removes solver-dependent excess flow from demand collection. It is non-retained because later priorities may legitimately need more inlet flow.
functionpreprocess_demand_collection!( allocation_model::AllocationModel, p_independent::ParametersIndependent,)::Nothing (; problem, subnetwork_id) = allocation_model@assert !is_primary_network(subnetwork_id) flow = problem[:flow]# The capacity bounds of the primary network connections were set from the current# allocation data, so demands are collected up to the physical capacity. The primary# network allocation will subsequently restrict these links to its result.for link in p_independent.allocation.primary_network_connections[subnetwork_id] flow_variable = flow[link] lower_bound = JuMP.has_lower_bound(flow_variable) ? JuMP.lower_bound(flow_variable) :0.0 JuMP.set_lower_bound(flow_variable, max(0.0, lower_bound))endreturnnothingend
Note that only the lower bound needs adjusting here: the upper bound already is the physical capacity of the link, set by update_flow_variable_bounds!. Since the connecting node is always an allocation controlled Pump or Outlet, that capacity is finite.
8.3.2 Stage 2: Allocation
In primary network:
Solve allocation considering:
Primary network’s own demands
Aggregated demands from secondary networks
Allocated amounts to secondary networks determine their inflow limits
In each secondary network:
Set inflow constraint to the allocated amount from primary network
Solve allocation to distribute water to internal demands
functionallocate_flows_to_subnetwork( allocation_models::Vector{AllocationModel}, primary_network_connections,)::Nothing primary_network =get_primary_network(allocation_models) primary_problem = primary_network.problemfor secondary_network inget_secondary_networks(allocation_models)# Get allocated flow from primary networkfor link in primary_network_connections[secondary_network.subnetwork_id] allocated_flow = JuMP.value(primary_problem[:flow][link])# Set as capacity for secondary network secondary_problem = secondary_network.problem JuMP.set_upper_bound( secondary_problem[:flow][link], allocated_flow )endendend
9 Solving from scratch
Every optimization solves the problem from scratch with optimize_cold!, without start values or the basis of the previous solve. HiGHS only uses presolve when there is no basis. For large subnetworks presolve removes nearly all rows, because most flows follow from linearized flow-level relations.
Start values make HiGHS construct a basis, which is slow for large subnetworks. Reusing the basis of the previous solve needs fewer iterations and has a lower median solve time, but on large subnetworks some solves take much longer and more of them end in status Unknown, so we prefer the more robust solve from scratch.
A solve from scratch can also end in status Unknown or a solver error, when its solution just violates the feasibility tolerance after unscaling. A solve that doesn’t reach the optimum is first retried with slightly relaxed constraints on the previous objectives, and if that fails or there are no previous objectives, with looser feasibility tolerances. The looser tolerances are the last resort, because the next objectives can’t reach a bound found with them, so each of those solves then needs a retry as well.
10 Output and Communication with Physical Layer
10.1 Parsing Results
After optimization, results must be extracted and stored. parse_allocations! has a dispatcher method plus one method for UserDemand and FlowDemand nodes and one for LevelDemand nodes:
functionparse_allocations!( integrator::DEIntegrator, allocation_model::AllocationModel, Δt_allocation,)::Nothing# Dispatches to the UserDemand, FlowDemand and LevelDemand methodsend# For UserDemand and FlowDemandfor node_id in demand_node_ids_subnetworkfor demand_priority inDemandPriorityIterator(node_id, p_independent)# Get allocated amount from the optimization result and unscale it allocated = JuMP.value(node_allocated[node_id, demand_priority]) * scaling.flow# Store on the node for use by the physical layer node.allocated[node_id.idx, demand_priority_idx] = allocated# Store for output in the demand results tablepush!(record_demand, DemandRecordDatum(...))endend
For LevelDemand nodes there is no allocated flow variable; instead the volume supplied to the Basin over the allocation timestep is derived from the optimized storage change and recorded.
10.2 Applying Allocation Results
Allocated amounts must be communicated to the physical layer.
10.2.1 UserDemand Nodes
The allocated flow rates per demand priority are written to user_demand.allocated. In the physical layer, the abstraction of a UserDemand node is limited by the sum of these allocated flow rates rather than by its demand, so that the node never takes more than it was allocated.
10.2.2 Pump, Outlet and TabulatedRatingCurve Control
When a Pump, Outlet or TabulatedRatingCurve has allocation_controlled set to true:
functionapply_control_from_allocation!( node::Union{Pump, Outlet, TabulatedRatingCurve}, allocation_model::AllocationModel, integrator::DEIntegrator,)::Nothing (; problem, scaling) = allocation_model flow = problem[:flow]for node_id in controlled_node_ids# Get optimized flow from allocation link = (inflow_id(node_id), outflow_id(node_id)) optimized_flow = JuMP.value(flow[link]) * scaling.flow# Set as target flow rate in physical layer node.flow_rate[node_id.idx] = optimized_flowendend
The physical layer still applies the usual reduction factors (low storage, level difference, maximum downstream level) to this flow rate, so the realized flow can be lower than the allocated flow rate.
11 Handling Infeasibility
When the optimization problem is infeasible, diagnostic tools help identify the cause:
11.1 Infeasibility Analysis
functionanalyze_infeasibility( allocation_model::AllocationModel, t::Float64, config::Config,)::JuMP.TerminationStatusCode (; problem) = allocation_model# Find Irreducible Inconsistent Subsystem (IIS) data_infeasibility = MathOptAnalyzer.analyze( MathOptAnalyzer.Infeasibility.Analyzer(), problem; optimizer =get_optimizer(), )# Extract conflicting constraints violated_constraints = [...]# Try relaxing constraints to identify issues constraint_to_penalty =Dict( constraint => (isempty(JuMP.name(constraint)) ? 1.0:0.5)for constraint in violated_constraints ) constraint_to_slack = JuMP.relax_with_penalty!(problem, constraint_to_penalty)optimize_cold!(problem)# Report which constraints are in conflictfor irreducible_infeasible_subset in data_infeasibility.iis constraint_violations = [...]@error"Set of incompatible constraints found" constraint_violationsendreturn JuMP.INFEASIBLEend
Key concepts: - IIS = minimal set of constraints that cannot be satisfied simultaneously - Constraint relaxation helps identify which constraints are problematic - Named constraints are more informative for debugging
11.2 Common Infeasibility Causes
Basin overflows: In allocation all variables must have upper bounds. The upper bound of the basin level is defined by the user’s input. If more water flows into the basin than is possible based on the user defined basin dimensions, the model is infeasible.
Tips for performance: - Use scaling to keep values O(1) - Minimize number of priorities (each adds objectives) - Simplify network topology where possible
12.3 Allocation Timestep Selection
By default the allocation timestep \(\Delta t_{\text{allocation}}\) within an intervention interval is adapted to the success of the optimization, see Adaptive timestepping and update_allocation_until!. It can be overridden with a fixed value using the allocation.dtconfiguration setting. The trade-off is:
Too small: Frequent optimization overhead, linearization always valid
Too large: Linearization may be inaccurate, less responsive to changes
13 Debugging Tips
13.1 Inspecting the Problem
# Write problem to file for manual inspectionwrite_problem_to_file(problem, config)# Check variable valuesprintln(JuMP.value(flow[link]))println(JuMP.value(storage_change[basin_id]))# Check constraint valuesprintln(JuMP.normalized_rhs(constraint))println(JuMP.normalized_coefficient(constraint, variable))
If many variables are at bounds, the problem may be under-constrained or infeasible.
13.2.2 Numerical Scaling Issues
functionanalyze_scaling( allocation_model::AllocationModel, t::Float64, config::Config,)::Nothing data_numerical = MathOptAnalyzer.analyze( MathOptAnalyzer.Numerical.Analyzer(), problem; threshold_small =1e-12, threshold_large =1e6, )# Check for poorly scaled coefficientsfor data in data_numerical.matrix_small@error"Too small coefficient" data.coefficientendfor data in data_numerical.matrix_large@error"Too large coefficient" data.coefficientendend
14 Summary
This technical reference provides the link between mathematical formulation and code implementation:
Two-phase approach: Structure defined at initialization with placeholders, values updated before each optimization
Physical layer integration: Linearization of basin profiles and connector nodes around current state
Objectives: Lexicographic goal programming for demands, fairness, route priorities
Scaling: Improves numerical stability by keeping values O(1)
Data structures: NodeID-based indexing, sparse JuMP arrays, link tuples
Primary-secondary networks: Two-stage allocation process
For user-facing documentation, see Allocation Concept. For implementation code, see core/src/allocation_init.jl, core/src/allocation_optim.jl, core/src/allocation_util.jl, and core/src/validation.jl.