source: docs/P7-AdvancedDatabaseDevelopment/AdvancedDatabaseDevelopment.md

main
Last change on this file was ef1c1c7, checked in by Stefan <trsunovstefan@…>, 6 days ago

Wiki docs, phase 6 and phase 7 added

  • Property mode set to 100644
File size: 53.0 KB
RevLine 
[ef1c1c7]1# Advanced Database Development
2
3EduBerza's users trade virtual money and virtual crypto with each other and with a simulated
4market. Up to P6 the prototype only knew market orders that filled immediately against the
5latest price, so an order was either untouched or completely done. This phase adds what makes an
6exchange consistent once orders can **wait in an order book, fill in parts, and trade with each
7other**, and puts every rule that keeps orders, trades and balances in agreement into the
8database itself — so it holds no matter who writes the data (the CLI, the market bot, a script,
9or someone typing SQL in DBeaver).
10
11Only rules that span several rows or several tables are listed. `NOT NULL`, `UNIQUE`, `CHECK`,
12primary and foreign keys are P2 ([RelationalDesign](../P2-RelationalDesign/RelationalDesign.md))
13and are not presented as P7 features.
14
15All of it is in [`server/db/advanced_db.sql`](../../server/db/advanced_db.sql), run by `-init`
16between `schema_creation.sql` and `data_load.sql`. Every rule is exercised by
17[`server/db/advanced_db_tests.sql`](../../server/db/advanced_db_tests.sql) (see
18[Tests](#tests-proving-the-rules)).
19
20## Overview
21
22| # | Requirement | Triggers | Procedures / functions | Views | Tables affected |
23|---|---|---|---|---|---|
24| 1 | [Order lifecycle and filled/remaining consistency](#order-lifecycle-and-filledremaining-consistency) | `orders_lifecycle` | | | `orders` |
25| 2 | [Trade consistency](#trade-consistency) | `market_trades_validate`, `market_trades_fill`, `market_trades_immutable` | `execute_trade` | | `market_trades`, `orders`, `users`, `holdings`, `transactions` |
26| 3 | [Balance and reservation consistency](#balance-and-reservation-consistency) | `reserved_cash_matches_orders`, `reserved_crypto_matches_orders`, `cash_matches_ledger` (deferred) | `order_reservation` | | `users`, `holdings`, `orders`, `transactions` |
27| 4 | [Placing and cancelling orders](#placing-and-cancelling-orders) | | `place_order`, `match_order`, `cancel_order` | | all of the above |
28| 5 | [Automatic recording of order events](#automatic-recording-of-order-events) | `orders_events` | | | `order_events` (new) |
29| 6 | [Views for derived trading data](#views-for-derived-trading-data) | | | `v_order_book`, `v_active_orders`, `v_order_history`, `v_trader_balances` | read-only |
30| 7 | [Background job: filling resting limit orders](#background-job-filling-resting-limit-orders) | | `fill_marketable_orders` | | `orders`, `market_trades`, … |
31
32### Schema additions
33
34The existing design had no place for three things the requirements need, so four columns were
35added — nothing else in the design changed:
36
37| Column | Why it is needed |
38|---|---|
39| `orders.filled_quantity` (+ status value `partially_filled`) | "filled / remaining quantity" cannot be kept consistent without storing how much was filled; remaining = `quantity − filled_quantity` |
40| `users.reserved_balance` | cash committed to open buy orders must be set aside somewhere; `holdings.reserved_quantity` already did this for crypto on sell orders |
41| `market_trades.buy_order_id`, `market_trades.sell_order_id` | "a trade can only happen between compatible orders" needs the trade to say which orders it filled. `NULL` on a side means the simulated market was the counterparty; the bot's price ticks have both `NULL` |
42
43One new table, `order_events`, holds the automatically recorded events (requirement 5).
44
45## Order lifecycle and filled/remaining consistency
46
47### Data requirements description
48
49**Business rule.**
50
51- An order's status follows from how much of it has been filled: nothing filled → `open`,
52 partly filled → `partially_filled`, completely filled → `executed`. The only status that is
53 set explicitly is `cancelled`, and only on an order that is still active.
54- `executed` and `cancelled` are final — such an order can never be processed again (no further
55 fills, no cancellation, no changes).
56- `filled_quantity` only grows, never exceeds `quantity`, and only changes when a trade fills
57 the order.
58- What was ordered — user, market, side, type, quantity, price, time of placement — never
59 changes after placement.
60- `executed_at` is set exactly when the order becomes `executed`.
61- New orders are only accepted on active markets, need a positive price, and start unfilled
62 (the one exception is importing an order that was completely executed in the past, used by the
63 sample data).
64
65**Why it is non-trivial.** The rule compares the *old* and the *new* version of a row (valid
66transitions, "final" states, "only grows", immutable columns) and ties two columns together
67(status ↔ filled quantity). A `CHECK` constraint only sees one version of one row. It also
68depends on *who* changes `filled_quantity` — a trade may, a manual `UPDATE` may not.
69
70**PostgreSQL feature.** `BEFORE INSERT OR UPDATE` row trigger on `orders`. The trade trigger
71(requirement 2) sets a transaction-local setting (`set_config('eduberza.trade_fill', 'on', true)`)
72while it fills an order; the lifecycle trigger only accepts a change of `filled_quantity` when
73that setting is on.
74
75**Tables affected.** `orders` (reads `markets` for the active check).
76
77### Implementation
78
79#### Triggers
80
81```sql
82CREATE OR REPLACE FUNCTION project.trg_orders_lifecycle()
83RETURNS trigger LANGUAGE plpgsql AS $$
84DECLARE
85 v_derived varchar(20);
86BEGIN
87 IF TG_OP = 'UPDATE' THEN
88 IF OLD.status IN ('executed', 'cancelled') THEN
89 RAISE EXCEPTION 'order % is already % and cannot be processed again', OLD.id, OLD.status
90 USING ERRCODE = 'check_violation';
91 END IF;
92 IF (NEW.user_id, NEW.market_id, NEW.side, NEW.type, NEW.quantity, NEW.price, NEW.placed_at)
93 IS DISTINCT FROM
94 (OLD.user_id, OLD.market_id, OLD.side, OLD.type, OLD.quantity, OLD.price, OLD.placed_at) THEN
95 RAISE EXCEPTION 'user, market, side, type, quantity, price and placed_at of an order cannot change'
96 USING ERRCODE = 'check_violation';
97 END IF;
98 IF NEW.filled_quantity <> OLD.filled_quantity THEN
99 IF current_setting('eduberza.trade_fill', true) IS DISTINCT FROM 'on' THEN
100 RAISE EXCEPTION 'filled quantity of order % can only change through a trade', OLD.id
101 USING ERRCODE = 'check_violation';
102 END IF;
103 IF NEW.filled_quantity < OLD.filled_quantity THEN
104 RAISE EXCEPTION 'filled quantity of order % cannot decrease', OLD.id
105 USING ERRCODE = 'check_violation';
106 END IF;
107 END IF;
108 IF NEW.status = 'cancelled' AND OLD.status <> 'cancelled' THEN
109 IF NEW.filled_quantity <> OLD.filled_quantity THEN
110 RAISE EXCEPTION 'an order cannot be filled and cancelled in the same step'
111 USING ERRCODE = 'check_violation';
112 END IF;
113 NEW.executed_at := NULL;
114 RETURN NEW;
115 END IF;
116 ELSE
117 IF NOT EXISTS (SELECT 1 FROM project.markets WHERE id = NEW.market_id AND is_active) THEN
118 RAISE EXCEPTION 'market % is not active; no new orders accepted', NEW.market_id
119 USING ERRCODE = 'check_violation';
120 END IF;
121 IF NEW.price IS NULL OR NEW.price <= 0 THEN
122 RAISE EXCEPTION 'an order needs a positive price (limit price, or the market price for a market order)'
123 USING ERRCODE = 'check_violation';
124 END IF;
125 IF NEW.status = 'cancelled' THEN
126 RAISE EXCEPTION 'an order cannot be created already cancelled'
127 USING ERRCODE = 'check_violation';
128 END IF;
129 -- A new order starts unfilled. The only exception is importing an
130 -- order that was completely executed in the past (sample data).
131 IF NEW.filled_quantity NOT IN (0, NEW.quantity) THEN
132 RAISE EXCEPTION 'a new order is either unfilled or (imported history) completely filled'
133 USING ERRCODE = 'check_violation';
134 END IF;
135 END IF;
136
137 v_derived := CASE
138 WHEN NEW.filled_quantity = 0 THEN 'open'
139 WHEN NEW.filled_quantity < NEW.quantity THEN 'partially_filled'
140 ELSE 'executed'
141 END;
142 IF NEW.status IS DISTINCT FROM v_derived
143 AND (TG_OP = 'INSERT' OR NEW.status IS DISTINCT FROM OLD.status) THEN
144 RAISE EXCEPTION 'order status % does not match filled quantity % of %; status is derived automatically',
145 NEW.status, NEW.filled_quantity, NEW.quantity
146 USING ERRCODE = 'check_violation';
147 END IF;
148 NEW.status := v_derived;
149
150 IF v_derived = 'executed' THEN
151 NEW.executed_at := COALESCE(NEW.executed_at, now());
152 ELSE
153 NEW.executed_at := NULL;
154 END IF;
155 RETURN NEW;
156END $$;
157
158CREATE TRIGGER orders_lifecycle
159 BEFORE INSERT OR UPDATE ON project.orders
160 FOR EACH ROW EXECUTE FUNCTION project.trg_orders_lifecycle();
161```
162
163Because the status is derived, the application never sets `open`, `partially_filled` or
164`executed` itself — it records trades, and the status follows.
165
166## Trade consistency
167
168### Data requirements description
169
170**Business rule.** A trade that fills orders must be possible for those orders:
171
172- the buy side is a buy order and the sell side is a sell order;
173- both are on the trade's market and still active — a cancelled or executed order can never
174 trade again;
175- the trade quantity does not exceed the remaining quantity of either order;
176- the price respects both limits: at most the buy order's price, at least the sell order's price;
177- the two orders belong to different users (no self-trade).
178
179Recording the trade must fill both orders by exactly the traded quantity (and so move their
180status), and move the money and crypto for both users, all together. A trade that filled orders
181is history and can't be changed or deleted afterwards — the fills and the money moved would no
182longer match it.
183
184**Why it is non-trivial.** One trade row has to be checked against two other rows of another
185table (the orders), including their current remaining quantity, and a single insert must cause
186consistent changes in five tables (`market_trades`, both `orders`, both users' `users` and
187`holdings` rows, two `transactions` rows).
188
189**PostgreSQL features.**
190
191- `BEFORE INSERT` trigger `market_trades_validate` — the compatibility rules. It locks both
192 orders (`FOR UPDATE`), so two concurrent trades can't both take the same remaining quantity.
193- `AFTER INSERT` trigger `market_trades_fill` — raises `filled_quantity` on both orders
194 (automatic status change through requirement 1).
195- `BEFORE UPDATE OR DELETE` trigger `market_trades_immutable`.
196- Stored function `execute_trade` — the settlement of money and crypto for one trade.
197
198Because the rules sit on `market_trades` itself, even a trade inserted by hand is validated and
199fills the orders.
200
201**Tables affected.** `market_trades`, `orders`, `users`, `holdings`, `transactions`.
202
203### Implementation
204
205#### Triggers
206
207```sql
208CREATE OR REPLACE FUNCTION project.trg_market_trades_validate()
209RETURNS trigger LANGUAGE plpgsql AS $$
210DECLARE
211 o project.orders%ROWTYPE;
212 v_uid uuid;
213 v_id uuid;
214 v_role varchar(4);
215BEGIN
216 IF NEW.buy_order_id IS NULL AND NEW.sell_order_id IS NULL THEN
217 RETURN NEW;
218 END IF;
219 IF NEW.buy_order_id IS NOT DISTINCT FROM NEW.sell_order_id THEN
220 RAISE EXCEPTION 'an order cannot trade with itself'
221 USING ERRCODE = 'check_violation';
222 END IF;
223
224 FOREACH v_role IN ARRAY ARRAY['buy', 'sell'] LOOP
225 v_id := CASE v_role WHEN 'buy' THEN NEW.buy_order_id ELSE NEW.sell_order_id END;
226 CONTINUE WHEN v_id IS NULL;
227
228 SELECT * INTO o FROM project.orders WHERE id = v_id FOR UPDATE;
229 IF o.side <> v_role THEN
230 RAISE EXCEPTION 'order % is a % order and cannot be the % side of a trade', v_id, o.side, v_role
231 USING ERRCODE = 'check_violation';
232 END IF;
233 IF o.market_id <> NEW.market_id THEN
234 RAISE EXCEPTION 'order % is on a different market than the trade', v_id
235 USING ERRCODE = 'check_violation';
236 END IF;
237 IF o.status NOT IN ('open', 'partially_filled') THEN
238 RAISE EXCEPTION 'order % is % and cannot trade', v_id, o.status
239 USING ERRCODE = 'check_violation';
240 END IF;
241 IF NEW.quantity > o.quantity - o.filled_quantity THEN
242 RAISE EXCEPTION 'trade quantity % exceeds the remaining quantity % of order %',
243 NEW.quantity, o.quantity - o.filled_quantity, v_id
244 USING ERRCODE = 'check_violation';
245 END IF;
246 IF v_uid IS NOT NULL AND v_uid = o.user_id THEN
247 RAISE EXCEPTION 'a user cannot trade with their own order'
248 USING ERRCODE = 'check_violation';
249 END IF;
250 IF v_role = 'buy' AND NEW.price > o.price OR v_role = 'sell' AND NEW.price < o.price THEN
251 RAISE EXCEPTION 'trade price % is outside the limit % of % order %', NEW.price, o.price, v_role, v_id
252 USING ERRCODE = 'check_violation';
253 END IF;
254 v_uid := o.user_id;
255 END LOOP;
256 RETURN NEW;
257END $$;
258
259CREATE TRIGGER market_trades_validate
260 BEFORE INSERT ON project.market_trades
261 FOR EACH ROW EXECUTE FUNCTION project.trg_market_trades_validate();
262```
263
264```sql
265CREATE OR REPLACE FUNCTION project.trg_market_trades_fill()
266RETURNS trigger LANGUAGE plpgsql AS $$
267BEGIN
268 IF NEW.buy_order_id IS NULL AND NEW.sell_order_id IS NULL THEN
269 RETURN NULL;
270 END IF;
271 PERFORM set_config('eduberza.trade_fill', 'on', true);
272 PERFORM set_config('eduberza.trade_price', NEW.price::text, true);
273 UPDATE project.orders
274 SET filled_quantity = filled_quantity + NEW.quantity,
275 executed_at = CASE WHEN filled_quantity + NEW.quantity = quantity
276 THEN NEW.executed_at END
277 WHERE id IN (NEW.buy_order_id, NEW.sell_order_id);
278 PERFORM set_config('eduberza.trade_fill', 'off', true);
279 RETURN NULL;
280END $$;
281
282CREATE TRIGGER market_trades_fill
283 AFTER INSERT ON project.market_trades
284 FOR EACH ROW EXECUTE FUNCTION project.trg_market_trades_fill();
285```
286
287```sql
288CREATE OR REPLACE FUNCTION project.trg_market_trades_immutable()
289RETURNS trigger LANGUAGE plpgsql AS $$
290BEGIN
291 IF OLD.buy_order_id IS NULL AND OLD.sell_order_id IS NULL THEN
292 -- a simulated tick filled no order; allow it unless it is being
293 -- turned into one that did
294 IF TG_OP = 'DELETE' THEN
295 RETURN OLD;
296 ELSIF NEW.buy_order_id IS NULL AND NEW.sell_order_id IS NULL THEN
297 RETURN NEW;
298 END IF;
299 END IF;
300 RAISE EXCEPTION 'a trade that filled orders cannot be changed or deleted'
301 USING ERRCODE = 'check_violation';
302END $$;
303
304CREATE TRIGGER market_trades_immutable
305 BEFORE UPDATE OR DELETE ON project.market_trades
306 FOR EACH ROW EXECUTE FUNCTION project.trg_market_trades_immutable();
307```
308
309#### Stored procedures/functions
310
311`execute_trade(buy_order, sell_order, quantity, price)` — one trade. Either order may be `NULL`
312when the simulated market is the counterparty. The `INSERT` into `market_trades` validates the
313trade and fills the orders (triggers above); the function then settles both users:
314
315- **Buyer:** the reservation for the filled part is released. The buyer pays the actual cost
316 (trade price × quantity); if the trade price is below the order's limit, the difference goes
317 back to `available_balance`. The crypto is added to the holding at a running weighted average
318 price, and a `buy` ledger row is written.
319- **Seller:** the reserved crypto is delivered out of the holding, the proceeds are credited, the
320 cost basis is removed from `invested_balance`, and a `sell` ledger row is written.
321
322Both orders are locked in a fixed order (by id), so two trades on the same pair of orders can't
323deadlock.
324
325```sql
326CREATE OR REPLACE FUNCTION project.execute_trade(
327 p_buy_order uuid, p_sell_order uuid, p_quantity numeric, p_price numeric,
328 p_aggressor varchar DEFAULT NULL)
329RETURNS bigint LANGUAGE plpgsql AS $$
330DECLARE
331 b project.orders%ROWTYPE;
332 s project.orders%ROWTYPE;
333 v_market project.markets%ROWTYPE;
334 v_trade_id bigint;
335 v_release numeric;
336 v_cost numeric;
337 v_proceeds numeric;
338 v_avg numeric;
339BEGIN
340 IF p_buy_order IS NULL AND p_sell_order IS NULL THEN
341 RAISE EXCEPTION 'a trade needs at least one order' USING ERRCODE = 'check_violation';
342 END IF;
343
344 -- lock both orders in a fixed order (by id) so two concurrent trades on
345 -- the same pair of orders cannot deadlock
346 PERFORM 1 FROM project.orders WHERE id IN (p_buy_order, p_sell_order) ORDER BY id FOR UPDATE;
347 SELECT * INTO b FROM project.orders WHERE id = p_buy_order;
348 SELECT * INTO s FROM project.orders WHERE id = p_sell_order;
349 SELECT * INTO v_market FROM project.markets WHERE id = COALESCE(b.market_id, s.market_id);
350
351 INSERT INTO project.market_trades
352 (market_id, executed_at, price, quantity, side, source, buy_order_id, sell_order_id)
353 VALUES (v_market.id, now(), p_price, p_quantity,
354 COALESCE(p_aggressor, CASE WHEN p_sell_order IS NULL THEN 'buy' ELSE 'sell' END),
355 CASE WHEN p_buy_order IS NOT NULL AND p_sell_order IS NOT NULL THEN 'match' ELSE 'market' END,
356 p_buy_order, p_sell_order)
357 RETURNING id INTO v_trade_id;
358
359 IF p_buy_order IS NOT NULL THEN
360 v_release := project.order_reservation(b.quantity - b.filled_quantity, b.price)
361 - project.order_reservation(b.quantity - b.filled_quantity - p_quantity, b.price);
362 v_cost := LEAST(round(p_quantity * p_price, 4), v_release);
363
364 UPDATE project.users
365 SET reserved_balance = reserved_balance - v_release,
366 available_balance = available_balance + (v_release - v_cost),
367 invested_balance = invested_balance + v_cost,
368 updated_at = now()
369 WHERE id = b.user_id;
370
371 INSERT INTO project.holdings AS h (user_id, crypto_id, quantity, avg_price, updated_at)
372 VALUES (b.user_id, v_market.crypto_id, p_quantity, p_price, now())
373 ON CONFLICT (user_id, crypto_id) DO UPDATE
374 SET avg_price = (h.quantity * h.avg_price + EXCLUDED.quantity * EXCLUDED.avg_price)
375 / (h.quantity + EXCLUDED.quantity),
376 quantity = h.quantity + EXCLUDED.quantity,
377 updated_at = now();
378
379 INSERT INTO project.transactions (user_id, type, amount, currency, related_order, description)
380 VALUES (b.user_id, 'buy', -v_cost, v_market.quote_currency, b.id,
381 format('Buy %s @ %s (trade %s)', p_quantity, p_price, v_trade_id));
382 END IF;
383
384 IF p_sell_order IS NOT NULL THEN
385 SELECT avg_price INTO v_avg FROM project.holdings
386 WHERE user_id = s.user_id AND crypto_id = v_market.crypto_id FOR UPDATE;
387
388 UPDATE project.holdings
389 SET quantity = quantity - p_quantity,
390 reserved_quantity = reserved_quantity - p_quantity,
391 updated_at = now()
392 WHERE user_id = s.user_id AND crypto_id = v_market.crypto_id;
393
394 v_proceeds := round(p_quantity * p_price, 4);
395 UPDATE project.users
396 SET available_balance = available_balance + v_proceeds,
397 invested_balance = GREATEST(invested_balance - round(p_quantity * v_avg, 4), 0),
398 updated_at = now()
399 WHERE id = s.user_id;
400
401 INSERT INTO project.transactions (user_id, type, amount, currency, related_order, description)
402 VALUES (s.user_id, 'sell', v_proceeds, v_market.quote_currency, s.id,
403 format('Sell %s @ %s (trade %s)', p_quantity, p_price, v_trade_id));
404 END IF;
405
406 RETURN v_trade_id;
407END $$;
408```
409
410## Balance and reservation consistency
411
412### Data requirements description
413
414**Business rule.**
415
4161. **Reserved cash matches active buy orders.** A user's `reserved_balance` always equals what
417 their active buy orders still reserve: remaining quantity × order price, summed.
4182. **Reserved crypto matches active sell orders.** A user's `holdings.reserved_quantity` for a
419 crypto always equals the remaining quantity of their active sell orders for it.
4203. **Cash matches the ledger.** `available_balance + reserved_balance` always equals the sum of
421 the user's ledger (`transactions`). Reserving only moves cash between the two columns; money
422 actually arrives or leaves only with a ledger row (deposit, buy fill, sell fill).
423
424Together these make it impossible for an order operation to leave a balance in an inconsistent
425state: money or crypto reserved for nothing, an order that is not backed by a reservation (and
426so could spend the same money twice), or cash that appeared or vanished without a ledger entry.
427
428**Why it is non-trivial.** Each rule is an equality between a column and an aggregate over
429*other* rows of *other* tables. Worse, every legitimate operation breaks it for a moment:
430placing a buy moves cash to `reserved_balance` in one statement and inserts the order in the
431next; a trade releases the reservation, pays, and writes the ledger in several statements. Only
432the state at the end of the transaction has to be consistent.
433
434**PostgreSQL feature.** `CONSTRAINT TRIGGER … DEFERRABLE INITIALLY DEFERRED` — row triggers
435whose check runs at `COMMIT`, on the final state. A transaction that leaves any of the three
436equalities broken fails at `COMMIT` and is rolled back as a whole. Each rule has a trigger on
437every table whose change can break it. `order_reservation(remaining, price)` is the one place
438that defines how a reservation is rounded, so placing, filling, cancelling and checking always
439agree to the last decimal.
440
441**Tables affected.** `users`, `holdings`, `orders`, `transactions`.
442
443### Implementation
444
445#### Stored procedures/functions
446
447```sql
448CREATE OR REPLACE FUNCTION project.order_reservation(p_remaining numeric, p_price numeric)
449RETURNS numeric LANGUAGE sql IMMUTABLE AS $$
450 SELECT round(p_remaining * p_price, 4)
451$$;
452```
453
454#### Triggers
455
456```sql
457CREATE OR REPLACE FUNCTION project.trg_reserved_cash_matches_orders()
458RETURNS trigger LANGUAGE plpgsql AS $$
459DECLARE
460 v_user uuid := CASE WHEN TG_TABLE_NAME = 'users' THEN NEW.id END;
461 v_reserved numeric;
462 v_needed numeric;
463BEGIN
464 IF TG_TABLE_NAME = 'orders' THEN
465 v_user := NEW.user_id;
466 END IF;
467 SELECT reserved_balance INTO v_reserved FROM project.users WHERE id = v_user;
468 IF NOT FOUND THEN
469 RETURN NULL;
470 END IF;
471 SELECT COALESCE(SUM(project.order_reservation(quantity - filled_quantity, price)), 0)
472 INTO v_needed
473 FROM project.orders
474 WHERE user_id = v_user AND side = 'buy' AND status IN ('open', 'partially_filled');
475 IF v_reserved <> v_needed THEN
476 RAISE EXCEPTION 'reserved balance % does not match the % needed by active buy orders (user %)',
477 v_reserved, v_needed, v_user
478 USING ERRCODE = 'check_violation', CONSTRAINT = 'reserved_cash_matches_orders';
479 END IF;
480 RETURN NULL;
481END $$;
482```
483
484```sql
485CREATE OR REPLACE FUNCTION project.trg_reserved_crypto_matches_orders()
486RETURNS trigger LANGUAGE plpgsql AS $$
487DECLARE
488 v_user uuid;
489 v_crypto uuid;
490 v_reserved numeric;
491 v_needed numeric;
492BEGIN
493 IF TG_TABLE_NAME = 'holdings' THEN
494 v_user := NEW.user_id;
495 v_crypto := NEW.crypto_id;
496 ELSE
497 IF NEW.side <> 'sell' THEN
498 RETURN NULL;
499 END IF;
500 v_user := NEW.user_id;
501 SELECT crypto_id INTO v_crypto FROM project.markets WHERE id = NEW.market_id;
502 END IF;
503 SELECT COALESCE(SUM(reserved_quantity), 0) INTO v_reserved
504 FROM project.holdings WHERE user_id = v_user AND crypto_id = v_crypto;
505 SELECT COALESCE(SUM(o.quantity - o.filled_quantity), 0) INTO v_needed
506 FROM project.orders o
507 JOIN project.markets m ON m.id = o.market_id
508 WHERE o.user_id = v_user AND m.crypto_id = v_crypto
509 AND o.side = 'sell' AND o.status IN ('open', 'partially_filled');
510 IF v_reserved <> v_needed THEN
511 RAISE EXCEPTION 'reserved quantity % does not match the % needed by active sell orders (user %, crypto %)',
512 v_reserved, v_needed, v_user, v_crypto
513 USING ERRCODE = 'check_violation', CONSTRAINT = 'reserved_crypto_matches_orders';
514 END IF;
515 RETURN NULL;
516END $$;
517```
518
519```sql
520CREATE OR REPLACE FUNCTION project.trg_cash_matches_ledger()
521RETURNS trigger LANGUAGE plpgsql AS $$
522DECLARE
523 v_user uuid;
524 v_cash numeric;
525 v_ledger numeric;
526BEGIN
527 IF TG_TABLE_NAME = 'users' THEN
528 v_user := NEW.id;
529 ELSIF TG_OP = 'DELETE' THEN
530 v_user := OLD.user_id;
531 ELSE
532 v_user := NEW.user_id;
533 END IF;
534 SELECT available_balance + reserved_balance INTO v_cash FROM project.users WHERE id = v_user;
535 IF NOT FOUND THEN
536 RETURN NULL;
537 END IF;
538 SELECT COALESCE(SUM(amount), 0) INTO v_ledger FROM project.transactions WHERE user_id = v_user;
539 IF v_cash <> v_ledger THEN
540 RAISE EXCEPTION 'cash % (available + reserved) does not match the ledger total % (user %)',
541 v_cash, v_ledger, v_user
542 USING ERRCODE = 'check_violation', CONSTRAINT = 'cash_matches_ledger';
543 END IF;
544 RETURN NULL;
545END $$;
546```
547
548```sql
549CREATE INDEX idx_orders_active ON project.orders (user_id, side)
550 WHERE status IN ('open', 'partially_filled');
551
552CREATE CONSTRAINT TRIGGER reserved_cash_matches_orders
553 AFTER INSERT OR UPDATE OF reserved_balance ON project.users
554 DEFERRABLE INITIALLY DEFERRED
555 FOR EACH ROW EXECUTE FUNCTION project.trg_reserved_cash_matches_orders();
556
557CREATE CONSTRAINT TRIGGER reserved_cash_matches_orders
558 AFTER INSERT OR UPDATE OF status, filled_quantity ON project.orders
559 DEFERRABLE INITIALLY DEFERRED
560 FOR EACH ROW EXECUTE FUNCTION project.trg_reserved_cash_matches_orders();
561
562CREATE CONSTRAINT TRIGGER reserved_crypto_matches_orders
563 AFTER INSERT OR UPDATE OF reserved_quantity ON project.holdings
564 DEFERRABLE INITIALLY DEFERRED
565 FOR EACH ROW EXECUTE FUNCTION project.trg_reserved_crypto_matches_orders();
566
567CREATE CONSTRAINT TRIGGER reserved_crypto_matches_orders
568 AFTER INSERT OR UPDATE OF status, filled_quantity ON project.orders
569 DEFERRABLE INITIALLY DEFERRED
570 FOR EACH ROW EXECUTE FUNCTION project.trg_reserved_crypto_matches_orders();
571
572CREATE CONSTRAINT TRIGGER cash_matches_ledger
573 AFTER INSERT OR UPDATE OF available_balance, reserved_balance ON project.users
574 DEFERRABLE INITIALLY DEFERRED
575 FOR EACH ROW EXECUTE FUNCTION project.trg_cash_matches_ledger();
576
577CREATE CONSTRAINT TRIGGER cash_matches_ledger
578 AFTER INSERT OR UPDATE OR DELETE ON project.transactions
579 DEFERRABLE INITIALLY DEFERRED
580 FOR EACH ROW EXECUTE FUNCTION project.trg_cash_matches_ledger();
581```
582
583The partial index `idx_orders_active` covers exactly the rows the reservation checks sum, and
584stays small because active orders are few compared with the whole order history.
585
586## Placing and cancelling orders
587
588### Data requirements description
589
590**Business rule.** Placing an order must, as one unit:
591
5921. check the market, side, type, quantity and price;
5932. reserve what the order commits — cash (quantity × price) for a buy, crypto for a sell — and
594 refuse the order if not enough is free;
5953. record the order;
5964. match it against the order book: other users' active limit orders on the same market whose
597 price is acceptable, best price first and oldest first (price–time priority), each trade at
598 the resting order's price;
5995. fill whatever is still unfilled from the simulated market if it is marketable at the current
600 market price.
601
602A **market order** is priced at the current market price, so it always fills completely in step
6034 or 5 and never waits. A **limit order** that isn't marketable stays in the order book.
604Cancelling an order must release exactly what it still reserves, and only for an active order of
605the caller.
606
607**Why it is non-trivial.** It is a multi-step operation over five tables whose steps depend on
608each other (how much is left after each match, what to release), and it has to be correct under
609concurrency: two orders of the same user must not both see the same free cash.
610
611**PostgreSQL feature.** Stored functions (PL/pgSQL). `place_order` locks the user's row
612(`SELECT … FOR UPDATE`) before checking free cash or crypto, so concurrent orders of the same
613user are serialised. Every step's consistency is still checked by the triggers of requirements
6141–3.
615
616**Tables affected.** `orders`, `users`, `holdings`, `market_trades`, `transactions`.
617
618### Implementation
619
620#### Stored procedures/functions
621
622```sql
623CREATE OR REPLACE FUNCTION project.latest_price(p_market_id uuid)
624RETURNS numeric LANGUAGE sql STABLE AS $$
625 SELECT price FROM project.market_trades
626 WHERE market_id = p_market_id
627 ORDER BY executed_at DESC, id DESC
628 LIMIT 1
629$$;
630```
631
632```sql
633CREATE OR REPLACE FUNCTION project.match_order(p_order_id uuid)
634RETURNS int LANGUAGE plpgsql AS $$
635DECLARE
636 o project.orders%ROWTYPE;
637 r record;
638 v_rem numeric;
639 v_qty numeric;
640 v_count int := 0;
641BEGIN
642 SELECT * INTO o FROM project.orders WHERE id = p_order_id FOR UPDATE;
643 FOR r IN
644 SELECT id, price, quantity - filled_quantity AS remaining
645 FROM project.orders
646 WHERE market_id = o.market_id
647 AND side <> o.side
648 AND type = 'limit'
649 AND status IN ('open', 'partially_filled')
650 AND user_id <> o.user_id
651 AND (o.side = 'buy' AND price <= o.price
652 OR o.side = 'sell' AND price >= o.price)
653 ORDER BY CASE WHEN o.side = 'buy' THEN price END ASC,
654 CASE WHEN o.side = 'sell' THEN price END DESC,
655 placed_at, id
656 FOR UPDATE
657 LOOP
658 SELECT quantity - filled_quantity INTO v_rem FROM project.orders WHERE id = p_order_id;
659 EXIT WHEN v_rem = 0;
660 v_qty := LEAST(v_rem, r.remaining);
661 IF o.side = 'buy' THEN
662 PERFORM project.execute_trade(o.id, r.id, v_qty, r.price, 'buy');
663 ELSE
664 PERFORM project.execute_trade(r.id, o.id, v_qty, r.price, 'sell');
665 END IF;
666 v_count := v_count + 1;
667 END LOOP;
668 RETURN v_count;
669END $$;
670```
671
672```sql
673CREATE OR REPLACE FUNCTION project.place_order(
674 p_user_id uuid, p_market_id uuid, p_side varchar, p_type varchar,
675 p_quantity numeric, p_limit_price numeric DEFAULT NULL)
676RETURNS uuid LANGUAGE plpgsql AS $$
677DECLARE
678 v_market numeric := project.latest_price(p_market_id);
679 v_price numeric;
680 v_available numeric;
681 v_free numeric;
682 v_crypto uuid;
683 v_order uuid;
684 v_rem numeric;
685BEGIN
686 IF p_side NOT IN ('buy', 'sell') OR p_type NOT IN ('market', 'limit') THEN
687 RAISE EXCEPTION 'invalid side % or type %', p_side, p_type USING ERRCODE = 'check_violation';
688 END IF;
689 IF p_quantity IS NULL OR p_quantity <= 0 THEN
690 RAISE EXCEPTION 'quantity must be positive' USING ERRCODE = 'check_violation';
691 END IF;
692 IF p_type = 'limit' THEN
693 IF p_limit_price IS NULL OR p_limit_price <= 0 THEN
694 RAISE EXCEPTION 'a limit order needs a positive limit price' USING ERRCODE = 'check_violation';
695 END IF;
696 v_price := p_limit_price;
697 ELSE
698 IF v_market IS NULL THEN
699 RAISE EXCEPTION 'market has no price yet' USING ERRCODE = 'check_violation';
700 END IF;
701 v_price := v_market;
702 END IF;
703
704 SELECT available_balance INTO v_available FROM project.users WHERE id = p_user_id FOR UPDATE;
705 IF NOT FOUND THEN
706 RAISE EXCEPTION 'user % does not exist', p_user_id USING ERRCODE = 'no_data_found';
707 END IF;
708
709 IF p_side = 'buy' THEN
710 IF v_available < project.order_reservation(p_quantity, v_price) THEN
711 RAISE EXCEPTION 'insufficient funds: the order needs %, available %',
712 project.order_reservation(p_quantity, v_price), v_available
713 USING ERRCODE = 'check_violation';
714 END IF;
715 UPDATE project.users
716 SET available_balance = available_balance - project.order_reservation(p_quantity, v_price),
717 reserved_balance = reserved_balance + project.order_reservation(p_quantity, v_price),
718 updated_at = now()
719 WHERE id = p_user_id;
720 ELSE
721 SELECT crypto_id INTO v_crypto FROM project.markets WHERE id = p_market_id;
722 SELECT quantity - reserved_quantity INTO v_free FROM project.holdings
723 WHERE user_id = p_user_id AND crypto_id = v_crypto FOR UPDATE;
724 IF COALESCE(v_free, 0) < p_quantity THEN
725 RAISE EXCEPTION 'insufficient holding: trying to sell %, free to sell %', p_quantity, COALESCE(v_free, 0)
726 USING ERRCODE = 'check_violation';
727 END IF;
728 UPDATE project.holdings
729 SET reserved_quantity = reserved_quantity + p_quantity, updated_at = now()
730 WHERE user_id = p_user_id AND crypto_id = v_crypto;
731 END IF;
732
733 INSERT INTO project.orders (user_id, market_id, side, type, status, quantity, price)
734 VALUES (p_user_id, p_market_id, p_side, p_type, 'open', p_quantity, v_price)
735 RETURNING id INTO v_order;
736
737 PERFORM project.match_order(v_order);
738
739 SELECT quantity - filled_quantity INTO v_rem FROM project.orders WHERE id = v_order;
740 IF v_rem > 0 AND v_market IS NOT NULL
741 AND (p_side = 'buy' AND v_market <= v_price OR p_side = 'sell' AND v_market >= v_price) THEN
742 IF p_side = 'buy' THEN
743 PERFORM project.execute_trade(v_order, NULL, v_rem, v_market, 'buy');
744 ELSE
745 PERFORM project.execute_trade(NULL, v_order, v_rem, v_market, 'sell');
746 END IF;
747 END IF;
748 RETURN v_order;
749END $$;
750```
751
752```sql
753CREATE OR REPLACE FUNCTION project.cancel_order(p_order_id uuid, p_user_id uuid DEFAULT NULL)
754RETURNS void LANGUAGE plpgsql AS $$
755DECLARE
756 o project.orders%ROWTYPE;
757 v_rem numeric;
758BEGIN
759 SELECT * INTO o FROM project.orders WHERE id = p_order_id FOR UPDATE;
760 IF NOT FOUND THEN
761 RAISE EXCEPTION 'order % does not exist', p_order_id USING ERRCODE = 'no_data_found';
762 END IF;
763 IF p_user_id IS NOT NULL AND o.user_id <> p_user_id THEN
764 RAISE EXCEPTION 'order % does not belong to this user', p_order_id
765 USING ERRCODE = 'insufficient_privilege';
766 END IF;
767 IF o.status NOT IN ('open', 'partially_filled') THEN
768 RAISE EXCEPTION 'order % is % and cannot be cancelled', p_order_id, o.status
769 USING ERRCODE = 'check_violation';
770 END IF;
771
772 v_rem := o.quantity - o.filled_quantity;
773 IF o.side = 'buy' THEN
774 UPDATE project.users
775 SET reserved_balance = reserved_balance - project.order_reservation(v_rem, o.price),
776 available_balance = available_balance + project.order_reservation(v_rem, o.price),
777 updated_at = now()
778 WHERE id = o.user_id;
779 ELSE
780 UPDATE project.holdings h
781 SET reserved_quantity = h.reserved_quantity - v_rem, updated_at = now()
782 FROM project.markets m
783 WHERE m.id = o.market_id AND h.user_id = o.user_id AND h.crypto_id = m.crypto_id;
784 END IF;
785
786 UPDATE project.orders SET status = 'cancelled' WHERE id = p_order_id;
787END $$;
788```
789
790In the prototype, placing an order is now a single call:
791
792```go
793err = db.DB.QueryRow(
794 `SELECT place_order($1, $2, $3, $4, $5, $6)`,
795 s.UserID, m.ID, side, orderType, qty, limit.value(),
796).Scan(&orderID)
797```
798
799Cancelling is `SELECT cancel_order($1, $2)` with the order the user picked from a numbered list
800of their open orders ([`server/trade.go`](../../server/trade.go)).
801
802## Automatic recording of order events
803
804### Data requirements description
805
806**Business rule.** Every important thing that happens to an order is recorded with its time:
807placement, each fill (with the quantity filled and the trade price), and cancellation. This is
808the order's audit trail — the history of *how* it reached its current state, which the order row
809alone (only the current state) cannot show.
810
811**Why it is non-trivial.** Events come from several places — `place_order`, trades made by
812`match_order`, trades made by the background job, `cancel_order`, and any direct SQL. Recording
813them in each of those places would miss some; recording them where the change actually happens
814cannot.
815
816**PostgreSQL feature.** `AFTER INSERT OR UPDATE` row trigger on `orders`, writing into the new
817table `order_events`. The fill price is handed over by the trade trigger through a
818transaction-local setting.
819
820**Tables affected.** `order_events` (new), written from changes to `orders`.
821
822### Implementation
823
824```sql
825CREATE TABLE project.order_events (
826 id bigserial PRIMARY KEY,
827 order_id uuid NOT NULL REFERENCES project.orders(id) ON DELETE CASCADE,
828 event_type varchar(20) NOT NULL
829 CHECK (event_type IN ('placed', 'partially_filled', 'filled', 'cancelled')),
830 quantity numeric(20,4) NOT NULL,
831 price numeric(18,6),
832 status_after varchar(20) NOT NULL,
833 created_at timestamptz NOT NULL DEFAULT clock_timestamp()
834);
835```
836
837#### Triggers
838
839```sql
840CREATE OR REPLACE FUNCTION project.trg_orders_events()
841RETURNS trigger LANGUAGE plpgsql AS $$
842BEGIN
843 IF TG_OP = 'INSERT' THEN
844 INSERT INTO project.order_events (order_id, event_type, quantity, price, status_after)
845 VALUES (NEW.id, 'placed', NEW.quantity, NEW.price, NEW.status);
846 ELSIF NEW.filled_quantity > OLD.filled_quantity THEN
847 INSERT INTO project.order_events (order_id, event_type, quantity, price, status_after)
848 VALUES (NEW.id,
849 CASE WHEN NEW.status = 'executed' THEN 'filled' ELSE 'partially_filled' END,
850 NEW.filled_quantity - OLD.filled_quantity,
851 current_setting('eduberza.trade_price', true)::numeric,
852 NEW.status);
853 ELSIF NEW.status = 'cancelled' AND OLD.status <> 'cancelled' THEN
854 INSERT INTO project.order_events (order_id, event_type, quantity, price, status_after)
855 VALUES (NEW.id, 'cancelled', NEW.quantity - NEW.filled_quantity, NEW.price, NEW.status);
856 END IF;
857 RETURN NULL;
858END $$;
859
860CREATE TRIGGER orders_events
861 AFTER INSERT OR UPDATE ON project.orders
862 FOR EACH ROW EXECUTE FUNCTION project.trg_orders_events();
863```
864
865## Views for derived trading data
866
867### Data requirements description
868
869The application needs several things that are *derived* from orders, trades and balances. They
870are defined once as views, instead of repeating the calculations in the application code:
871
872| View | Derived data | Used by |
873|---|---|---|
874| `v_active_orders` | active orders with remaining quantity and what each one holds in reserve | CLI `[13] My open orders`, `[14] Cancel an order` |
875| `v_order_book` | current order book: resting limit orders aggregated per market, side and price level | CLI `[12] Order book`, and when placing an order |
876| `v_order_history` | every order with its fill progress, number of trades and average fill price (from its trades) | CLI result of placing an order |
877| `v_trader_balances` | cash split into available and reserved, the ledger total it must equal, holdings at market value, net worth | CLI `[1] View balance` |
878
879None of these repeats a P6 report: P6 aggregates performance over a period, while these show
880the current state of the order book and accounts.
881
882### Implementation
883
884#### Views
885
886```sql
887CREATE VIEW project.v_active_orders AS
888SELECT o.id AS order_id,
889 o.user_id,
890 u.username,
891 o.market_id,
892 c.symbol,
893 m.quote_currency,
894 o.side,
895 o.type,
896 o.status,
897 o.quantity,
898 o.filled_quantity,
899 o.quantity - o.filled_quantity AS remaining,
900 o.price,
901 CASE WHEN o.side = 'buy'
902 THEN project.order_reservation(o.quantity - o.filled_quantity, o.price) ELSE 0 END AS reserved_cash,
903 CASE WHEN o.side = 'sell' THEN o.quantity - o.filled_quantity ELSE 0 END AS reserved_crypto,
904 o.placed_at
905FROM project.orders o
906JOIN project.users u ON u.id = o.user_id
907JOIN project.markets m ON m.id = o.market_id
908JOIN project.crypto c ON c.id = m.crypto_id
909WHERE o.status IN ('open', 'partially_filled');
910```
911
912```sql
913CREATE VIEW project.v_order_book AS
914SELECT market_id,
915 symbol,
916 quote_currency,
917 side,
918 price,
919 SUM(remaining) AS quantity,
920 COUNT(*) AS orders
921FROM project.v_active_orders
922WHERE type = 'limit'
923GROUP BY market_id, symbol, quote_currency, side, price;
924```
925
926```sql
927CREATE VIEW project.v_order_history AS
928SELECT o.id AS order_id,
929 o.user_id,
930 u.username,
931 c.symbol,
932 o.side,
933 o.type,
934 o.status,
935 o.quantity,
936 o.filled_quantity,
937 o.quantity - o.filled_quantity AS remaining,
938 o.price,
939 f.trades,
940 f.avg_fill_price,
941 o.placed_at,
942 o.executed_at
943FROM project.orders o
944JOIN project.users u ON u.id = o.user_id
945JOIN project.markets m ON m.id = o.market_id
946JOIN project.crypto c ON c.id = m.crypto_id
947LEFT JOIN LATERAL (
948 SELECT COUNT(*) AS trades,
949 round(SUM(t.quantity * t.price) / NULLIF(SUM(t.quantity), 0), 6) AS avg_fill_price
950 FROM (SELECT quantity, price FROM project.market_trades WHERE buy_order_id = o.id
951 UNION ALL
952 SELECT quantity, price FROM project.market_trades WHERE sell_order_id = o.id) t
953) f ON true;
954```
955
956```sql
957CREATE VIEW project.v_trader_balances AS
958SELECT u.id AS user_id,
959 u.username,
960 u.available_balance,
961 u.reserved_balance,
962 u.available_balance + u.reserved_balance AS total_cash,
963 COALESCE(l.ledger_total, 0) AS ledger_total,
964 u.invested_balance,
965 COALESCE(p.holdings_value, 0) AS holdings_value,
966 u.available_balance + u.reserved_balance + COALESCE(p.holdings_value, 0) AS net_worth
967FROM project.users u
968LEFT JOIN (SELECT user_id, SUM(amount) AS ledger_total
969 FROM project.transactions GROUP BY user_id) l ON l.user_id = u.id
970LEFT JOIN (SELECT user_id, SUM(market_value) AS holdings_value
971 FROM project.v_portfolio GROUP BY user_id) p ON p.user_id = u.id;
972```
973
974## Background job: filling resting limit orders
975
976### Data requirements description
977
978**Business rule.** In EduBerza the market price is moved by the simulator (the market bot), not
979by users' orders. A limit order that rests in the book — a buy at or above, or a sell at or
980below, the current market price — must then be filled by the simulated market, at the market
981price, just as it would have been had the price already been there when the order was placed.
982
983**Why it is relevant, and why a background job.** Without it, a limit order could only ever
984fill against another user's order, and with few users most limit orders would wait forever while
985the market price has long passed them. That would make limit orders useless in the simulation.
986Nothing happens at the moment the price crosses an order that a trigger could react to: the
987price moves through the bot's inserts into `market_trades`, which deliberately stay cheap single
988inserts. Scanning and filling every crossed order on every tick inside that insert would make
989each tick expensive. So the work runs as a periodic job after each round of price ticks.
990
991**PostgreSQL feature.** Stored function `fill_marketable_orders()`. It is scheduled by the
992application, because PostgreSQL has no built-in scheduler and the faculty server provides no
993`pg_cron` (checked: only `plpgsql` and `pgcrypto` are available, and the project role is not a
994superuser).
995
996- An advisory lock (`pg_try_advisory_xact_lock`) keeps two runs from filling the same orders
997 twice.
998- `FOR UPDATE … SKIP LOCKED` leaves alone an order a user is cancelling at that moment; the next
999 run picks it up.
1000- Every fill goes through `execute_trade`, so all the rules above apply to it.
1001
1002**Tables affected.** `orders`, `market_trades`, `users`, `holdings`, `transactions`,
1003`order_events`.
1004
1005### Implementation
1006
1007#### Stored procedures/functions
1008
1009```sql
1010CREATE OR REPLACE FUNCTION project.fill_marketable_orders()
1011RETURNS int LANGUAGE plpgsql AS $$
1012DECLARE
1013 r record;
1014 v_count int := 0;
1015BEGIN
1016 IF NOT pg_try_advisory_xact_lock(hashtext('project.fill_marketable_orders')) THEN
1017 RETURN 0;
1018 END IF;
1019 FOR r IN
1020 SELECT o.id, o.side, o.quantity - o.filled_quantity AS remaining, lp.price AS market_price
1021 FROM project.orders o
1022 JOIN project.markets m ON m.id = o.market_id AND m.is_active
1023 CROSS JOIN LATERAL (SELECT project.latest_price(o.market_id) AS price) lp
1024 WHERE o.type = 'limit'
1025 AND o.status IN ('open', 'partially_filled')
1026 AND (o.side = 'buy' AND o.price >= lp.price
1027 OR o.side = 'sell' AND o.price <= lp.price)
1028 ORDER BY o.placed_at, o.id
1029 FOR UPDATE OF o SKIP LOCKED
1030 LOOP
1031 IF r.side = 'buy' THEN
1032 PERFORM project.execute_trade(r.id, NULL, r.remaining, r.market_price, 'sell');
1033 ELSE
1034 PERFORM project.execute_trade(NULL, r.id, r.remaining, r.market_price, 'buy');
1035 END IF;
1036 v_count := v_count + 1;
1037 END LOOP;
1038 RETURN v_count;
1039END $$;
1040```
1041
1042#### Scheduling
1043
1044In [`bots/main.go`](../../bots/main.go), after every round of price ticks:
1045
1046```go
1047// P7 background job: the prices just moved, so fill any resting
1048// limit order the new market price has reached.
1049var filled int
1050if err := db.QueryRow(`SELECT fill_marketable_orders()`).Scan(&filled); err != nil {
1051 log.Printf("fill_marketable_orders: %v", err)
1052} else if filled > 0 {
1053 log.Printf(" filled %d resting limit order(s) at the new market price", filled)
1054}
1055```
1056
1057It can also be run by hand from any SQL client: `SELECT project.fill_marketable_orders();`.
1058
1059A run with the bot, after charlie placed a limit buy of 0.1 ETH at 3599 while the market was at
10603600: the bot's random walk took the price below 3599 and the job filled the order at the market
1061price.
1062
1063```
10642026/09/24 13:19:02 filled 1 resting limit order(s) at the new market price
1065```
1066
1067```
1068 username | side | type | status | quantity | filled_quantity | price | avg_fill_price
1069----------+------+-------+-----------+----------+-----------------+-------------+----------------
1070 charlie | buy | limit | cancelled | 0.1000 | 0.0000 | 3400.000000 |
1071 charlie | buy | limit | executed | 0.1000 | 0.1000 | 3599.000000 | 3593.979167
1072```
1073
1074## Tests proving the rules
1075
1076[`server/db/advanced_db_tests.sql`](../../server/db/advanced_db_tests.sql) plays a short trading
1077story on the sample data and, along the way, tries to break every rule. It starts from alice with
10788250 USD and 0.5 ETH, bob with 5000 USD, charlie with 2500 USD, and ETH/USD last traded at 3520:
1079
10801. alice places a limit sell of 0.3 ETH at 3600, and bob a limit buy of 0.1 at 3500. Both rest in
1081 the book.
10822. bob places a limit buy of 0.2 at 3650. It crosses alice's ask, so they trade 0.2 at 3600:
1083 alice's order becomes partially filled, and bob gets back the 10 he had reserved above the
1084 trade price.
10853. charlie places a market buy of 0.15. He takes alice's remaining 0.1 from the book, and the
1086 other 0.05 comes from the simulated market.
10874. Invalid trades, state changes and balance changes are attempted directly in SQL.
10885. bob cancels his bid.
10896. The simulator moves the price to 3450, and the background job fills bob's new limit buy at
1090 3500.
1091
1092The deferred checks are forced with `SET CONSTRAINTS ALL IMMEDIATE`, so a violation shows up
1093inside the test instead of at the final `COMMIT`. Everything is rolled back at the end. Run on
1094PostgreSQL 17 after `-init`:
1095
1096```
1097PASS place: limit sell above the market rests in the book, crypto reserved: alice ETH reserved = 0.3000
1098PASS place: limit buy below the market rests in the book, cash reserved: bob available 4650.0000 reserved 350.0000
1099PASS event: placement recorded automatically:
1100PASS view: order book shows both price levels: buy 0.1000 @ 3500.000000, sell 0.3000 @ 3600.000000
1101PASS consistency holds after placing:
1102PASS place: buy without enough free cash: insufficient funds: the order needs 60000.0000, available 2500.0000
1103PASS place: sell more than is free (0.2 of 0.5 is already reserved): insufficient holding: trying to sell 0.3, free to sell 0.2000
1104PASS match: trade between the two orders at the resting price:
1105PASS status: seller partially filled, buyer executed (automatic): alice_ask partially_filled 0.2000/0.3000
1106PASS money: buyer paid 720, got back the 10 reserved above the trade price: bob available 3930.0000 reserved 350.0000
1107PASS crypto: 0.2 ETH moved from alice (0.1 still reserved) to bob:
1108PASS ledger: one buy and one sell row, linked to the orders:
1109PASS event: fills recorded automatically:
1110PASS consistency holds after the trade:
1111PASS market order: filled completely in two trades: executed, 2 trades, avg 3600.000000
1112PASS market order: alice's ask is now executed, nothing left reserved:
1113PASS consistency holds after the market order:
1114PASS trade: price above the buyer's limit: trade price 3700.000000 is outside the limit 3500.000000 of buy order …
1115PASS trade: more than the order has remaining: trade quantity 0.500000 exceeds the remaining quantity 0.1000 of order …
1116PASS trade: a sell order used as the buy side: order … is a sell order and cannot be the buy side of a trade
1117PASS trade: order of another market: order … is on a different market than the trade
1118PASS trade: executed order cannot trade again: order … is executed and cannot trade
1119PASS trade: a user with their own order: a user cannot trade with their own order
1120PASS trade: a trade that filled orders cannot be deleted: a trade that filled orders cannot be changed or deleted
1121PASS state: status cannot be set to executed by hand: order status executed does not match filled quantity 0.0000 of 0.1000; status is derived automatically
1122PASS state: filled quantity cannot be changed by hand: filled quantity of order … can only change through a trade
1123PASS state: executed order cannot be processed again: order … is already executed and cannot be processed again
1124PASS state: ordered quantity cannot change: user, market, side, type, quantity, price and placed_at of an order cannot change
1125PASS state: cancelling by hand without releasing the reservation: reserved balance 350.0000 does not match the 0 needed by active buy orders (user …)
1126PASS cancel: someone else's order: order … does not belong to this user
1127PASS cancel: 350 back from reserved to available, event recorded: bob available 4280.0000 reserved 0.0000
1128PASS cancel: a cancelled order cannot be cancelled again: order … is cancelled and cannot be cancelled
1129PASS trade: cancelled order cannot trade: order … is cancelled and cannot trade
1130PASS balance: reserving cash with no order behind it: reserved balance 100.0000 does not match the 0 needed by active buy orders (user …)
1131PASS balance: reserving crypto with no order behind it: reserved quantity 0.0100 does not match the 0 needed by active sell orders (user …, crypto …)
1132PASS balance: cash changed without a ledger row: cash 2060.0000 (available + reserved) does not match the ledger total 1960.0000 (user …)
1133PASS job: fills exactly the orders the new price reached: bob_bid3 executed @ 3450.000000, charlie_ask (3700) still open
1134PASS job: buyer paid 345, got the 5 above the fill price back:
1135PASS job: nothing more to do on a second run:
1136PASS consistency holds after the job:
1137PASS views: every trader's cash equals their ledger:
1138
1139 passed | failed
1140--------+--------
1141 41 | 0
1142```
1143
1144The same rules seen from the prototype (bob, after alice put 0.3 ETH up for sale at 3600):
1145
1146```
1147-- Place buy order --
1148Latest price for ETH/USD = 3520.000000
1149Order book asks (other users' limit orders):
1150 3600.000000 0.3000 (1 orders)
1151[1] Market order (fills now at the best available price)
1152[2] Limit order (fills only at your price or better, otherwise waits in the order book)
1153> 2
1154Quantity: 0.2
1155Limit price: 3650
1156Order executed: buy 0.2000 ETH, average price 3600.000000
1157```
1158
1159## Changes to earlier phases
1160
1161- **Schema** ([RelationalDesign](../P2-RelationalDesign/RelationalDesign.md), [ERModel](../P1-ConceptualModel/ERModel.md)):
1162 - `users.reserved_balance`;
1163 - `orders.filled_quantity` and the status value `partially_filled`;
1164 - `market_trades.buy_order_id` / `sell_order_id` (optional references to `orders` — a new
1165 relationship "trade fills order");
1166 - the new table `order_events`.
1167- **Sample data** (`data_load.sql`):
1168 - deposit rows for bob and charlie, whose balances previously had no ledger entries behind
1169 them;
1170 - alice's seeded order is imported as completely filled;
1171 - the script runs as one transaction.
1172
1173 The balances documented in [BuildInstructions](../P4-Prototype/BuildInstructions.md) are
1174 unchanged.
1175- **P6 demo data** (`reports_demo_data.sql`): it adjusts the users' balances by exactly what it
1176 adds to the ledger, and imports its orders as completely filled. Both
1177 [AdvancedReports](../P6-AdvancedReports/AdvancedReports.md) outputs are unchanged.
1178- **Prototype:**
1179 - placing an order is one call to `place_order`, and market or limit can be chosen;
1180 - new menu items `[12] Order book`, `[13] My open orders`, `[14] Cancel an order`;
1181 - the balance screen shows reserved cash;
1182 - the bot runs the background job.
1183
1184## AI usage
1185
1186AI was used in this phase and is logged in full, per the course rule for P1 onward.
1187
1188- **Phase log:** [AdvancedDatabaseDevelopmentAIUsage.md](AdvancedDatabaseDevelopmentAIUsage.md)
1189
1190**In short:** the requirements — order state consistency, filled/remaining quantities, reserved
1191money and assets, trades only between compatible orders, the kinds of triggers, procedures and
1192views, and a background job only if relevant — were mine. I asked the AI to turn them into
1193concrete rules for the existing EduBerza database, implement them without redesigning it, test
1194them and document them.
Note: See TracBrowser for help on using the repository browser.