Alert escalation: EDA notes

Hackathon task: predict which monitoring alerts get escalated. Metric: ROC-AUC.

1. Approach

We have one row per alert and a long transaction log behind each one, so most of the work was turning that log into features. For every transaction we calculated how many days before the alert date it happened, and built all the windows (last day, last week, last month and so on) from that.

We ended up with 284 features per alert. We trained LightGBM with pretty heavy regularization and a logistic regression on quantile-transformed features, then averaged their ranks (60/40). Validation is 5-fold stratified CV repeated with 3 seeds. Out-of-fold AUC is .

2. Data

FileRowsOne row isColumns

Some things we checked early on:

3. Target

17.2% of training alerts were escalated. The rate barely moves over time (16–19% per quarter), so the alert date on its own doesn't help much.

Escalation rate by quarter

4. Activity over time

Activity goes down as the alert gets closer: about 3 transactions a day around four months before, and about 1 a day in the last few weeks. The drop at day 179 is only the edge of the 180-day window. The day before the alert is very different: about 40 transactions, nearly all in the last hour (23:00–23:59). We think this burst is what fires the alert. But it looks the same for escalated and dismissed alerts, so it doesn't help separate them.

Escalated customers are a little more active overall (about 6% more transactions), and they have more transactions on the alert date itself.

Transactions per alert by day before the alert (day 0 excluded)
DismissedEscalated
Hour of transaction: last day vs. earlier
Days 2–180Last day
Escalated / dismissed transaction count
Above 1 = escalated customers are more active

5. Transaction amounts

This was the most useful finding. Escalated customers have smaller amounts: the mean miqdor_indeksi is 0.05–0.09 lower in every window. It shows up most in bank transfers, where small transfers are up to 15% more common and large ones about 30% less common for escalated alerts. Card payments show the same thing, but weaker. Cash shows almost no difference.

Mean amount index by window
DismissedEscalated
Density ratio of amounts, escalated / dismissed
1 = no difference. Bins with very little data are hidden.
Bank transferCardCash

6. Transaction types and direction

We expected cash or international transfers to matter, but the type mix is almost identical between the two groups (within 0.2 percentage points), and so is the incoming/outgoing split. The difference is in the amounts inside each type, so we built amount features for each type and direction separately.

Share of transactions by type
DismissedEscalated
Median amount index by type and direction

7. Features and model

Feature groups we kept:

What we tried and dropped, because CV didn't improve: hour and weekday histograms, features about the last-day burst, in-then-out transfer pairs, CatBoost, and a transaction-level model (AUC 0.556).

LightGBM feature importance (gain)
Best single features (univariate AUC)
Below 0.5 = higher value means less likely escalated
FeatureAUC
Out-of-fold ROC-AUC

8. Further analysis after the first model

After the first submission we kept looking for signal the model might be missing. We compared every idea against the same baseline, 5-fold CV repeated with 5 seeds (0.6470 ± 0.0012), so small gains could be told apart from noise.

Three new patterns showed up in the data:

Escalation rate by weekly-rhythm quintile
Lag-7 autocorrelation of daily counts, low → high
Escalation rate by amount-jump quintile
Mean |amount − previous amount|, low → high

These patterns are real, but they overlap with features the model already had. For example, the floor effect is mostly covered by the bank-transfer amount quantiles. None of the experiments below cleared the +0.005 bar we set before resubmitting, so we kept the original model.

ExperimentCV AUC
Original model (submitted)0.6470
+ windowed per-type amount quantiles, IQR/MAD, money volume0.6475
+ burst fine structure, weekly rhythm, hour entropy, in/out transitions, floor counts0.6477
+ amounts relative to type×direction average (customer amount level)0.6478
+ full amount histogram per type×direction (78 features)0.6460
LightGBM tuning (7 settings)0.6431–0.6473
Ensemble of 6 model variants0.6482
Top-30 features selected inside each fold0.6485

We also checked structure outside the transactions: alert ID and file order, how many alerts arrive on the same day, and escalation rate by date, week and month. All of it looked like random noise (AUC 0.49–0.51).

9. Conclusion

  1. All alerts come after a similar late-night burst of transactions, so the burst alone doesn't tell you much. The customer's earlier history matters more.
  2. Transaction size is the main signal. Escalated customers have smaller bank transfers and card payments over the whole 180 days.
  3. They are slightly more active, but their type mix is the same as everyone else's.
  4. No single feature gets above ~0.575 AUC. Combining them gets to 0.649, and 30+ further experiments stayed within 0.646–0.6485, which looks like the ceiling for this data. Logistic regression is close to LightGBM, which suggests the signal is mostly simple and monotonic.