Quick scan for drift

28 thg 10, 2024

🙏🏻

ML based algorading is all about detecting any kind of non-randomness & exploiting it, kinda speculative stuff, not my way, but still...

Drift is one of the patterns that can be exploited, because pure random walks & noise aint got no drift.

This is an efficient method to quickly scan tons of timeseries on the go & detect the ones with drift by simply checking wherther drift < -0.5 or drift > 0.5. The code can be further optimized both in general and for specific needs, but I left it like dat for clarity so you can understand how it works in a minute not in an hour



^^ proving 0.5 and -0.5 are natural limits with no need to optimize anything, we simply put the metric on random noise and see it sits in between -0.5 and 0.5

You can simply take this one and never check anything again if you require numerous live scans on the go. The metric is purely geometrical, no connection to stats, TSA, DSA or whatever. I've tested numerous formulas involving other scaling techniques, drift estimates etc (even made a recursive algo that had a great potential to be written about in a paper, but not this time I gues lol), this one has the highest info gain aka info content.

The timeseries filtered by this lil metric can be further analyzed & modelled with more sophisticated tools.

Live Long and Prosper

P.S.: there's no such thing as polynomial trend/drift, it's alwasy linear, these curves you see are just really long cycles

P.S.: does cheer still work on TV? admin

19 thg 11, 2024

Phát hành các Ghi chú

Fixes:
* Corrected data intergration formula

New:
* Added Type 1 formula (Type 0 is the one from the original version). Type 1 formula can be used incrementally with no need to recalculate the whole thing on each data udpade, alors it has lesser info gain and no sensefull thresholds, so ain't no confirming/rejecting drift hypothesises, it's rather a tool to feed the drift/trend intensity to other metrics (more about that in later drops)