AVM Bargain Detection V2: What the Model Really Learned After Removing Target Leakage

V1 of this model claimed 98.6% F1. That number was wrong. The price-to-AVM ratio was the top feature with an importance score of 2,904. That ratio is the target variable dressed as an input. The model was reading the answer sheet.

V2 removes all leaky features: the ratio, the absolute price difference, and the extreme discount flag. It then expands the feature set from 1,651 to 4,047 by joining seven data sources through the GNAF mesh block. The honest cross-validated F1 is 0.883.

The classifier identified 1,412 actionable deals averaging $93,714 in savings per property. It flagged 924 data errors that would have looked like bargains to an unfiltered search.

When the model says a deal is genuine, it is right 94% of the time. That is the number that matters for investors. And 90% of actionable deals carry confidence above 80%.

AVM models compare a property's asking price to nearby recent sales. When the asking price sits well below the estimate, it looks like a bargain. But many apparent bargains are data errors.

Missing bedrooms. Wrong land size. Too few comparable sales. The AVM fills gaps with assumptions and those assumptions inflate the estimate. The "discount" is the model working with incomplete information.

V1 tried to solve this with a LightGBM classifier. It achieved 98.6% F1 and 98.55% ROC AUC. Those numbers looked too good. They were.

The problem was target leakage. V1 included the price-to-AVM ratio as a feature. That ratio is essentially the label in numeric form. Properties with low ratios are more likely to be errors. The model learned to read the ratio and barely looked at anything else. The ratio's importance score was 2,904. The next feature scored 1,628.

A previous rule-based system tried to catch errors with 12 hand-tuned rules and penalty points. Missing bedrooms cost 30 points. Few comparable sales cost 25 points. That approach also had problems. The weights were arbitrary. Nobody tested them against labelled data. And 12 rules cannot capture feature interactions.

Target leakage occurs when a feature encodes information about the label that would not be available at prediction time. In V1, three features were leaky.

V1's top feature was the ratio with a score of 2,904. The second feature (price std dev 180d 1km) scored 1,628. The ratio dominated every other signal by nearly 2x.

In V2, all three derived features are removed. The model must learn from the raw data: property attributes, neighbourhood price statistics, census demographics, crime rates, tax data, and socioeconomic indexes. No shortcuts.

V2 expands the feature set from 1,651 to 4,047 by joining seven data sources through the GNAF mesh block. Each source contributes a different perspective on the property and its neighbourhood.

Seven data sources join through the GNAF mesh block to produce 4,047 features per property. AVM and SAL features make up 97% of the total.

Join Method: GNAF Mesh Block

All seven sources join through the GNAF (Geocoded National Address File). Each property address maps to a mesh block code. From there, mesh blocks map upward to SA1, SA2, SA3, and SAL boundaries. GNAF provides 100% address coverage across the dataset.

The join chain: property address to GNAF mesh block, then mesh block to SA1 (for census, SEIFA, and SA1 features), mesh block to SA3 (for CMA stats), and mesh block to SAL (for suburb-level crime, tax, and real estate data).

What Changed from V1

V1 had 1,651 features from three sources: AVM, census, and three derived (leaky) variables. V2 adds four new sources (SAL, SEIFA, SA1 other, CMA stats) and removes the three leaky derived features. The net gain is 2,396 columns, mostly from SAL-level data.

Despite adding 2,367 SAL columns covering crime, tax, and real estate, neighbourhood price statistics from the AVM source still dominate feature importance. The SAL data provides breadth. The AVM data provides depth.

We drew a random sample of 2,000 properties from the full 9,329. No stratification. The natural distribution gave us this breakdown.

Each property was evaluated on three criteria. Attribute completeness: are bedrooms, bathrooms, and land size present? Comparable sales coverage: how many recent sales exist nearby? Price plausibility: does the asking price make sense for the neighbourhood?

Final labels split into two classes. 184 properties were marked as not-genuine (errors or ambiguous). 1,816 were genuine.

V2 labels are stricter than V1. V1 flagged 89 errors from 2,000 (4.5%). V2 flags 184 from 2,000 (9.2%). The higher error rate reflects tighter criteria on comparable coverage and attribute quality.

Ambiguous cases went into the not-genuine class. This is a conservative choice. It means the model may occasionally flag a genuine bargain as suspicious. But it will rarely approve a data error as a real deal. For property investors, that trade-off makes sense. A missed bargain costs opportunity. A fake bargain costs money.

We used LightGBM, a gradient boosted decision tree framework. Binary classification. The target: genuine (1) or not-genuine (0).

LightGBM handles missing values natively. It splits on missingness without needing imputation. This is critical for property data, where missing values are signal rather than noise. A missing bedroom count is itself informative. Imputing it with the median would destroy that signal.

We used is_unbalance=True because the classes are imbalanced: 184 errors against 1,816 genuine. This tells LightGBM to adjust its loss function to weight the minority class more heavily.

All 4,047 features were fed to the model. No manual feature selection. No dimensionality reduction. The model's own splitting acts as automatic feature selection. Of 4,047 inputs, most carry zero importance. The model ignores what does not help.

Early stopping at 50 rounds prevents overfitting. The model trains until validation performance stops improving, then rolls back to the best iteration.

The model was evaluated using stratified 5-fold cross-validation. Every property in the labelled set was scored exactly once, on a fold it was not trained on.

The numbers tell a different story from V1. Error precision dropped from 93% to 37%. Error recall dropped from 76% to 47%. The model is less aggressive at catching errors now that it cannot read the ratio.

But the metric that matters most for investors held up. Genuine precision is 94%. When the model says a deal is genuine, it is right 94 times out of 100. That is the number an investor acts on.

Genuine recall of 92% means the model approves 92% of all genuine deals. The other 8% are false negatives: real bargains flagged as suspicious. That is an acceptable trade-off. Better to miss a few deals than to approve bad data.

Confusion Matrix

The 2,000 labelled properties were classified across all 5 folds.

87 true errors correctly caught. 1,668 genuine properties correctly approved. 148 genuine properties wrongly flagged. And 97 errors that slipped through. The false positive count is higher than V1 because the model no longer has the ratio shortcut.

In practical terms: 90% of the 1,412 actionable deals carry confidence above 80%. Only 1 suspicious genuine prediction appeared across the entire dataset. That was 14 Gibbs Rd, Montrose, where bedroom data was missing.

V2 model confidence distribution. Genuine bargains cluster at high confidence. The model shows clear separation between classes even without the leaky ratio feature.

With the ratio removed, the model reveals what genuinely predicts data quality. The answer: neighbourhood price stability and sales coverage. Every top-10 feature measures how stable and well-covered local pricing is.

V2 feature importance scores (split count) for the top 10 features. Price standard deviations at various radii and time windows dominate. No leaky features appear.

V1 vs V2 Feature Comparison

Three real properties from the V2 scored dataset.

Distribution of discount percentages across all 1,412 genuine bargains in V2. Most cluster between 5% and 20%. Discounts above 25% are uncommon but do exist with clean underlying data.

After scoring all 9,329 properties and removing the 924 flagged errors, 1,412 actionable deals remain. Here is where they are.

Top 15 suburbs ranked by number of genuine bargains in V2. Ferntree Gully leads with 94 deals. Rowville and Croydon follow. Average confidence exceeds 90% in all 15 suburbs.

Ferntree Gully leads with 94 genuine deals at 93.5% average confidence. This makes sense. It has high turnover, strong comparable sales coverage, and stable local pricing. The model's top features (price standard deviations at 2km) are consistently low here.

Nunawading has the highest average savings at $119,884 per deal. Wantirna South follows at $109,467. Higher-value suburbs produce larger dollar savings even at similar percentage discounts.

Average confidence across all 15 suburbs exceeds 91%. Chirnside Park leads at 98.8%. Croydon is close behind at 98.7%. These are areas where the model has high certainty in its predictions.

This model works well within its scope. But that scope has clear boundaries.

Error precision is 37%. When the model flags a property as an error, it is right only 37% of the time. The other 63% are false alarms. This is the trade-off for removing the leaky ratio. The model is cautious. It flags more than it should rather than approving bad data. For investors, the genuine precision of 94% matters more.

Labels were programmatic, not expert. The labelling process used attribute completeness, comparable coverage, and price plausibility rules. A human property analyst might label some cases differently. True expert labelling would improve accuracy. But it is expensive at scale.

Single region only. This model was trained on Melbourne's Outer East. It would need retraining for other corridors. The feature importances might change entirely for inner-city areas, regional towns, or other states.

No outcome validation. We do not know if the "genuine bargains" actually sold at these prices. A property asking $500,000 might sell at $580,000 after competitive bidding. The model predicts data quality, not sale outcomes.

Census and SEIFA features added little. We included 89 SA1-level census variables and 4 SEIFA indexes expecting them to carry some predictive weight. They did not. This was also true in V1. Neighbourhood price statistics from the AVM source contain the signal.

SAL features: breadth without depth. The 2,367 SAL columns covering crime, tax, and real estate added the most new features by volume. But they did not displace the AVM's neighbourhood price stats from the top of the importance ranking. More data is not always better data.