Property Analysis

In data analysis, much as in life, cleaning is important. Unfortunately, while keeping ones house and body cleaning rarely takes up all of one's time, data cleaning, can, and often does take up almost all of the time. So, enjoying (or at least not hating) this process, is an important career skill.

Anyways, so the first thing we do is look at the data, to assess what needs to be done.

dat  <- readr::read_csv("PPR_2020.csv")
  ## dat <- readxl::read_excel("PPR-ALL.xlsx", sheet="PPR-ALL")
  names(dat)

OK, so we have date (nicely, they have specified the format), address, postal code, county.

I have some problems with these names. The spaces will cause problems, and capitalisation is so passe. I'll fix these problems with some code.

normalise_names <- function(df) {
    nms <- names(df)
    normed <- iconv(
        tolower(
            gsub("([[:space:]]|[[:punct:]])+", "_",
                 x=nms)),
        "latin1", "ASCII", sub="")
    drop_usc <- gsub("([a-z_])_*$", "\\1",
                     x=normed)
    names(df) <- drop_usc
    df
}

This function does a few things:

1. it replaces spaces or punctuation with underscores⁷
2. it then converts all text to lowercase
3. Then we exert cultural imperialism and export to ascii⁹
4. finally, we replace multiple underscores with one (for tidyness, mostly)

dat2 <- normalise_names(dat)
names(dat2)

Much better. Little things like this add up to a better, more productive life.

Next, we need to remove the commas from the input doc (which have become hopelessly mangled). Again, we abuse iconv, this time on a particular column. My life would be much easier if I took the time to learn how to handle encodings ¹⁰

fix_price <- function(x) {
    nopunct <- gsub(",|", "", x=x)
    nums <- iconv(nopunct, "latin1", "ASCII", sub="")
}
dat2$price <- with(dat2, fix_price(price))

These preliminaries accomplished, we can move towards actually looking at the data

prop_df <- dat2 %>%
    mutate(date_of_sale=lubridate::dmy(date_of_sale_dd_mm_yyyy),
           not_full_market_price=as.factor(not_full_market_price),
           postal_code=as.factor(fct_explicit_na(postal_code)),
           vat_exclusive=as.factor(vat_exclusive),
           county=as.factor(county),
           price=as.numeric(price))
prop_df2 <- mutate(prop_df,
                   year=lubridate::year(date_of_sale),
                   month=lubridate::month(date_of_sale),
                   day=lubridate::day(date_of_sale))

prop_sample <- sample_frac(prop_df2, size=0.1)

We also take a ten percent sample, just to speed up the analysis ¹¹

So, when we do the simplest plot ever (a one-way summary of price), it looks pretty weird. Most of the density is in the one line near zero, and there's a long tail that goes up to 100mn. Who are these people buying houses for many millions of euros? More importantly, how can I become one of them?

The first thing to do when you find something weird like this is to take a look at the observations and see if they are different ¹². Let's sort the data by price and look at the addresses.

arrange(prop_df, desc(price)) %>%
    select(address, county, date_of_sale, price) %>%
    head()

OK, so the Cannon house thing must be more than one apartment. Otherwise things have gone even madder than I thought. Also Apt 204 the Alliance, and 182 the Elysian ¹³. Entire apartment blocks for 76-86 million makes more sense, however.

arrange(prop_df, desc(price)) %>%
    select(address, county, date_of_sale, price) %>%
    tail()

• there must be something really wrong with these houses.

It does tell us something about the tails¹⁴ of the crash, in that property was worth so little in Roscommon in 2015 that this happened. It may also be an error, as I don't know what kind of checking goes into this data.

However, it's interesting to note that there weren't that many sales. Even at 60k it's only just over 5% of the housing stock ¹⁵. That's not a huge amount, but given the lack of data for the boom, it's very difficult to make those judgements.

So the overall trend is pretty clear. The chart above shows median prices. For those of you who aren't stats nerds, the median is the middle value. If you laid out all the prices in a line, the median would be the middle of the line. This means that it's a reasonable summary of the distribution.

Really, you should look at the distribution itself, but aggregations such as the median are useful for plots and telling people about.

The mean, which is the sum divided by the length of the set of numbers is much higher. This is pretty normal in most sets of data because huge values at the top or bottom have more impact on the mean. Again, the same trend is pretty visible. The mean price in Dublin is 400k, which assuming a 10% deposit means that a household would need a combined income of approximately 102k.

This puts houses in Dublin far out of reach of ordinary earners, which is perhaps not surprising given the incredible inflation in asset prices we've seen since the financial crisis in 2008.

Note that I've filtered out the properties sold for more than 2mn, as these are mostly apartment blocks and groups of houses. These probably shouldn't be looked at for this analysis ¹⁶.

Looking closer at Dublin (and breaking down by postal code), we can see some more interesting patterns.

Prices are definitely trending up, but there are some large outliers across the board.

Like this former house owned by an executive in Dublin 20, for example. That place definitely doesn't reflect the average experience in that area. In general, I'm beginning to think that there are multiple housing markets in Ireland, not one. This makes sense as one can only compete against others in the same price range. This would suggest that I should regard the house buying denizens of South Dublin to be some kind of alien species. I do enjoy when a data-driven theory agrees with my priors.

• Long periods of not much activity, followed by sudden spikes.
• We should probably dig into some of the spikes to figure out what they mean.
• It's possible that they may be one overall transaction recorded as multiples (for instance, when an entire apartment block is sold to one developer, and it's recorded unit by unit).

So, above we've reviewed the property price register, and done a bunch of data cleaning, but not really accomplished that much overall. Part Two will cover some other interesting datasets ¹⁷, while future parts will cover geocoding the data and looking at spatial patterns, as well as some bayesian modelling with Stan.

Hopefully someone makes it thus far, at least ;).