Why Did Weatherlogics Create a Climate Portal?
Earlier this week we announced that Weatherlogics had launched its Climate Portal. This portal allows users to retrieve climate data, records, and normals for any location in Canada. You can see full details about the climate portal, and its capabilities, by visiting the website at https://climate.weatherlogics.com
With the launch of this portal you may be wondering why Weatherlogics chose to build it in the first place? Many people assume climate data are already available and therefore we are just duplicating what already exists. While it’s true that the underlying data are already available – they’re just that, raw, underlying data. To use the data properly requires painstaking manipulation and analysis. Our database makes these climate data much more accessible and has three critical advantages:
We will briefly describe each of these three advantages below:
When you retrieve historical climate data for a location, you want to maximize the period of record. However, in Canada this is difficult because weather stations have changed locations and names many times. For example, if you wanted to get Winnipeg’s climate history, all weather stations would have to be identified and downloaded. Afterwards, a method to combine stations into a single dataset would have to be established. In Winnipeg, there are a total of 41 stations to choose from, but Weatherlogics has identified six as primary weather stations to combine:
- St. John’s College: 1872-1938
- Winnipeg Richardson Intl A: 1938-2008
- Winnipeg Richardson AWOS: 2008-2013
- Winnipeg A CS: 1996-2020
- Winnipeg Intl A: 2013-2020
- Charleswood2: 2004-2020
As you can see, it is not easy, given that these stations overlap in time, have varying levels of data quality, and some only contain certain variables. For example, Charleswood2 is mainly used for snow measurements, while Winnipeg Intl A is mainly used for hourly weather conditions. Similar caveats apply to the other stations.
In theory, Winnipeg’s climate history extends from 1872 to present, but only if you can properly combine the six above stations. Luckily for you, the Weatherlogics database has already completed the complex process of combining (or joining) stations. Not only has this process been done for Winnipeg, it has also been completed for more than 700 active weather stations across Canada.
The table below shows some examples of how many stations needed to be joined at some locations across Canada.
|Location||First Year||Last Year||Number of Stations Joined|
|Thunder Bay, ON||1877||Present||7|
|Edmonton City, AB||1880||Present||5|
|Toronto Island, ON||1905||Present||5|
If our discussion about completeness hasn’t already got you concerned about trying to assemble climate histories yourself, just wait, because that’s just the tip of the iceberg. Let’s assume you managed to combine all the stations together – great, you’re done…actually not so fast. Unfortunately, climate data in Canada are minimally quality-controlled (QC). Most data only undergo limited automated QC, which often misses critical errors. Furthermore, much of the historical climate data were manually input, so input errors are more common than you might think. For this reason, Weatherlogics has instituted a rigorous quality-control process which identifies erroneous values and fills missing data.
The first step in QC is checking a data point to see if it’s valid. You can see a list of all our QC methods on the methods page of the climate portal. This process identifies all sorts of errors, ranging from unrealistic values to inconsistent data. Once the QC is complete, missing data are detected. If a value is missing, we always attempt to fill it. It can be filled in a variety of ways, ranging from using hourly data as an estimate, to using another nearby station. The reason we fill missing data is because many statistics cannot be calculated if data are missing. Monthly climate values, like total precipitation, cannot be computed if even a single data point is missing.
You might think QC is not that big a deal, but actually it is. Let’s say you want to know the highest wind gust ever recorded in Estevan, SK or Kelowna, BC. In the case of Estevan, the value that might come up is 298 km/h on July 24, 2017. In Kelowna, the value that might come up is 276 km/h on June 14, 2015. In both cases, these are erroneous values caused by sensor errors. In another situation, you might be looking for the wettest month on record in Manitoba. Your search might reveal that Carberry had 409.8 mm in November 1995, making it the wettest month. However, this is actually an erroneous total, as a quick comparison with another station in Carberry shows that there was only 49.6 mm in November 1995. If you used these data for something important, your conclusions could be completely wrong. These few examples show how QC is critical to ensuring the integrity of the data. That’s why we have associated a flag with each data point. Even if we think a data point might erroneous, but aren’t entirely sure, we’ll mark it suspect, so at least you know to look into it further.
After completing the climate history and performing quality control, we have a database filled with great data. However, unless all you’re looking for is past weather observations, the data itself isn’t that useful. It is far more powerful when it can be searched for specific records or if normals can be calculated. The ability to search our database for specific information is the third critical advantage of our portal.
One spin-off of having a database with complete climate histories is that our climate records and normals are also complete. If you view climate records for Winnipeg they are often based only on Richardson Intl A, which means they will only be using data from 1938-2008. This means all records prior to 1938 and after 2008 are missing. Therefore, you aren’t actually seeing the records, because a large amount of the data are missing. The same can be said for normals. Since Richardson Intl A ends in mid-2008, the Winnipeg normals for 1981-2010 would only be from 1981-2007 if based on that station. Since Weatherlogics has the complete underlying data, our records and normals are also complete!
There are two primary ways to search the database: through the climate portal website or using APIs. The website allows many basic searches for records and normals. However, due to the difficulty in customizing the user interface for all possibilities, some records and normals can only be retrieved using APIs. The APIs allow more complex searches than can be done with the website. Here are some examples of unique queries?
- When was the longest blizzard?
- What is the earliest or latest thunderstorm day?
- Which month has the most 33 C days?
- What is the driest first half of July?
- What was the wettest month of all time?
- What was the highest dewpoint in July?
- What was the lowest relative humidity in April?
- When were the most consecutive days below -30 C?
These are just a few examples of literally hundreds of possible searches. Even though the APIs are quite thorough, some very complex or large queries require one of our meteorologists to write a custom script for the data you require. Contact us if you run into such a situation.
In the previous few sections we’ve outlined the main advantage of our climate database. However, there are other advantages too. Since our database is updated hourly, the latest data are always available. This also means that our 1991-2020 normals will be available immediately on January 1, 2021. You don’t even have to wait until 2020 to get the latest normals, the 1990-2019 normals can already be searched!
As part of our ongoing updates, we also store a lot of data that isn’t available anywhere else. Some examples of this include hourly precipitation, sea-level pressure, and hourly wind gusts (to name a few). If you want access to these specialized datasets, just contact us!
The sky’s the limit when it comes to possible uses of our climate database. However, a few obvious cases come to mind. Television newscasts often focus on the weather and using records or normals is a good way to put the current weather in context. Our climate records are also a great way to wow viewers with interesting stats. There are other obvious use cases too, like using our data to help inform insurance decisions, or integrating our APIs into apps. Scientists can also benefit from our quality-controlled data sets. While we’ve noted a few specific industry-based examples, our data are open to anyone, so don’t be shy, take a look today!
While climate data might seem rather benign at first, this post has shown how tricky it can be. We’ve gone into quite a bit of detail here about what makes our climate data unique. If you have more questions, there’s plenty of information under the Help section of the climate portal. You can also contact us if you need assistance.
Starting searching now: https://climate.weatherlogics.com