Day to Day Danish Resident

In Denmark, it’s probably intuitive to say:

  1. We consume less energy on warm days between 0100 and 0300 hours.
  2. We consume more energy cold days between 1800 and 2000 hours.
  3. As temperature drops, the consumption increases. This increase is marked at 5 to 10 degrees.

But it’s really cool to watch it in 3D. Click on the image to get a 3D figure that you can play with :)

Consumption

Plot depicting Temperature vs Hours of the Day vs Electricity Consumption.

Advertisements
Posted in Uncategorized | Tagged , , , | Leave a comment

Getting Reliable Temperature Readings

I wanted to correlate temperature to household energy usage in Denmark. To do this, I needed to acquire reliable temperature measurements from Zealand but it has turned out to be a very hard thing to do.

  1. I logically began my journey at the Danish Meteorological Institute (DMI) but I could not find raw data that I could use. There was a lot of very useful information everywhere, but no historical data that I could download in CSV file (or in any other format).
  2. European Climate Assessment and Dataset (ECA&D): This was my first successful hit where I was able to acquire daily temperature means from a station called LANDBOHOJSKOLEN. This was a good beginning, but I’m looking for measurements that are taken every hour.
  3. Weather Underground (WU): In my search for a finer granularity (hours means instead of day means), my search has taken me to Weather Underground. They seem to have the temperature data measured at 30 minutes intervals, which was really good for my purposes. But they round the temperature values to the nearest degree which might introduce some in the correlation. In any case, I used the “comma separated file” functionality (Here is map of stations) to download temperature data by automating the process with a python script (I might have made it a bit too complicated, but it works).
import urllib2
import os.path
import datetime
import random
import time

WEATHER_PLACE="http://www.wunderground.com/history/airport/EKRK/%s/%s/%s/" \
 + "/DailyHistory.html?format=1"
WEATHER_PLACE="http://www.wunderground.com/history/airport/EKRK/%s/%s/%s/" \
 + "DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA&format=1"
WEATHER_UNITS="http://www.wunderground.com/cgi-bin/findweather/getForecast?setunits=english"
MAX_WAIT_TIME=1 #in seconds.

def dwGenerateDateDict( D1, D2 ):

 # Initialize dateDict
 numdays = abs(D1 - D2).days
 dateList = [ D2 - datetime.timedelta(days=x) for x in range(0,3) ]
 dateDict = dict(zip(dateList, [""]*len(dateList)))

 while ( len(dateList) > 0 ):
 print("Days left: " + str(len(dateList)))

 # Wait for a random amount of seconds
 secs = random.randint(0,MAX_WAIT_TIME)
 print("Waiting for " + str(secs) + " seconds.")
 time.sleep(secs)

 # Select a random date
 ranInd = random.randint(0,len(dateList)-1)
 rd = dateList[ranInd]
 del(dateList[ranInd])

 # Get the csv weather for the date
 urllib2.urlopen(WEATHER_UNITS)
 urlRes = urllib2.urlopen(WEATHER_PLACE%(rd.year, rd.month, rd.day))

 # Read rest into dictionary.
 dateDict[rd] = urlRes.read()
 print("Date : " + rd.strftime("%Y%m%d") \
 + ". Read : " + str(len(dateDict[rd])) + " Bytes")

 return (dateDict)

def dwGenerateCSVFile ( dateDict, filename ):

 fd = open(filename, 'a')
 keys = dateDict.keys()
 keys.sort()
 for date in keys:
 datestr = str(date)
 datedata = dateDict[date]
 timetype = datedata[1:datedata.find(",")] # CET or CEST
 # Remove first list of header
 datedata = datedata[datedata.find("
\n") + 7:]
 # Last separator to front
 datedata = datedata[-7:] + datedata[:-7];
 datedata = datedata.replace("<br />\n",
 "\n"+datestr+","+timetype+",")
 fd.write(datedata)
 fd.close()

if __name__ == "__main__":
 D1 = datetime.date(2011,9,27)
 D2 = datetime.date(2013,9,23)
 DD = dwGenerateDateDict(D1,D2)
 dwGenerateCSVFile(DD, "UnderGroundWeatherRoskildeAirport.csv")

I’m going to start trying different weather APIs and see if any of them can give me the historical information that I need with the granularity level that I want.  More to come…

Posted in Uncategorized | Tagged , | Leave a comment

Matlab: Putting labels on plot axis

I can’t just plot the X-axis with strings and expect the Matlab plot function to do what it is supposed to do (Which is label each X point with the string). The work around for this inconvenience is to use the XTick and XTickLabel. Found this useful stack overflow article that explains it all :)

x = 1:5;
y = rand(size(x));
plot(x, y, 'b')
set(gca, 'XTick',1:5, 'XTickLabel',{'A' 'B' 'C' 'D' 'E'})

Use the XTick to signal how many X axis elements there are. And use XTickLabel to signal the labels for each X axis element. In my case I only used XTickLabel.

 

Posted in Uncategorized | Tagged | Leave a comment

EcoIS is out!!!!

Checkout our new publication here. It’s all about how to normalize image taken of field plots. This is not only interesting because it can normalize hand-held photography, but it also has the potential to influence how images taken from Areal Unmanned Vehicles are processed.

Posted in Uncategorized | Tagged , , , , , , , | Leave a comment

ACM Ecological Data Workshop

These are the links to the new ACM ecological data effort :). This should be cool….

 

First ACM International Workshop on Multimedia Analysis for Ecological Data

2ND ACM INTERNATIONAL WORKSHOP ON MULTIMEDIA ANALYSIS FOR ECOLOGICAL DATA

This field is moving :)

Posted in Uncategorized | Tagged , | Leave a comment

R: Date to day of year.

This is very handy :). More discussion here.

strftime(today, format = "%j")
Posted in Uncategorized | Leave a comment

GIT create repositories on Linux Servers

Put an existing repo on a server (personal use):

1. Make special clone:

git clone --bare /LOCAL/PATH/project.git

2. Put clone on server:

 scp -r project.git USER@example.com:/REMOTE/PATH/

3. Test your setup:

 git clone ssh://USER@example.com/REMOTE/PATH/project.git

Shared repo from scratch

To share any git repo on a server you need to make sure of two things:

  1. Have the correct permissions. All objects in the repository must be read.write.
  2. And the repo objects must be owned by the “relative” group. All the people collaborating should be in the “relative” group. Go here and here for more information.

Make sure that git knows that you are sharing. Look at the –shared option for git-init. Since creating a shared git repository requires you to modify permissions on the server, you should create it from within the server. These are the steps you should follow for an empty repository.

1. ssh into server and make the git directory in the path of your choice.

ssh USER@example.com
cd /REMOTE/PATH/
mkdir project.git

2. Change the permissions to allow collaboration:

chown USER:GROUP project.git

* USER is your user.

* GROUP is the group where all your collaborators are.

3. Initialize a special bare shared git repository.

cd procject.git
git init --bare --shared=0660

* This will create a repository that lets everyone in the group GROUP push and pull.

4. Test your repository in your local machine

git clone ssh://USER@example.com/REMOTE/PATH/project.git

* To push the first time you should do a `git push origin master`

Shared from existing repository:

Follow the four previous steps.

1. In your LOCAL repository run

git remote add origin ssh://USER@example.com/REMOTE/PATH/project.git

* You might have to rename the remote to something else than ‘origin’ if you already have a remote.

2. In your LOCAL repository run

git push origin master
Posted in git | Tagged , , , , , , | Leave a comment