Valorin's Thoughts
Less sense than many, but more than most.
DTAG&Gears Step 2: Implementing the database
It has taken me ages to post this up, but hey, it is here now.
This is the third post in my series on implementing DTAG with Google Gears.
I have since released the Beta of DTAGv5, so you can simply go there and check it out. Also, feel free to look through my JavaScript/Gears code if you are interested in seeing how I have done things.
After getting the Manifest file working, and the general Gears recognition running, I set about setting up the database.
The first step is to create the thing:
db = google.gears.factory.create('beta.database', '1.0');
db.open('dtag_gears');
Surprisingly, that was it. No need to use users, passwords, or any other fancy "If db exists". Gears implements its security on a per-domain basis, so only scripts running within the same domain can access the db. If the DB does not exist, gears will create it.
The next question is predictable; "How do I create a table within the DB, without overriding existing data if it is in there?"
The simple answer is... remember your SQL: CREATE TABLE IF NOT EXISTS.
So, I follow the code above with these lines:
db.execute('CREATE TABLE IF NOT EXISTS dtag_pages' +
' (pageid INTEGER PRIMARY KEY,' +
' source LONGTEXT,' +
' info LONGTEXT,' +
' title VARCHAR(255),' +
' timestamp DATETIME)');
db.execute('CREATE TABLE IF NOT EXISTS dtag_settings' +
' (key VARCHAR(255) PRIMARY KEY,' +
' value VARCHAR(255))');
Done. I now have a database, which is created if it is new, which I can use locally.
Now, to access data within the DB.
This is one area in which Gears falls down, there are only very limited database functions. From the official database documentation, the classes which are used as as follows:
Database class
void open([name])
ResultSet execute(sqlStatement, [argArray])
void close()
readonly attribute int lastInsertRowId
ResultSet class
boolean isValidRow()
void next()
void close()
int fieldCount()
string fieldName(int fieldIndex)
variant field(int fieldIndex)
variant fieldByName(string fieldName)
The ResultSet is very limited. Although you can step through each row within the Result, you cannot go backwards or reset the pointer. Annoyingly, there also is no rowCount() to match the fieldCount(). This is a big problem for me. If you wish to know the number of results returned, and THEN parse/extract the data, it is impossible. You need to parse/extract WHILE counting the number of rows. Finally, you are required to close the ResultSet after use. This is annoying, but, something good programmers should do anyway :)
So, for example, to extract the settings into variables within DTAG, I have done the following:
var result = db.execute("SELECT * FROM dtag_settings");
while (result.isValidRow()){
gearsSettings[result.fieldByName("key")] = result.fieldByName("value");
addDebug("Loading Setting: "+result.fieldByName("key")+" = "+result.fieldByName("value"));
result.next();
}
result.close();
Thats basically it. Using Databases with Gears is a piece of cake, when you get your head around the simplistic API. The DB Gears uses is SQLite, so check that out for any SQL specifics.
In the next installment I will be going over how I keep the local database in sync with the server database.
On a side note, I have set up a Bug Tracker for DTAGv5, so if you find any bugs post them in there.
~Valorin
Fri, 07 Mar 2008 21:23:25 +1100
(link here/comments[0])
DTAGv5 +GoogleGears is live!
It has taken me a while, but the Beta of DTAGv5 +GoogleGears is now live. You can find it at http://dtag.valorin.net/. I still have a few blurbs to post about the GoogleGears implementation, but for now this will have to do.
Yup, I am lazy.
Anyway, DTAGv5 has been designed to easier to use then the previous one and to look nicer. The page is edited within one large text box, and the different things such as repeaters and links can be added into the text wherever is required.
So, go and have fun. DTAG is awaiting people to write in it and expand it :)
~Valorin
Wed, 20 Feb 2008 20:46:51 +1100
(link here/comments[0])
DTAG&Gears Step 1: Getting the Manifest file to work!
A while ago I posted about setting up Google Gears support with DTAG. I have just got the Manifest file to work so I figured I should post about how I have done it. If you are out there wondering how to use the Manifest file, this should be helpful.
The first step was to generate the manifest file, after following the tutorials provided by Google I came up with something which looks like this:
{
"betaManifestVersion": 1,
"version": "1.0001",
"entries": [
{ "url": "index.php"},
{ "url": "styles.css"},
{ "url": "scripts/dtag_gears.js"},
{ "url": "scripts/dtag_jue.js"},
{ "url": "scripts/gears_init.js"},
{ "url": "scripts/json2.js"},
{ "url": "images/bg_repeating.jpg"},
{ "url": "images/bg_rep_top.jpg"},
{ "url": "images/jue-loader.gif"},
{ "url": "favicon.ico"}
]
}
All of the files which the browser requires to load DTAG are included in there. The version number is there to tell the script when it has been updated, and to re-download each of the files.
The nest step is to actually get Gears to download it. After a bit of reading through this tutorial, I generated this code, which appeared to work:
var localServer;
var store;
var storeName = "dtagGearsBeta";
var manifestUrl = "manifest.json";
function dtagGearsStart(){
if (!window.google || !google.gears){
setStatus("GoogleGears not Installed!");
addDebug("Gears not installed.");
addDebug("Ending script.");
}else{
setStatus("GoogleGears Installed!");
addDebug("Gears installed");
addDebug("Loading Manifest file.");
localServer = google.gears.factory.create("beta.localserver","1.0");
store = localServer.createManagedStore("dtagGearsBeta");
store.manifestUrl = manifestUrl;
store.checkForUpdate();
setStatus("Loading Manifest...");
}
}
The above code works, however, I wanted to have an update status actively working, mostly so I could see it was updating live. Running on a local network makes the basic requests virtually instant, so a simple php time-waster was required;
for ($i = 0; $i < 100000000; $i++);
I added this to the manifest.json file, and then had to add some status updates to the JavaScript file. Hence a new function:
function updateStatus(){
if (store.updateStatus == 1 || store.updateStatus == 2){
addDebug(".");
setTimeout("updateStatus()", 1000);
}else if (store.updateStatus == 3){
setStatus("Load failed... try again later.");
addDebug("Loading Manifest file Failed!");
}else if(store.updateStatus == 0){
setStatus("Manifest Loaded successfully!");
addDebug("Manifest Loaded successfully!");
}
}
And called this just after the store.checkForUpdate();. After running this, I could clearly see a line of dots appearing in the debug section while it waited for the timewaster.php to finish. While this was happening, I still had full functionality in the foreground.
Thats it. Google Gears manifest file working.
I'll post the next step when I have done it: downloading the pages to be viewed offline.
I just have to say that, the
So, here are the complete files I have:
manifest.json
{
"betaManifestVersion": 1,
"version": "1.0009",
"entries": [
{ "url": "index.php"},
{ "url": "styles.css"},
{ "url": "scripts/dtag_gears.js"},
{ "url": "scripts/dtag_jue.js"},
{ "url": "scripts/gears_init.js"},
{ "url": "scripts/json2.js"},
{ "url": "images/bg_repeating.jpg"},
{ "url": "images/bg_rep_top.jpg"},
{ "url": "images/jue-loader.gif"},
{ "url": "favicon.ico"},
{ "url": "timewaster.php"}
]
}
dtag_gears.js
var localServer;
var store;
var storeName = "dtagGearsBeta";
var manifestUrl = "manifest.json";
function dtagGearsStart(){
if (!window.google || !google.gears){
setStatus("GoogleGears not Installed!");
addDebug("Gears not installed.");
addDebug("Ending script.");
}else{
setStatus("GoogleGears Installed!");
addDebug("Gears installed");
addDebug("Loading Manifest file.");
localServer = google.gears.factory.create("beta.localserver","1.0");
store = localServer.createManagedStore("dtagGearsBeta");
store.manifestUrl = manifestUrl;
store.checkForUpdate();
setStatus("Loading Manifest...");
updateStatus();
}
}
function updateStatus(){
if (store.updateStatus == 1 || store.updateStatus == 2){
addDebug(".");
setTimeout("updateStatus()", 1000);
}else if (store.updateStatus == 3){
setStatus("Load failed... try again later.");
addDebug("Loading Manifest file Failed!");
}else if(store.updateStatus == 0){
setStatus("Manifest Loaded successfully!");
addDebug("Manifest Loaded successfully!");
}
}
function addDebug(line){
document.getElementById("gearsDebug").innerHTML+= line+"<br />";
}
function setStatus(status){
document.getElementById("gearsStatus").innerHTML = status;
}
~Valorin
Wed, 16 Jan 2008 12:14:49 +1100
(link here/comments[1])
Genesis of a Puppet
It's been a while since I last posted something up here. So, here is the long awaited post... I finally found something interesting to talk about :)
I have just spent this weekend making a puppet, and here is a pictorial story of the genesis of the puppet.
The first step was to model the head in foam. After much playing around we decided upon the following:
I pinned some eyes onto the head, to get some idea of how it will look, and then added some fur to enhance the look to make sure it will look good. Then I had an awesome idea :)
So the puppet was born.. Well, its character atleast. Zoot MKII, a Sax playing puppet!
The next step was to work out the fur for the head. After multiple trials with some fabric, and with that came another idea. Add some ears! So, we got the fur for the head cut out, and sewed up, and gave it some ears. *It is worth nothing that the fur was not stuck on, to make it easy to sew on the ear insert and the body.
After that, the body was developed and sown onto the head. This is basically just two pieces of fabric with hands on either side and joined at the top to the head. The bottom is open wide to allow the puppeters arms to go in easily. (And upper body if required).
Inside the head I had to put some foam supports, to hold the puppeters hands into the head. So it doesn't fall off when doing a long show.
Finally, we cleaned up the puppet, and added the eyes. From the moment the eyes went on, he was born.
Thats it, the Genesis of a Puppet.
And the mess I made...
So, two puppets made. One to go :) (in the first batch)
I promise, next time I'll post more in depth and useful pictures if you actually want to make one and don't know how.
Hope you enjoyed this little insight into my world!
~Valorin
Sun, 13 Jan 2008 21:50:38 +1100
(link here/comments[0])
Designing Google Gears interaction for DTAG
Firstly, some useful explanations:
Google Gears is a tool developed by Google which allows websites to work offline. It basically provides its own mini-webserver, complete with a database, and the JavaScript tools required to add and manipulate the data within the local-store.
http://gears.google.com/
DTAG, or Decision Tree Adventure Generator, is a game I wrote which is basically a modern version of those Choose Your Own Adventure books. You read through what others have written, and when you hit a page which does not exist, you write it yourself. The aim is that it grows with the community of users.
http://dtag.valorin.net/
Now you know what they both are... My plan is to revise DTAG into a new streamlined JUE version, and include Google Gears functionality on top of that. I am aiming on documenting the process here, and up first is the Designing Phase.
Step One: Non-Gears revisions
Currently DTAG is a static server-side each page per load design. This is one of the flaws in DTAG, as it does not flow like a story. The aim is to revise this design to make it flow more. (Note: when I use 'page', I am talking about a page in the book/story, defined by choices at the end of the page.)
- The first page is loaded up, and choices are below.
- User clicks on one of the choices
- The choice clicked is highlighted
- The chosen page is appended on the bottom of the story (using JUE)
- The user reads through the next page
- go to 2.
This functionality needs to be replicated across both the client, in JUE, and the server, in PHP. In theory this should be simple, it just relies on keeping a history/log of the users actions.
Step Two: How do we add Gears into the Mix?
Here many questions are raised as to the design of the system, for the implementation of gears...
- Does Gears download the Entire database pages, or just pages located around the user (proximity with depth limit)?
- Are the users allowed to Edit pages? If so, will this mean sending HTML and user-markup data, to be stored client-side? Or allow the JS to fully render the user-markup
- Are users allowed to create new pages? Will this save the data and create it on the server when synced, or allow the user to view and continue on this page?
- Will DTAG automatically sync on- and off- line? Or will it be done manually for some or all?
- Why am I doing this at all?
After a bit of thought, here are my answers:
- The database will be incrementally downloaded, in the background. The only data sent on user-request is the requested page
- Users can read-only existing pages
- Users can create new pages, but they are not usable until synced with the server
- DTAG will notify the user that "there are unsaved pages in the system", and if an online presence is detected, it shall prompt the user "click here to publish these pages to the server"
- I am doing it to learn Gears and enhance DTAG.
I am going for a basic implementation of DTAG+Gears. It is designed to allow users to read DTAG offline, and add pages. Nothing complicated - that is a task for another day.
Step Three: Defining the process and requests involved
What needs to be defined is how and when Gears will talk to the server, what it sends, and what it receives. This is a simplified use-case of how it will work, and all the defined numbers will be adjusted until they are at optimal levels.
- User tries to load up a page
- Client sends request for page X, with the datestamp from the local page copy
- Server responds with either "Local copy is latest", or a new copy of the requested page
- Client displays the page to the user
- Client requests data for each of the children of the current page (to cache it for quick-load, and populate the database)
- Server sends back data, if newer
- Client stores data
- Client waits 10 seconds
- Client sends incremental-check-last-id to the server
- Server sends the next 20 pages to client
- Client saves data & updates the incremental-check-last-id
- Client waits 10 seconds
- Client sends the last-datestamp-checked to the server
- Server returns the oldest 20 pages which are newer then the last-datestamp-checked date
- Client saves data & updates last-datestamp-checked
- Loop to 8
The aim with the above process is to get the entire DTAG database locally, without downloading the lot in one hit. For a small database this is not a problem, but for a large database downloading the entire thing is bad. Not just for bandwidth, but also due to the processing required on the client-side to store the data. The delay of 10 seconds, and page-count of 20 will need to be tweaked, I just pulled random numbers out of the sky.
Also, it is important to keep the local database up to date, rather then just filled with a snapshot of how it was when the user first downloaded the server. Unfortunately, the incremental, and last-datestamp updates will download duplicate data (unless existing ids and datestamps are known on-server?). This is unavoidable, yet will only happen on some pages.
That is basically all of the DTAG+Gears design, in a broad sense. The next phase is implement the design, somehow :) I will post the next step up here as I work through it.
I hope you found this interesting and informative. If you have any suggestions, questions or comments, please let me know.
~Valorin
Sun, 28 Oct 2007 13:00:49 +1100
(link here/comments[0])
