{"assets":[],"author_link":"author\/william_edwards\/","author_name":"Will Edwards","cat":"LD #27","categories":["LD #27"],"comments":[],"epoch":1378037520,"event":"LD27","likes":7,"metadata":{"p_key":"80206","p_author":"Will Edwards","p_authorkey":"0","p_urlkey":"289216","p_title":"Running SQL in your browser","p_cat":"LD #27","p_event":"LD27","p_time":"1378037520","p_likes":"7","p_comments":"0","p_status":"WAYBACK","us_key":null,"us_name":null,"us_username":null,"event_start":"1377216000","event_key":"18","event_name":"LD27"},"source_url":"2013\/09\/01\/running-sql-in-your-browser\/","text":"<p><a href=\"http:\/\/www.ludumdare.com\/compo\/ludum-dare-27\/?action=preview&amp;uid=10313\"><img alt=\"\" class=\"alignnone\" height=\"491\" src=\"http:\/\/www.ludumdare.com\/compo\/wp-content\/compo2\/273708\/10313-shot0.jpg\" width=\"680\"\/><\/a><\/p>\n<p>As a key part of my Ludum Dare entry <a href=\"http:\/\/www.ludumdare.com\/compo\/ludum-dare-27\/?action=preview&amp;uid=10313\" target=\"_blank\">The NSA\u2019s Where\u2019s Snowden?<\/a> game I wanted you \u2013 the elite NSA Analyst \u2013 to be able to <em>hack<\/em> the game world.\u00a0 Imagine you could put your agents onto flights, or fiddle with the flight manifests so that targets were refused boarding or even arrested\u2026 and how would you do this?\u00a0 I decided I wanted you to have SQL access to the world\u2019s flight plan database\u2026<\/p>\n<p><code>SELECT T.* FROM MANIFESTS AS M, MANIFESTS AS M2, TARGETS AS T WHERE T.PASS = M.PASS AND M.PASS = M2.PASS AND M.FLIGHT = 'AF-321' AND M2.FLIGHT = 'IB-794';<\/code><\/p>\n<p>But I couldn\u2019t find any working SQL engines for Javascript!\u00a0 The only thing I could find was <a href=\"http:\/\/code.google.com\/p\/trimpath\/wiki\/TrimQuery\" target=\"_blank\">TrimQuery<\/a>, but I couldn\u2019t get it to work and its quite old and not actively developed.\u00a0 There\u2019s are also a few LINQ-likes but they are not really SQL.<\/p>\n<p>Now SQL is the lingua franca of structured data querying so I was surprised at this sorry state of affairs.\u00a0 Whilst LINQ is nice (because of the VS tooling!) I feel that a lot of the LINQ-likes are driven by hipster cargo-culting and that a solid SQL that can query conventional JS objects and arrays would be a powerful and useful thing.<\/p>\n<p>So I set out to build an SQL engine in Javascript (<a href=\"https:\/\/github.com\/williame\/ludum_dare_27_snowden\/blob\/gh-pages\/sql.js\" target=\"_blank\">sql.js<\/a>).\u00a0 It was perhaps more fun to develop than the other parts of the game and more fun to examine than the game is to play <img alt=\":)\" class=\"wp-smiley\" src=\"http:\/\/ludumdare.com\/compo\/wp-includes\/images\/smilies\/simple-smile.png\" style=\"height: 1em; max-height: 1em;\"\/>\u00a0 This blog post describes how it works (and how it can be done better):<\/p>\n<p><strong>How to use sql.js<\/strong><\/p>\n<p>It consists of a single function SQL().\u00a0 The parameters are:<\/p>\n<ul>\n<li>the SQL statement,\u00a0 e.g. \u201cINSERT INTO \u2026\u201d<\/li>\n<\/ul>\n<ul>\n<li>the definition of the tables, which is a Javascript object acting as a dictionary.\u00a0 Each table is itself an object, detailing the column names and specifying any constraints on them (e.g. if they must be unique or can be null) and optionally a description.\u00a0 E.g.\n<pre><code>{\r\nairports: {\r\n  code:{description:\"IATA code\",unique:true,notNull:true,},\r\n  name:{notNull:true,},\r\n  country:{notNull:true,},\r\n  category:{description:\"level of CIA infiltration\",},\r\n  lat:{description:\"latitude (decimal degrees)\",},\r\n  lng:{description:\"longitude (decimal degrees)\",},\r\n},\r\nairlines: {\r\n  code:{unique:true,notNull:true},\r\n  name:{},\r\n  country:{},\r\n},...<\/code><\/pre>\n<\/li>\n<li>table access; an object where each table name that the user can update has a boolean e.g.\n<pre><code>(\r\nairports: false,\r\nflights: true,...<\/code><\/pre>\n<p>Any table name that is omitted is read-only.<\/p><\/li>\n<li>The data; an object where each table is a key and each table is an array of objects e.g.\n<pre><code>{\r\nairports: [\r\n  {code:\"<\/code>LHR\",name:\"London Heathrow\",...\r\n],\r\nmanifests: [\r\n  ...<\/pre>\n<\/li>\n<li>params.\u00a0 This object allows you to override things like setting the default row count LIMIT.<\/li>\n<\/ul>\n<p>If the SQL cannot be parsed or if the table data does not have the expected fields then it throws an Error() exception.<\/p>\n<p>Otherwise it returns an object with a columns key and a rows key.\u00a0 The columns are an array of strings which are the column titles.\u00a0 Rows are an array of arrays.<\/p>\n<p><strong>Parsing SQL<\/strong><\/p>\n<p>To parse a language you often use a parser generator like lex\/flex yacc\/bison.\u00a0 There are parser generators for Javascript too \u2013 jison and peg and so on \u2013 but I haven\u2019t used them and, to be honest, its quicker to just write our own parser from scratch.<\/p>\n<p>To parse SQL we can eat the statement a token at a time.\u00a0 A token might be a number, an identifier or a symbol like an opening brace.<\/p>\n<p>Normally when I hand-roll a parser \u2013 its a bit of a recurring hobby of mine \u2013 I write the tokeniser to see what the next character is, and then based on whether that is a number, a quote mark or the beginning of an identifier or so on loop through until the token closes.\u00a0 In sql.js however, I took a messy shortcut.<\/p>\n<p>When asking for the next token, I pass into the tokeniser the list of terminating strings e.g. when consuming table names in a FROM clause I\u2019d ask for the next token (which I expect to be a table name) which can be followed by a comma, the word AS or the word WHERE or a semi-colon.\u00a0 If my engine supported it, I could just add those additional possible keywords that follow a table name in the FROM clause e.g. ORDER, LIMIT and HAVING and JOIN and so forth.<\/p>\n<p>This meant that the next() function only had to do a Javascript string search \u2013 indexOf() \u2013 to find the nearest terminator.\u00a0 However, this approach does have limitations e.g. it doesn\u2019t properly parse strings (if the terminator occurs in the string the indexOf() will incorrectly find it) and words that contain keywords (the Airport table namehas OR in it, for example).\u00a0 By the time you\u2019ve added code to address this, you end up with a proper tokeniser anyway.\u00a0 So in hindsight I should have doubled my line-count and just gone with a classic tokeniser in the first place.<\/p>\n<p><strong>Executing SQL<\/strong><\/p>\n<p>Once the query is parsed and the array of table names built and, its just to execute the query.\u00a0 And this is surprisingly simple using the <a href=\"http:\/\/en.wikipedia.org\/wiki\/Cartesian_product\" target=\"_blank\">Cartesian Product<\/a>.\u00a0 If table A had two rowsand table B had two rows then the Cartesian Product would be:<\/p>\n<pre><code>SELECT A.V, B.V FROM A, B;\r\nA.V | B.V\r\n1   | 1\r\n1   | 2\r\n2   | 1\r\n2   | 2<\/code><\/pre>\n<p>No surprises there.\u00a0 But what if we add a WHERE clause?<\/p>\n<pre><code>SELECT A.V, B.V FROM A, B WHERE A.V = 2;\r\nA.V | B.V\r\n2   | 1\r\n2   | 2\r\n<\/code><\/pre>\n<p>We can get that by first generating the Cartesian Product \u2013 simple by iteration and recursion \u2013 and then filtering the rows by evaluating the WHERE clause against each line!<\/p>\n<p>This is thoroughly inefficient.\u00a0 If there are 100 rows in one table and 100 rows in another then the Cartesian Product is (100+100)^2 = 40,000 rows!\u00a0 It would be much more efficient to evaluate \u2013 and cull \u2013 candidate rows as soon as possible e.g. before recursing into every B row first check if A.V is 2.<\/p>\n<p>In the NSA game it\u2019s easy to be evaluating a Cartesian Product that is hundreds of thousands long, and this can take several seconds.<\/p>\n<p>The engine supports SELECT, INSERT, UPDATE and DELETE.\u00a0 It doesn\u2019t support the JOIN keyword but it does let you join in the WHERE clause (as described above).\u00a0 It doesn\u2019t support IN.\u00a0 It additionally has SHOW TABLES, DESCRIBE table and CHECK table which provide utility.\u00a0 Because of the needs of the game there is also some rudimentary access control so the \u2018hacker\u2019 can\u2019t really hack \u2013 and break \u2013 the game itself e.g. rescheduling or deleting flights and other havoc.<\/p>\n<p>I made the whole thing case-insensitive but that could easily be removed for data values and comparisons themselves.<\/p>\n<p>I added support for literals but the only literal I added was \u2018NOW\u2019.\u00a0 I treated all dates as strings and this allowed simple Javascript string comparison of dates.\u00a0 I didn\u2019t support any functions, but this would be very easy to add too.<\/p>\n<p><strong>Improving performance<\/strong><\/p>\n<p>A proper SQL query planner (the code that decides the order that table rows are evaluated) has to understand the cost of retrieving rows (from disk and so on), but in Javascript RAM access speed is fairly equal and without indices you need full table spans so its just to filter tables by the parts of the WHERE clause that reference them as greedily as possible.<\/p>\n<p>Each table is an array of objects.\u00a0 It would be straightforward to support arrays of arrays, and objects of arrays and objects of objects.\u00a0 Using objects as the collections themselves would effectively be a primary key \u2013 an index!\u00a0 Knowledge of this could dramatically speed up joins.<\/p>\n<p>The engine turns the incoming SQL into a chain of closures.\u00a0 TrimQuery, by comparison, turns the SQL statement into a Javascript fragment that is then eval()ed.\u00a0 I am not sure that is faster; doesn\u2019t that cause the Javascript VM itself to abandon a lot of cached JITed code?\u00a0 But it may be faster to instead compile the SQL down to a simple VM.<\/p>\n<p>If you are running the same query again and again it would certainly help to \u2018compile\u2019 or \u2018prepare\u2019 the SQL statement once and have a way to run it again and again against data<\/p>\n<p>sql.js is surprising short given what it can do, and surprisingly simple.\u00a0 I found it easier to write SQL statements myself to work out what was happening in the model than to use array.filter() once joins were involved.\u00a0 With a better tokeniser and an expanded grammar and a simple query planner it could be widely useful.<\/p>\n<p><a href=\"https:\/\/github.com\/williame\/ludum_dare_27_snowden\/blob\/gh-pages\/sql.js\" target=\"_blank\">sql.js<\/a> is doubtlessly more interesting than actually <a href=\"http:\/\/williame.github.io\/ludum_dare_27_snowden\/index.html\" target=\"_blank\">playing the game<\/a> \ud83d\ude09<\/p>","time":"September 1st, 2013 12:12 pm","title":"Running SQL in your browser","title_was_empty":false}