{"id":5,"date":"2015-09-11T12:05:31","date_gmt":"2015-09-11T12:05:31","guid":{"rendered":"http:\/\/acquireforensics.com\/blog\/?p=5"},"modified":"2016-01-27T09:31:17","modified_gmt":"2016-01-27T09:31:17","slug":"sqlite-database-structure","status":"publish","type":"post","link":"https:\/\/www.acquireforensics.com\/blog\/sqlite-database-structure.html","title":{"rendered":"Sqlite Database Structure and File Format"},"content":{"rendered":"<p>Sqlite has become the favorite database management system of developers since it supports on different platforms such as iOS, Android, etc. Sqlite is an in-process library management, is fast and simple. As the name suggests \u2018Lite\u2019, it is a lightweight database and hence applicable for all applications. It has several bindings with the programming languages such as, C, C++, Java, Delphi, etc. and the source code is available for the users from the public domain thus, makes effortful for both private as well as commercial purposes. This session is all about bringing an idea about the structure and the file format of the Sqlite database. Let us start the discussion with the structure of the database.<\/p>\n<h2>Sqlite Database Structure<\/h2>\n<p><a href=\"https:\/\/www.acquireforensics.com\/blog\/wp-content\/uploads\/2015\/09\/database1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-14 size-full\" src=\"https:\/\/www.acquireforensics.com\/blog\/wp-content\/uploads\/2015\/09\/database1.png\" alt=\"Sqlite File Format\" width=\"450\" height=\"480\" \/><\/a><\/p>\n<p>The above figure describes the block diagram showing the Sqlite database components and their relationship.<\/p>\n<p>Let us grab an idea of each components from the below segment.<\/p>\n<p><u>Interface<\/u><\/p>\n<p>The library of Sqlite is implemented using the functions; <strong>main.c<\/strong>, <strong>legacy.c<\/strong>, and <strong>vdbeapi.c. <\/strong>In-order to avoid the collision with other application, Sqlite library starts with <strong>Sqlite3<\/strong>.<\/p>\n<p><u>Tokenizer<\/u><\/p>\n<p>Tokenizer breaks downs the strings and separates the token then, passes the tokens to the parser.<\/p>\n<p><u>Parser<\/u><\/p>\n<p>The parser is generated with the Lemon LALR(1) parser generator. Parse assigns meaning to the token.<\/p>\n<p><u>Code Generator<\/u><\/p>\n<p>Code generator generates virtual machine codes for the SQL statements thus, enables to satisfy the request made by the user. There are many files in the generator, which helps to generate the codes such as, <strong>build.c<\/strong>,<strong> delete.c<\/strong>,<strong> attach.c<\/strong>,<strong> expr.c<\/strong>,<strong> insert.c<\/strong>, <strong>pragma.c<\/strong>, <strong>select.c<\/strong>,<strong> auth.c<\/strong>, etc.<\/p>\n<p><u>Virtual Machine<\/u><\/p>\n<p>The virtual machine under Sqlite database structure runs the code generated by the code generator. It has its own header file called; <strong>vdbe.h <\/strong>and the entire machine is stored in single <strong>vdbe.c <\/strong>file.<\/p>\n<p><u>B-Tree<\/u><\/p>\n<p>The database follows B-tree structure and different B-trees are meant for index as well as the tables. All these trees are stored under one disk.<\/p>\n<p><u>Page Cache<\/u><\/p>\n<p>Information is seen in fixed chunks and the B-trees module request this data or information. The page cache helps to attain the reading, writing, etc. of the chunks.<\/p>\n<p><u>OS Interface<\/u><\/p>\n<p>To get interfaced with the OS, Sqlite uses abstraction layer and it is defined in <strong>os.h<\/strong>. The implementation may vary according to the Operating Systems.<\/p>\n<p><u>Utilities<\/u><\/p>\n<p>The <strong>util.c <\/strong>contains the memory allocations and the routines of the string.<\/p>\n<p>All the reads and writes of Sqlite database structure are done to an ordinary disk and since all the transactions follow ACID property, failure or power loss does not affect the transaction processing. Sqlite is a self-contained, serverless database system engine, has a cross-platform file format. The detail of its file format is discussed in the coming up sessions. And that will give you an idea on the file format of Sqlite.<\/p>\n<h2>Sqlite File Format<\/h2>\n<p>The database is carried in a single file called \u201cmain database file\u201d and there is a second file for storing the additional information that may be added during the transaction process called as, \u2018rollback journals\u2019 or write-ahead log file. Suppose, suddenly the system crashed then, rollback journal or write-ahead log file that contains critical state information is used to restore the main database file. In such situations, rollback journal are called as, \u201chot journals or \u201chot WAL file\u201d; contains necessary information for bringing back the database.<\/p>\n<p><strong><u>Pages<\/u><\/strong><\/p>\n<p>The database file consists of more than one page and the size of a page depends on the power of two between 512 and 65536. The page numbering starts from 1 and the maximum number is (2<sup>31<\/sup>\u00a0&#8211; 2). All the pages in the main database have a single use either;<\/p>\n<ul>\n<li>Lock-byte page<\/li>\n<li>Freelist page<\/li>\n<li>B-tree page<\/li>\n<li>Pointer map page<\/li>\n<li>Payload overflow page<\/li>\n<\/ul>\n<p>Remember, before modifying the database page, the unmodified contents will be written to WAL file. The first two pages are only described here. For more details about Sqlite file format, you can refer sqlite.org.<\/p>\n<p><strong><u>Database Header<\/u><\/strong><\/p>\n<p>The header is comprised in the first 100 bytes.<\/p>\n<p><u>Header format<\/u>:<\/p>\n<p><a href=\"https:\/\/www.acquireforensics.com\/blog\/wp-content\/uploads\/2015\/09\/data.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15 size-full\" src=\"https:\/\/www.acquireforensics.com\/blog\/wp-content\/uploads\/2015\/09\/data.png\" alt=\"Sqlite Database Structure\" width=\"650\" height=\"841\" \/><\/a><\/p>\n<p><strong><u>Lock-Byte Page<\/u><\/strong><\/p>\n<p>During the study of Sqlite file format, It is single page of database file that contains bytes at offsets between 1073741824 and 1073742335. A database file less than 1073741824 bytes contain no page but, file larger than 1073741824 has one such page. It is used by OS and Sqlite does not use this page.<\/p>\n<p><strong><u>Freelist Page<\/u><\/strong><\/p>\n<p>Sometimes information may be deleted from the database pages making the page unused. These unused pages are stored in freelist and is reused when additional pages are required, if any.<\/p>\n<p><em>How are the files stored in the Sqlite database?<\/em> <em>Which extension is followed?<\/em> The following session will discuss that.<\/p>\n<p><strong>.db <\/strong><\/p>\n<p>The primary database file of Sqlite is stored as .db file. Moreover, <strong>.db3 <\/strong>file is the primary database file of Sqlite version 3.<\/p>\n<p><strong>.db3-journal<\/strong><\/p>\n<p>The .db3-journal file is used to roll back the database.<\/p>\n<p><strong>.sqlite<\/strong><\/p>\n<p>The .sqlite file is created default whenever, a database is created.<\/p>\n<p><strong>.sqlitedb<\/strong><\/p>\n<p>When the database stores contacts for devices such as iPhone, iPad, etc. the .sqlitedb files are created.<\/p>\n<p><strong>.sqlite-shm<\/strong><\/p>\n<p>This file is used when Sqlite database works in WAL active mode.<\/p>\n<p>.<strong>sqlite-wal<\/strong><\/p>\n<p>These files are also deployed when Sqlite functions in a WAL active mode. The Write Ahead Log file stores the data generated by Write Ahead options.<\/p>\n<p>There are even other file systems or formats used by the Sqlite still, some of the important ones are mentioned in the above sessions. The evolution of Sqlite database engine has marked a remarkable growth in the development of applications. This database is even used in deploying some software products as well, thus making the favorite database engine for all enterprises.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sqlite has become the favorite database management system of developers since it supports on different platforms such as iOS, Android, etc. Sqlite is an in-process library management, is fast and simple. As the name suggests \u2018Lite\u2019, it is a lightweight database and hence applicable for all applications. It has several bindings with the programming languages [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":168,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[],"class_list":["post-5","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-file-format"],"_links":{"self":[{"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/posts\/5","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/comments?post=5"}],"version-history":[{"count":2,"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/posts\/5\/revisions"}],"predecessor-version":[{"id":162,"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/posts\/5\/revisions\/162"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/media\/168"}],"wp:attachment":[{"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/media?parent=5"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/categories?post=5"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.acquireforensics.com\/blog\/wp-json\/wp\/v2\/tags?post=5"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}