InMemory Database

A memory database is a database that is loaded fully into memory, and managed by a controller process similar to an on-disk database management system. Their are many challenges that arise when implementing CRUD in a memory database, including transaction handling, data synchronization with the data source, and data updates against the data source. To start with, many of the memory database products that are out there, require the application developer to manually load a database into the memory store. This puts a big responsibility on every application that must use the in-memory database. Every application must contain code specifically for loading the database into the memory database, which can amount to a significant amount of coding. Another approach taken by most of the memory database products is to force the application developer to register listeners in order to listen to changes to data in the memory database. The developer is then responsible to figure out and code the logic to update the proper tables and rows based on the changes dispatched by the in-memory database, this can be very complex and challenging, depending on the type of information that gets received from the event notifier. The other major shortcoming of many of the in-memory databases is their inability to provide transaction management functionality. This is a huge deal for applications that depend heavily on transactions, including financial and other similar types of applications. To resolve the first issue of relieving the developer from having to manually load the database into the in-memory database, an in-memory database can implement functionality to allow the developer to simply pass a JDBC or Ado.NET connection string, and the in-memory database is then able to discover the schema of the database by connecting to it using the connection string, and calling the various schema discovery methods that are provided by both interfaces. This solution is extremely simple to implement, and can have huge impact on the popularity on the in-memory database product. By implementing that logic inside the in-memory database code, application developers that use that product are no longer required to write code specifically to load the database into that product. By resolving the issue of loading the database within the in-memory database product, the second issue of updating the database with any changes in the memory database become much more manageable. Because the in-memory database can now connect and talk to the on-disk database using JDBC or ADO.NET, it can generate the required SQL syntax to update the database directly without having to notify the application listeners to take care of that. This relieves the developer from having to write code to manage the updates. The last but least issue to resolve is the transaction management of data updates. All commercial on-disk database products include this functionality to manage transactional updates of data. This is done by using a combination of temporary files and log files to temporarily save updates that belong to a group of updates until a commit transaction is triggered by the application code. The database product can then attempt to copy all the updates from the temporary store into the main database file(s). During this copying, writes may fail, and then the database ends up with partial transactions. To overcome this issue, the database server contains logic that is able to rollback the partial updates based on various techniques, including a snapshot of the original data before any updates begin, or other types of mechanism. This same type of logic can also be deployed by in-memory database products. Even though the database is supposed to be fully loaded in memory, that does not restrict the in-memory database server from utilizing the file system to temporarily store updates that occur within a transaction. By doing that, it combines the best of 2 worlds, in memory access to data which has optimal performance, and support for expected database capabilities such as transactions and CRUD in general.